Pierre Wellner

1
Browsing Recorded Meetings
Pierre Wellner IDIAP and IM2 SCSC 04
2
Outline

• IDIAP IM2
• Smart Meeting Room
• Visual Audio Tracking
• Browsing Meetings
• Browser Evaluation

3
IDIAP in brief
IDIAP in Brief

• A private non-profit research institute.
• Located in Martigny, Valais since 1991.
• 80 persons, 65 researchers.
• Annual budget 8 MCHF.
• Three missions
• Research
• Education
• Technology Transfer

4
IDIAP Research Themes
IDIAP Research Themes

• Machine Learning
• Algorithms for classification, regression and
  density estimation, etc.
• Speech Processing
• Spoken language understanding, noise-robust
  speech recognition, large vocabulary speech
  recognition, low bit rate transmission.
• Computer Vision
• Object recognition, motion analysis, text
  recognition.
• Media Indexing
• Structuring audio and video, noisy text
  information retrieval.
• Biometric Authentication
• Speaker verification, face verification,
  multimodal fusion.
• Multimodal Interaction
• Meeting browser, HCI design, brain-computer
  interfaces

5
(No Transcript)
6
Browsing Recorded Meetings

• Quickly find what happened in meetings.
• Recording is easy, but finding is hard.
• Technology for three steps

1. Recording
2. Analysis
3. Browsing
7
Recording Smart Meeting Room

• Synchronized recording
• 24 audio channels
• microphone arrays
• binaural mannequin
• 3 video channels
• Computer projector
• White board
• Notes on paper

8
Recording Smart Meeting Room
9
Recording Conference Calls
Talker Pierre Wellner, Spiderphone Callers 8
Talker Mike Flynn, IDIAP Callers 8
Talker ---- Callers 8
Talker Pierre Wellner, Spiderphone Callers 8
Talker Mike Flynn, IDIAP Callers 8
Talker ---- Callers 8
Talker Pierre Wellner, Spiderphone Callers 8
10
Analysis Visual Tracking
11
Analysis Audio Tracking
12
Analysis AudioVideo Tracking
13
Meeting Browsersbringing it all together

• Display multimodal Analysis results
• Speaker tracking (who is talking)
• Recognized speech
• Meeting Actions (e.g. presentation, discussion)
• Interest levels, etc
• Control audio and video playback

14
Ferret Architecture
Browser Architecture
Server Client
Media
Real Server
Real Player
XMLtoSVG Servlet
XMLtoSVG Servlet
Internet Explorer
XMLtoSVG Servlet
Processing
Servlets, CGI JSP
SVG Viewer
Processing
Processing
Processing
XML Data
Various Text Transcripts
Apache Tomcat
Demo
15
The Browser Evaluation Problem
The browser evaluation problem

• No evaluation, or...
• Tested by unique scheme
• Often very subjective
• from Cutler et al, Distributed Meetings A
  Meeting Capture and Broadcasting System, ACM
  Multimedia, 2002
• I was able to get the information I needed
• I would use this system again if I had to miss a
  meeting.
• I would recommend the use of this system to my
  peers.
• No standard Browsing task
• ? Objective comparisons not possible ?

16
Aims for a good BET
Aims for a good BET(Browser Evaluation Test)

• Performance, not judgment.
• Independent of experimenter perception.
• Directly comparable numeric scores.
• Replicable.

17
The Media Browsing Task
The Media Browsing Task

• Find a maximum number of
• observations of interest
• in a minimum amount of time.

But what are observations of interest?
18
BEToverview
19
Summary

• IDIAP and the IM2 project
• Recording in the Smart Meeting Room
• Visual Audio Processing
• Browsing Meetings
• Browser Evaluation

20
Related Research Projects

• EC-FP5-IST
• MultiModal Meeting Manager (M4)
• http//www.m4project.org
• EC-FP6-IST Integrated Project
• Augmented Multi-party Interaction (AMI)
• http//www.amiproject.org
• National (CH) Research Competence Center on
• Interactive Multimodal Information Management
  (IM2) http//www.im2.ch
• DARPA EARS (US)
• http//www.darpa.mil/iao/EARS.htm