Active Capture and Folk Computing

1
Active Capture and Folk Computing

• Ana Ramírez and Marc Davis
• ICME 2004 Taipei, Taiwan
• 29 June 2004

UC Berkeley - Garage Cinema Research - Group for
User Interface Research
2
Smart Multimedia Acquisition Systems

• First two papers automatic camera calibration
• Image
• Audio
• Third paper understand structure of what is
  being captured to edit in real time
• Active capture - smart cameras that interactively
  guide and capture human action

3
Outline

• Sample applications
• Active Capture
• Designing Active Capture algorithms
• Future work

4
Automatic Movie Trailers
Sample Applications

• Video of capture process

Play Video
5
Automatic Movie Trailers
Sample Applications

• Video of automatically created movie trailer

Play Video
6
Sports Instruction
Sample Applications
7
Telemedicine
Sample Applications
Rural Town
Large City
leishmaniasis
8
Automated Health Screening
Sample Applications
Rural Town
Large City
leishmaniasis
9
Active Capture
Direction/ Cinematography
Capture
Interaction
Active
Capture
Human- Computer Interaction
Computer Vision/ Audition
Processing
10
Active Capture
Direction/ Cinematography
Capture
Interaction
Active
Capture
Human- Computer Interaction
Computer Vision/ Audition
Processing
11
Active Capture
Direction/ Cinematography
Capture
Interaction
Active
Capture
Human- Computer Interaction
Computer Vision/ Audition
Processing
12
Active Capture
Direction/ Cinematography
Capture
Interaction
Active
Capture
Human- Computer Interaction
Computer Vision/ Audition
Processing
13
Active Capture
Direction/ Cinematography
Capture
Interaction
Active
Capture
Human- Computer Interaction
Computer Vision/ Audition
Processing
14
Active Capture

• Traditionally, signal processing algorithms avoid
  interacting with the user
• Signal processing interaction gt more
  sophisticated recognizers
• How to design hybrid algorithms that involve
  capture, interaction, and processing

15
Components of Active Capture Algorithms

• Simple computer vision and audition recognizers /
  sensors
• Motion
• Eyes
• Sound
• Desired action in terms of recognizers
• Interaction script

16
Design Process

• Input
• Desired action head turn
• Recognizers motion, eyes

Motion
Eyes
time
17
Design Process

• Input
• Desired action head turn
• Recognizers motion, eyes
• Step 1
• Express desired action in terms of recognizers

No Motion
Motion
No Motion
Motion
Eyes
No Eyes
Eyes
time
18
Design Process

• Input
• Desired action head turn
• Recognizers motion, eyes
• Step 1
• Express desired action in terms of recognizers
• Step 2
• Design interaction script

19
Design Process Step II
20
Design Process Step II
21
Design Process Step II
Play Video
22
Design Process Step II
23
Design Process Step II
Play Video
24
Design Challenges

• Step I - Description of action
• Approximate timing
• Strict and non strict ordering
• Step II Interaction script
• What to do if something goes wrong mediation

25
Step I Action Description
26
Visual Language
Step I - Action Description

• Observations
• Commands
• Capture
• Time constraints
• Strict ordering
• Non-strict ordering

27
Visual Language
Step I - Action Description

• Observations
• Commands
• Capture
• Time constraints
• Strict ordering
• Non-strict ordering

28
Visual Language
Step I - Action Description

• Observations
• Commands
• Capture
• Time constraints
• Strict ordering
• Non-strict ordering

29
Visual Language
Step I - Action Description

• Observations
• Commands
• Capture
• Time constraints
• Strict ordering
• Non-strict ordering

30
Visual Language
Step I - Action Description

• Observations
• Commands
• Capture
• Time constraints
• Strict ordering
• Non-strict ordering

31
Visual Language
Step I - Action Description

• Observations
• Commands
• Capture
• Time constraints
• Strict ordering
• Non-strict ordering

32
Visual Language
Step I - Action Description

• Observations
• Commands
• Capture
• Time constraints
• Strict ordering
• Non-strict ordering

33
Step II Interaction Script
34
Contextual Inquiries
Step II Interaction Script

• Golf instructor
• Aikido instructor
• 911 emergency phone operator
• Triage nurse
• Childrens portrait photographer
• Film and theatre directors

Jeffrey Heer, Nathaniel S. Good, Ana Ramirez,
Marc Davis, and Jennifer Mankoff. Presiding Over
Accidents System Direction of Human Action. In
Proceedings of the Conference on Human Factors in
Computing Systems (CHI 2004) in Vienna, Austria.
ACM Press, 463-470, 2004.
35
Direction and Feedback Strategies
Step II Interaction Script

• External aids

Play Video
36
Direction and Feedback Strategies
Step II Interaction Script

• Decomposition and Show

Play Video
37
Direction and Feedback Strategies
Step II Interaction Script

• Method shift from Show to Tell

Play Video
38
Direction and Feedback Strategies
Step II Interaction Script
39
Summary

• Active Capture smart cameras that interactively
  guide and capture human action
• Sample applications
• Automated health screening
• Automated movie clips
• Sports trainer
• Design Challenges
• Description of action
• Interaction script

40
Future Work

• Support design and implementation of Active
  Capture applications
• Evaluate the relative contribution of signal
  analysis and user interaction in these hybrid
  algorithms

41
Questions

• Ana Ramírez
• anar_at_cs.berkeley.edu
• www.cs.berkeley.edu/anar
• Garage Cinema Research
• http//garage.sims.berkeley.edu
• Group for User Interface Research
• http//guir.berkeley.edu