WP 4: Context Aware Video Acquisition

1
WP 4 Context Aware Video Acquisition

• James L. Crowley
• Professeur I.N.P.Grenoble
• Project PRIMA, Laboratoire GRAVIR
• INRIA Rhône Alpes

2
WP 4 Context Aware Video Acquisition

• Plan
• Situations and Actions for the Lecture Scenario
• Assembling a Process Federation
• Version 1.0 of the Intelligent Camera Man
• Current Work

3
FAME Augmented Meeting Environment

• 5 Sony Steerable Cameras
• Wide Angle Camera
• Microphone Array
• 3 Video Interaction Devices
• Vertical
• Horizontal
• Steerable

4
FAME Augmented Meeting Environment
5
WP 4 Context Aware Video Acquisition

• Plan
• Situations and Actions for the Lecture Scenario
• Process Federation for CA Video Acquistion v1.0
• Task, Situation and Context
• Compiling the Situation Graph
• Assembling a Process Federation
• Version 1.0 of the Intelligent Camera Man
• Current Work

6
Camera Control Federation
7
Task, Situation and Context

• Situation An configuration of roles and
  relations.
• Role Interpretation of an entity or agent
• Relation A predicate over entities and agents
• A task model describes the state space of
  situations
• and the actions of the system for each situation
• Approach Compile a federation of processes to
  observe the agents and entities that define
  situations.

8
Task, Situation and Context

• Basic Concepts
• Property Any value observed by a process
• Entity A correlated set of properties
• Composite entity A composition of entities
• Relation A predicate defined over entities
• Agent An entity that can act.
• Situation A configuration of roles and
  relations.
• Context A network of situations

9
Context Aware Video Acquisition

• Design Method
• 1) Define actions to be taken by system
• 2) Define situations for each action
• 3) Define roles and relations
• 4) Define observation processs
• 5) Compile situation graph into supervisor rules.

10
Actions to be taken by Context Aware Video
Acquisition System v1.0

• Record Shots
• A1 Record wide angle view of the scene
• A2 Record the speaker
• A3 Record the audience

11
Camera Angles
12
Situations for the Context Aware Video
Acquisition System

• Situations
• S0 empty room Æ A1
• S1 Actor enters the room Æ A1
• S2 Speaker (actor) speaks Æ A2
• S3 Audience (actor) asks a question Æ A3

13
Process Federation Tool

• JESS (CLIPS in Java) Environment sends messages
  to processes

14
Define the roles and relations for Context Aware
Video Acquisition System

• Roles and relations for camera controller
• R1 Agent in audience asks a question
• (Agent in audience is speaking)
• R2 Lecturer (Agent at lecture-position)
• R3 Arriver (Agent at door)
• R4 Audience (Agents in audience region)
• R5 Speaker (Agent currently speaking)

15
Compiling Situations to Rules
XML ltrole name"lecturer" arity"1"gt ltdescriptio
ngt The person giving a lecture lt/descriptiongt lt/ro
legt
Situations??Petri Net??XML Description-gtRules
16
Compiling Situations to Rules
CLIPS (defrule t2EventTransition ?tr lt-
(transitionTrigger (name "t2") (lock ?l(neq ?l
0))) ?pre_S1 lt- (situation (name "S1")
(entities ?newComer)) (situationPlace (name
"S1") (mark ?m_S1(neq ?m_S1 0))) (or (not
(newComer (isPlayedBy ?newComer))) ) (lecturer
(isPlayedBy ?new_lecturer)) (speaker
(isPlayedBy ?new_speaker)) (isSameAs
(isVerifiedBy ?new_speaker ?new_lecturer))
?post_S2 lt- (situation (name "S2")) gt
(modify ?tr (lock (- ?l 1))) (assert (event
(name "t2"))) (modify ?pre_S1 (entities))
(modify ?post_S2 (entities ?new_lecturer
?new_speaker)) )
Situations??Petri Net??XML Description-gtRules
17
WP 4 Context Aware Video Acquisition

• Plan
• Situations and Actions for the Lecture Scenario
• Assembling a Process Federation
• Perceptual Processes
• Tracking Bodies Hands and Faces
• Recognizing and Locating Speech Sounds
• Version 1.0 of the Intelligent Camera Man
• Current Work

18
Video Acquisition Process Federation
19
Processs to observe agents, entities and
relations

• P1 Supervisory Controller
• P2 Visual Tracking Process for agents with
  Camera 1 (Wide Angle camera)
• P3 Visual Tracking Process for agents with
  Camera 2 (Audience region)
• P4 Visual Tracking Process for agents with
  Camera 3 (lecturer region)
• P5 Speech preprocessing and detection
• P6 Speech position estimation.

20
Agent Detection and Tracking Process

• Observation Modules
• Color Histogram Ratio  Background Difference
• Receptive Field Histograms  Motion History Image

21
Agent Detection and Tracking Process

• Process Phases
• While True Do
• Acquire next image
• Calculate ROI for targets
• Verify and update targets
• Detect new targets
• Regulate module parameters
• Interpret entities
• Process messages

22
Agent Detection and Tracking

• Actors Composite Entities.
• Entity Tracker Background difference, motion and
  color
• Entity Grouper Assigns roles to blobs as body,
  hands, face or eyes

23
Acoustic Perception Processes
24
Acoustic Process Federation
25
Multi channel Acoustic Server Process
Channel selection
Speech Waveform and Spectrogram
Remove soundcard offset
Usage
Adaptive cepstral echo cancellation
Final voice activity detection (using energy and
neural net results)
Temporal energy analysis
Voice detection
26
Acoustic Position Estimation
27
Speech Detection Process
28
WP 4 Context Aware Video Acquisition

• Plan
• Situations and Actions for the Lecture Scenario
• Assembling a Process Federation
• Version 1.0 of the Intelligent Camera Man
• Current Work

29
Context Aware Video Acquisition v1.0
30
Context Aware Video Acquisition v1.0
31
WP 4 Context Aware Video Acquisition

• Plan
• Situations and Actions for the Lecture Scenario
• Assembling a Process Federation
• Version 1.0 of the Intelligent Camera Man
• Current Work
• Adding Camera Pan-Tilt Control
• Adding New cameras
• Estimating Face Orientation
• Integration of Topic Spotter

32
Camera Control
33
Face Orientation Estimation
34
WP 4 Context Aware Video Acquisition

• Plan
• Situations and Actions for the Lecture Scenario
• Assembling a Process Federation
• Version 1.0 of the Intelligent Camera Man
• Current Work

35
WP 4 Context Aware Video Acquisition

• Plan
• Situations and Actions for the Lecture Scenario
• Assembling a Process Federation
• Version 1.0 of the Intelligent Camera Man
• Current Work

36
WP 4 Context Aware Video Acquisition

• PRIMA Group, Laboratoire GRAVIR (UMR)
• INPG (P2), INRIA (P8), UJF (P9), CNRS(P10)
• Personnel Contributing during period
• James L. Crowley (Prof. INPG)
• Augustin Lux (Prof INPG)
• Patrick Reignier (MdC UJF)
• Dominique Vaufreydaz (Post Doc INPG)
• Alban Caparossi (Engineer UJF)
• Stan Borkowski (Doctoral Student, INPG)
• Hai Tranh (Doctoral Student, INPG)
• Nicolas Gourier (Doctoral Student, INRIA)