Recognition, Analysis and Synthesis of Gesture Expressivity

1
Recognition, Analysis and Synthesis of Gesture
Expressivity

• George Caridakis
• IVML-ICCS

2
Overview

• Corpus
• Image processing module
• Gesture Recognition
• Expressivity Analysis
• Expressivity Synthesis
• Applications

3
Overview
4
Corpus mint-IVML

• 7 subjects
• 7 gesture classes
• 20 gesture variations (3 quadrants)
• 20 minutes 30000 frames

5
Corpus EmoTV
6
Corpus GEMEP (on going)
7
Head detection

• Detect candidate facial areas
• Validate using skin probability
• Conclude on number of persons

8
Hand Detection

• Skin probability

• Thresholding Morphology Operations

• Distance Transform

• Frame difference

9
Tracking

• Scoring system based on
• Skin region size
• Distance wrt the previous position
• Optical flow alignment
• Spatial constraints
• Thresholding scores
• Periodical re-initialization

10
Head Hand Tracking
11
HMM parameters for gestures

• States are head and hands coordinates
• XL-XR XH-XR XH-XL YL-YR YH-YR YH-YL
• 6 output states
• Bakis left-to-right models
• Continuous output distribution
• 3 Gaussian mixtures
• Arbitrary training initial estimation of
  transition probabilities

12
Recognition via HMM (Why HMMs?)

• Stochastic models fit the nature of the gestures
• Fast convergence due to effective training
  algorithms
• Sufficient modeling of the temporal aspect of
  gestures
• Continuous HMMs suitable for gesture-level
  classification

13
HMM overview
14
Recognition via HMM
15
Results
16
Expressivity features analysis

• Overall activation
• Spatial extent
• Temporal
• Fluidity
• Power/Energy
• Repetitivity

17
Overall activation

• Considered as the quantity of movement during a
  conversational turn
• Computed as the sum of the motion vectors norm

18
Spatial extent

• Modeled by expanding or condensing the entire
  space in front of the agent that is used for
  gesturing
• Calculated as the maximum Euclidean distance of
  the position of the two hands
• The average spatial extent is also calculated for
  normalization reasons

19
Temporal

• The temporal parameter of the gesture determines
  the speed of the arm movement of a gestures
  meaning carrying stroke phase and also signifies
  the duration of movements (e.g., quick versus
  sustained actions)

20
Fluidity

• Differentiates smooth/graceful from sudden/jerky
  ones. This concept seeks to capture the
  continuity between movements, the arms
  trajectory paths as well as the acceleration and
  deceleration of the limbs
• To extract this feature from the input image
  sequences we calculate the sum of the variance of
  the norms of the motion vectors

21
Power/Energy

• The power is actually identical with the first
  derivative of the motion vectors calculated in
  the first steps

22
Results of expressivity analysis
Spatial Extent
EF variation
Overall Activation
Temporal
Fluidity
Power/Energy
23
Expressive synthesis

• A system that mimics users behaviour through the
  analysis of facial and gesture signals and
  expressivity

24
Synthesis

• Greta Platform
• BAP calculation
• Plane assumption
• Inverse kinematics
• Manual adaptation
• Expressivity features variations implemented in
  Gretas BAP interpolation

25
Synthesis Results