JC. Martin, L. Devillers, A. Zara LIMSICNRS

1
EmoTABOU Corpus

• J-C. Martin, L. Devillers, A. Zara LIMSI-CNRS
• V. Maffiolo, G. Le Chenadec France Télécom RD
• France

2
Research Context and Goals
3
Research context

• Long-term goal model of human computer emotional
  interaction
• Requires knowledge on emotional multimodal
  behaviors during human-human interaction (e.g.
  synchronization between modalities)
• Corpus-based approach
• Experimental data and studies
• Monomodal
• Acting single emotion
• Emotion but interaction not videotaped (EmoTV)
• Or interaction videotaped but not emotion

4
Related workDiscriminative features of mvt
quality (Wallbott 98)
How does that applies to not (instructed in-lab
acting of single emotions) ?
5
Research context

• Cognitive - motivational - emotive systems such
  as EMA (Gratch and Marsella) are mainly based on
• theoretical psycho-cognitive models
• behavioral models based on acted emotions
• Our aim is
• to observe and analyze corpora of spontaneous
  human-human interaction with emotional multimodal
  behavior
• to build more realistic and natural
  behavioral models

6
Research questions

• Questions
• how do emotion and interaction combine?
• what is the impact of both on the synchrony
  between modalities?
• Gesture stroke phase and lexical affiliate
  (McNeill 05), max F0
• Gaze during mental states (Baron-Cohen 97) and
  turn-taking (Allwood 06)
• How to collect behavior that
• are spontaneous
• are multimodal
• are emotional
• occur during interaction

7
Experimental Protocole
8
Experimental protocolAdaptation of the Taboo game

• Taboo game
• 1 card with one secret word and 5 forbiden
  words
• Multimodal elicitation
• Iconic
• Deictics

9
Experimental protocolAdaptation of the Taboo game

• Emotion elicitation
• uncommon word to elicit surprise or embarrassment
• 1 player was a naive subject
• other player instructed to elicit emotion using
  strategies (might not find the word on purpose)

10
Collected data

• 10 pairs of players
• 8 hours of video
• Upper body face close-up

11
Samplepalimpsest
12
Levels of annotation

• multimodal behavior (acoustic/gestures/face)
• linguistic behavior (dialog/com. acts)
• emotional/mental state behavior
• strategic behavior

13
Annotation of emotion context
14
Previous scheme

• Multi-level scheme for emotion and context
  representation
• Emotion labels (broad sense including attitude,
  emotion, mood)
• Dimensions (valence and intensity)
• Contextual information (quality, speaker, etc.)
• EmoTV (Devillers, Abrilian, Martin, 2005)
• CEMO (Devillers et al., 2005)

15
EmoTabou scheme

• Adaptation of our previous scheme for emotions
  annotation in interaction
• We added
• More general set of mental states
• Dialog acts
• Communicative acts
• Contextual information scheme (sub-dialog of the
  game, role of the subject, card, etc.)
• Meta information

16
Emotion
labels in EmoTabou
The protocol for obtaining this list was to rate
the emotion words of the Humaine list (55 terms)
in terms of their relevance for the task
(majority Voting procedure 5 people). In
order to represent complex emotions, we allow the
annotation of at most 5 emotions per segment.
Then we computed the different annotations of
several labelers in a soft vector
representation.
17

Mental states and emotions

• Other studies list of emotional labels extended
  with other mental states (Reidsma 06, Le
  Chenadec et al., 05)
• We added to our list the mental states defined by
  Baron-Cohen (96) (ie. Thinking, Unsure)
• Aims
• Study the relation between emotions and mental
  states between the two players
• Study how emotions and mental states are
  expressed through multimodal behaviors in human
  interactions

18
Mental states (Baron-Cohen, 1996)
19
Dialog acts (DAMSL)

• DAMSL scheme Dialog Act Markup in Several
  Layers
• annotation of interaction (4 levels
  Information-Level, Communicative Status, and
  Forward- and Backward-Looking Functions)
• more than 75 tags
• Experiments using multi-level annotations
  dialogic and emotion tags carried out with the
  FP5-AMITIES (Devillers 02) showed correlation
  between emotion/some dialog acts in speech
• ex anger with repetition.
• We use a reduced set of DAMSL tags adapted to
  EmoTabou

20
Dialog acts (DAMSL)

• Give a cue
• Suggest a word
• Assert other
• Ask a question
• Understand
• Answer Yes
• Answer No
• Dont know
• Interjection
• Inintelligible 

Forward-Looking Functions Backward-Looking
Functions Communicative Status
21
Communicative functions

• Previous works have already provided lists of
  communicative functions (Poggi, Pelachaud)
• Here, we defined a list after analysing our
  corpus
• Abandon, Disapprove, Criticize, Self-criticize,
  Lack-of-confidence, Doubt about other, Rush,
  Unkind, Irony, Mocking, Joke, Sarcastic.
• Admire, Approve, Congratulate, Encourage,
  Congratulate, propose strategy,

22
Contextual information and Meta-information

• We defined a contextual information scheme
• strategies list of strategies given to the
  associate or observed in the corpus.
• game phases give a card, play, give the result
• card to guess
• Player role devin (mind-reader) or mime
• Meta-information Post-game information
• subject personality (Eysenck personality
  inventory)
• questionnaire (emotions felt and elicited)

23
Examples of the different strategies

• Associate
• Irritation the associates have the instruction
  to criticize the subject
• Card
• Embarrassment to embarrass the subject, unusual
  words have been chosen like palimpseste
• Experimenter
• Stress the subject has 2 minutes to guess a
  word. After 1mn30, the experimenter announces 30
  seconds left, then 15 seconds

24
Annotation protocol for Emotion and context

• The coding scheme is implemented in Anvil (Kipp
  04)
• To annotate the corpus, we proceed in the
  following way
• 1) Segmentation
• 2) Annotation by four annotators
• Iterative definition of the coding scheme
• Test with one video
• Measure agreement (intra, inter-coder agreement)

25
Annotation of multimodal behaviors
26
Informal study of the collected behaviors

• iconic gesture describing the action "turning a
  split",
• deictic gesture indicating the scores listed on
  the black board,
• (c) adaptator gesture done by the naive subject
  and imitation by the instructed subject (d)

27
Annotation of multimodal expressive behaviors

• Gaze direction
• Gesture
• Phase
• Function (including manipulators)
• Expressivity (adapted from Pelachaud 05)
• Facial expressions (subset AU)
• Head mvt
• Posture

28
Annotation of multimodal expressive behaviors

• Anvil (Kipp 04)
• 1 coder
• Agreement

29
Future Directions Illustrations of possible
measures
30
Descriptive analysis of one clip Emotion
repartition
31
Emotion vs Mental states

• We observed some relations between our set of
  emotions and more general mental states
• Embarrassment -gt
• unsure

32
Ex soft-vectors representation (Emotion label,
weight, intensity, valence) (Mental state, weight)
Subject ok, I propose that we do not even try
this one and accept the penalty
33
Emotions/Communicative acts
34
Multimodal behaviorsIllustration of measures to
be done

• Naïve player gestures (RH symetrical gestures)
• 83 of video / 7 gesture units
• High percentage of manipulators (48) and hold
  (60)
• Gaze direction x gesture type
• Adaptators 59 Interlocutor, 31 elsewhere
• Deictics 63 Interlocutor, 18 Panel, 19
  Elsewhere
• Expressivity and gesture type
• 59 deictic expanded, 94 adaptator contracted
• Expressivity and phases
• 44 of smooth gestures occur during stroke

35
Future directionsSynchronization between
modalities

• Further annotations
• Lexical affiliate
• Facial expression close-up
• Temporal analysis
• Between modalities
• Between modalities / mental states
• Measures
• Gaze / mental state
• Individual behaviors

36
Future directionsSynchronization between
modalities

• Annotation and measures
• feedback, sequencing et turn management (Allwood
  06)
• Imitation
• Relations between the different levels of
  annotation in the behavior of the two players
• Comparison with pairs of naïve subjects