Christian Becker, Stefan Kopp, Ipke Wachsmuth

1
Simulating the Emotion Dynamics of a Multimodal
Conversational Agent

• Christian Becker, Stefan Kopp, Ipke Wachsmuth
• AG WBS, University of Bielefeld, Germany

Video
2
The agent Max
Kinematic skeleton with 53 degrees of freedom

• Body animated by model-based animation
• Cognitive architecture based on a BDI framework
• Text-to-speech system (German only)
• TXT2PHO (IKP Uni Bonn) and MBROLA
• Phoneme transcription as basis for automatic
  generation of visemes
• Facial animation based on 21 simulated muscles
• Coordinated control of face muscles based on
  Action Units (Ekman/Friesen)
• Six muscle sets defined for emotional expressions

surprised
angry
sad
bored
fearful
happy
Introduction
3
Two interaction scenarios
An embodied conversational agent situated in
virtual reality

• (1) Multimodal Assembly eXpert
• SFB 360 research scenario
• Located in a 3side cave-like projection
• User input via speech and gestures
• Action selection influenced by the concurrent
  emotion system

• (2) Presentation agent in the HNF museum
• Multimodal presentation module extended by
  eliza-like, smalltalk abilities
• User interaction via keyboard
• User perception via video camera
• Proactive behaviors triggered by the concurrent
  emotion system

Introduction
4
Why not OCC?
Ortony, Clore and Collins (1988) An appraisal
theory of emotions

• Positive aspects of the OCC model
• covers most situations an emotional agent might
  have to deal with
• offers a structure to determine the intensity of
  emotion types
• Untackled questions within the OCC model
• the OCC model contains no history
• the interactions of different emotion categories
  are not described
• the course of emotions over time is not
  considered
• and last but not least

The non-cognitive emergence of emotions is
neglected by this model! (E.g. The experience
of relaxation when sitting in front of a warm
oven.)
Related work
5
The emotion system an overview
Simulating the dynamics of emotional impulses

• Dynamic component Conceptual linkage of emotions
  and moods
• CategorizationMapping onto weighted emotion
  categories in PAD space by using a distance metric

Input
Output

• Valence of emotions short-time system state
• Valence of moods longer lasting system state
• (P)leasure ranges from joy (P) to reluctance
  (-P)
• (A)rousal ranges from mental awareness (A) to
  sleepiness (-A)
• (D)ominance describes the agents feelings of
  control over the situation

Emotion system
6
1. Dynamic component

• Simulating the alleviating and fortifying effects
  of emotions (x) on moods (y) (as indicated by the
  vertical arrows)
• Physical simulation of two spiral springs with
  spring constants dx and dy

Provides triple (x, y, z) at every timestep t
with 25Hz!
Emotion system
7
Mapping to PAD space
Emotion (x), mood (y) and boredom (z) are mapped
to (p)leasure and (a)rousal according to the
following equations
d
p
y
z
x
a
The dominance value is not deducible from the
dynamic component!
Emotion system
8
2) Categorization (PAD) component

• Nine emotion categories represented by their PAD
  triples
• ( p , a , d )
• angry (-0.8, 0.8, 1.0)
• annoyed (-0.5, 0.0, 1.0)
• bored ( 0.0, -0.8, 1.0)
• concentrated ( 0.0, 0.0, /- 1.0)
• depressed ( 0.0, -0.8, -1.0)
• fearfull (-0.8, 0.8, -1.0)
• friendly ( 0.5, 0.0, /- 1.0)
• sad (-0.5, 0.0, 1.0)
• surprised ( 0.8, 0.8, /- 1.0)
• Two thresholds are defined
• Activation threshold ?.
• Saturation threshold ?.

d
Emotion system
9
Visualisation of the emotional state

• Output of the emotion system
• Valence of mood and degree of boredom
• Corresponding PAD triple
• Activated emotion category with weight w or state
  of confusion otherwise
• 1) Non-cognitive effects
• Weighted facial expressions according to the
  actual emotion category
• Modulation of breath and eye-blink frequency by
  arousal value
• Modulation of pitch and rate of speech according
  to the emotion category
• Triggering of secondary actions by boredom value

• Cognitive effects
• Emotion categories incessantly asserted as belief
  of the agent
• Agents plan selection influenced by current
  emotion category

Emotion system
10
Example

• HNF scenario since January 2004
• Emotion system provided with emotional impulses
  by perception, interpretation and dialog manager
• It influences dialog manager (cognitive) and
  behavior generation (non-cognitive)

emotionsystem
behaviorplanning
dialogmanager
inter-pretation
behavior generation
perception
Video
Conclusion
11
Conclusion

• Combination of moods, emotions and boredom to
  achieve a coherent long-time behavior as well as
  convincing spontaneous reactions
• Involuntary facets of Maxs behavior, his facial
  expressions and his deliberation process are
  continuously modulated by the emotion system
• Higher-level emotion categories not representable
• Open questionHow can the dominance value be
  specified automatically?

Conclusion
12
Thank you for your attention!