Affective Technologies Update from the MIT Media Lab

1
Affective Technologies Update from the MIT
Media Lab
Rosalind Picard Hyungil Ahn Hoda Eydgahi
Shaundra Daily Rana el Kaliouby Seth
Raphael Alea Teeters

• http//affect.media.mit.edu

2
This character barges in when youre busy. He
doesnt apologize, and doesnt notice you are
annoyed. He offers you useless advice.You
express more annoyance. He ignores it. He
continues to be unhelpful. The clarity of your
emotional expression escalates. He ignores it.
(this goes on)
You finally
tell him Go Away He winks, and does a happy
little dance before exiting.
3
doesnt notice you are annoyed. Doesnt
recognize your emotion You express more
annoyance. He ignores it. Doesnt respond
appropriately to emotion He winks, and does a
happy little dance before exiting. Not clever
about expressing emotion.
4
Skills of Emotional Intelligence
Affective
affect

• Expressing emotion
• Recognizing emotion
• Handling anothers emotion
• Regulating emotion \
• Utilizing emotion /
• (Salovey and Mayer 90, Goleman 95)

if have emotion
5
Skills of Affective Intelligence

• Expressing affect
• Recognizing affect
• Handling anothers affect
• Regulating affect
• Utilizing affect

6
What to express when? Finding Match cars
vocal expression to driver affective state to
improve safety and performance.
Jonsson, I.-M. and Nass, C. (2004) Effects of
driver affect and car voice affect on actual and
perceived driving performance. Stanford CA
Stanford Univ.
7
What to express when? Finding Match cars
vocal expression to driver affective state to
improve safety and performance.
Jonsson, I.-M. and Nass, C. (2004) Effects of
driver affect and car voice affect on actual and
perceived driving performance. Stanford CA
Stanford Univ.
8
Challenge Recognize a persons affective state
before choosing what to express
Time to stop looking happy
9
Exception Its good to appear happy initially
(e.g. Apple computer) but then if the expression
persists, it needs to adjust to the person, e.g.
Klein
JK
10
Challenge Compile good default responses (and
dependence on context)
11
Skills of Affective Intelligence

• Expressing affect
• Recognizing affect
• Handling anothers affect
• Regulating affect
• Utilizing affect

12
Affect recognition

• Multimodal pattern analysis
• Flared nostrils, tightened lips, a quick sharp
  gesture, skin conductancehigh probably she is
  angry
• Common-sense situation reasoning
• That was an important goal to her and Bob just
  thwarted it again. She is likely to feel angry
  toward Bob but not show it in his presence.

13
MIT early recognition efforts (Wearables)
(SCskin conductance, BVPblood volume pulse, EMG
electromyogram, HRV heart rate variability,
Resp respiration)
14
MIT recognition efforts (Desktop)
15
Mouse pressure may increase with frustration,
distress
Pressure Sensitive Mouse
frustration linked to factors that cause wrist
problems (Dennerlein, et al., International
Ergonomics Association 03)
16
MIT recognition efforts (Text dialogue)
17
Challenge Make measurements out of the ways
feelings influence cognition

• For example
• How many times do you think you were interrupted
  today?
• How long do you think this task took?

18
People tend to estimate shorter times and fewer
interruptions with less frustrating experiences

• Relative Subjective Duration (Weybrew, 1984
  Czerwinski, Horvitz Cutrell, 2001)
• How long do you think you spent on that task?
  (vs. actual time that was spent)
• Relative Subjective Count (Picard and Liu, 2007)
• How many times does it seem like the system
  interrupted you? (vs. actual number of
  interruptions)

19
How many times does it seem like the system
interrupted you today? (10 subjects, 47 daily
logs) Relative Subjective Count (RSC) (Picard
Liu, 2007)
Empathetic system Estimated336 Actual
interruptions 496 RSC 0.68 Non-empathetic
system Estimated292 Actual interruptions
295 RSC 0.99
t(45), p .0024
20
MIT recognition efforts (Speech)
21
Understanding the Structure of Spoken Language
for Affect Modeling Open Source MATLAB code at
http//affect.media.mit.edu/publications.php
Intonation
Tempo
Syllables
Rhythmicality,
F0
Breaths
Pauses
Extralinguistic Markers
22
MIT recognition efforts (Video)
23
Can computer perceive complex cognitive-affective
states e.g., before interrupting?
hmm Roz is in the middle of concentrating now
is not a good time.
Inference and reasoning about mental states
Modify ones actions Persuade others
Analysis of nonverbal cues
24
Affective-Cognitive Mental StatesBaron-Cohen et
al. AUTISM RESEARCH CENTRE, CAMBRIDGE
Assertive Committed Persuaded Sure
Agreeing
Absorbed Concentrating Vigilant
Concentrating
Complex Mental States (subset)
Disapproving Discouraging Disinclined
Disagreeing
Asking Curious Impressed Interested
Interested
Brooding Choosing Thinking Thoughtful
Thinking
Baffled Confused Undecided Unsure
Unsure
25
Inferring Cognitive-Affective State from
FacialHead movements (el Kaliouby, 2005)

• Experimental Evaluation

• Conclusions

Facial feature extraction
Head facial action unit recognition
Head facial display recognition
Mental state inference
Head pose estimation
Hmm Let me think about this
Feature point tracking
Other examples Agree Disagree
26
Accuracy classifying six states computer scores
in top 6 of people
Accuracy of panel of 18 peopleAverage 54.5
Accuracy of computer Average 63.5 (better
than 17 of 18 panelists)
27
Helping people on the Autism Spectrum Center for
Disease Control and Prevention (2007)1 child in
155 has Autism
People w/autism usually have an Impaired
ability to recognize and predict affect in
social interaction
28
Skills of affectal Intelligence

• Expressing affect
• Recognizing affect
• Handling anothers affect
• Regulating affect \
• Utilizing affect /
• (Salovey and Mayer 90, Goleman 95)

if have affect
29
Results when computer handles affect by showing
empathy in context of frustrating experience
(short-term)
Users engage with frustrating task significantly
longer after empathetic response than after
control responses (Klein et al. 1999,
2002) Users have immediately lowered stress
level (via skin conductance) after empathy and
after apology (Prendinger et al. 2003 2005)
30
Challenge Can we measure/monitor stress many
times a day, every day for a week, without
elevating stress? (Picard Liu, IJHCS, 2007)
EKG, Accelerometer and Location Sensors
wirelessly talk to PDA (collaboration with
Fitsense/Motorola/HP)
31
How many times does it seem like the system
interrupted you today? (10 subjects, 47 daily
logs) Relative Subjective Count (RSC) (Picard
Liu, 2007)
Empathetic system Estimated336 Actual
interruptions 496 RSC 0.68 Non-empathetic
system Estimated292 Actual interruptions
295 RSC 0.99
t(45), p .0024
32
We compared 2 versions (person had 4 days with
each, average 12-19 interruptions per day, N10,
for a total of 79 days of usage data)

• System Morning, Jane!
• System Do you have a minute?
• User Yes.
• System You know the drill -- feeling stressed?
• User Its there - but not the worst.
• System Thanks so much for all your input. I
  hope I haven't been too frustrating.

Without Empathy With Empathy

• System Morning, Jane!
• System Do you have a minute?
• User Yes.
• System You know the drill -- feeling stressed?
• User Its there - but not the worst.
• System Wish it was better. Hope things start
  looking up.
• System Thanks so much for all your input. I
  hope I haven't been too frustrating.

33
System Preference (N10, each subject chooses one
system to keep using, after having used both
systems for eight days)
w/out empathy
w/empathy
34
Worlds first articulated home/office
computer (with Prof. Cynthia Breazeal, Hyungil
Ahn, Alea Teeters, Guy Hoffman, Andrew Wang)

• 75 sit in front of computers (static)
• Physical movement helps prevent/reduce back pain
• Back pain/injury 2 cause of missed work
• But its NOT just about ergonomics and health
  posture interacts with emotion and cognition

Animated Desktop Monitor Roco Robotic Computer
35
Emotion-posture congruence improves
perseverance( tracing attempts, two different
experiments)
36
Stoop to Conquer Posture congruent with
emotion improves perseverance( tracing
attempts, two different experiments)
37
Skills of affectal Intelligence

• Expressing affect
• Recognizing affect
• Handling anothers affect
• Regulating affect \
• Utilizing affect /
• (Salovey and Mayer 90, Goleman 95)

if have affect
38

• Can an affective agent that mirrors learner state
  improve learners self-reflection and learning
  meta-skills?

Study by Win Burleson with 76 children ages
11-13 2 X 2 Affective mirroring (from
Sensors vs. Script) Support (Empathic vs.
Domain) Examine influence on frustration,
perseverance, self-theories of intelligence,
motivation
39

• Can an affective agent that mirrors learner state
  improve learners self-reflection and learning
  meta-skills? Results

• Opposing effects boys and girls (ages 11-13)
• Affective interactions -gt positively related to
  girls meta-affective abilities, higher Flow,
  lower Stuck
• Coordination of intervention and non-verbals
  reduced girls frustration, especially for
  affect-support social mirroring
• (Burleson Picard, IEEE Intelligent Systems
  Special Issue on Intelligent Educational Systems,
  to appear)

40
We are giving machines internal mechanisms of
affect, computational instructions that perform
affect-like functions, especially for
decision-making. These do not give a machine
FEELINGS LIKE WE HAVE even though they perform
some of the functions feelings perform.

• (Ahn and Picard, EMCSR 2006 Vienna)

41
Challenge Can mobile/wearable sensor technology
help people better handle the affect that
influence risky decisions?

• The influence to make a poor choice is
  especially strong at anxious/stressful moments,
  or when cues trigger cravings.
• Can we create new technology to help a person
  better regulate their actions at that moment of
  risk?

42
Skills of Affective Intelligence

• Expressing affect
• Recognizing affect
• Handling anothers affect
• Regulating affect
• Utilizing affect

43
Summary

• We can begin to implement affective intelligence
  (much harder than making computers that appear
  emotional)
• The affectively intelligent system will both
  RECOGNIZE and RESPOND to human affect. It may
  also require some internal mechanisms of affect.
• Multiple channels, combined, provide improved
  affective inference, but far from 100
  recognition (and people are also far from 100)
• MANY applications Agents, Robots, Driver
  navigation systems, Learning tutors, Autism,
  Addiction and health behavior change, Pain
  management, Call Center support

44
Papers/Details http//affect.media.mit.edu

• Acknowledgments Ashish Kapoor, Win Burleson,
  Rana el Kaliouby, Carson Reynolds, Marc Strauss,
  Selene Mota, Hyungil Ahn, Shaundra Daily, Alea
  Teeters, Seth Raphael, Hoda Eydgahi, Yuan Qi,
  John Rebula, Barry Kort, Rob Reilly, Ken Perlin,
  John Lippincott, Rich Fletcher.
• This work has been supported in part by the
  National Science Foundation under ITR
  0325428 and SGER IIS-0555411, and by the Media
  Lab Things That Think consortium.