Cues to Emotion: Face, Body, Brain

1
Cues to Emotion Face, Body, Brain

• Bob Coyne (cs6998)

2
Outline

• Emotion in voice and brain
• The voice of emotion an FMRI study of neural
  responses to angry and happy vocal expressions
  (2006) Tom Johnstone et al
• Facial Expressions and emotion
• http//en.wikipedia.org/wiki/Paul_Ekman
• Facial Action Coding System (FACS) Ekman
  Friesen
• Microexpressions - Ekman
• Poker tells The Body Language of Poker (1984)
  Mike Caro
• 3D graphics facial expressions FaceGen software
• www.facegen.com

3
The voice of emotion an FMRI study of neural
responses to angry and happy vocal expressions
(2006) Tom Johnstone et al

• Range of issues
• Emotions in general vs specific emotions
• General regions of brain (left/right) vs
  localized
• Perception vs production
• Prosody vs semantic content in speech
• Facial expressions vs speech
• Attention vs inattention
• Type of experiment, controls, and measurement
  (FRMI, MEG, PET, EEG, other)
• What function does vocal affect have and how to
  test

4
Importance of response to vocal affect

• From the earliest stages of development, infants
  respond to affect-laden vocal expressions from
  their mothers (Fernald, 1989 Fernald and
  Morikawa, 1993)
• Why might this be?
• Relevance to the 4 approaches (darwinian,
  jamesian)?
• Still important/relevant after?
• When?
• Emoticons?
• Emotion in voice recognized as well as faces
  across cultures (Scherer and Wallbott, 1994)
• True?

5
Early Prior work

• Production/comprehension experiments
• Production vs perception. Makes a difference?
• Production in Right hemisphere (Hughlings-Jackson
  - 1915)
• How did they know?
• Neurological evidence for comprehension (Tucker
  et al., 1977)
• Reduced affective speech comprehension in
  right-hemisphere-damaged listeners (bowers 1987,
  Heilman et al., 1984 Peper and Irle, 1997 Ross,
  1981)
• Whats missing?
• Where processed?
• Controlled experiments?
• How else to test/identify region?

6
More recent Prior work

• Specific regions for facial emotion processing
  (e.g. amygdala for fear, insula for disgust)
• Little prior work on localized regions for
  emotional speech for specific emotions.
• Propositional content vs emotional prosody
• Right prefrontal activation (PET - George et al.
  1996)
• EEG scalp electroencephalography Pihan et al.
  1997
• Accented syllable duration and fundamental
  frequency
• Are those correlated with emotional speech?
• Magnetoencephalography (MEG) (Imaizumi et al.
  1998)
• Also found some left-hemisphere processing of
  propositional and prosodic content

7
Recent Prior work

• non-differentiated per emotion
• Selective activation
• Congruence effect with facial and vocal
• Experiments
• Mitchell et al. (2003) found areas of posterior
  middle temporal gyrus (MTG) and superior temporal
  sulcus (STS) that activated more when attending
  to affective prosody as compared with semantic
  content of spoken words
• Grandjean et al. (2005) and Sander et al. (2005)
  have reported FMRI data that revealed a region in
  STS that showed greater activation in response to
  angry speech as compared with neutral speech
• Wildgruber et al. (2005) identified a right
  hemisphere network consisting of posterior STS,
  and dorsolateral and orbitobasal prefrontal
  cortex that showed selective activation during an
  emotion recognition task. Differential
  activations for the five emotions were not
  observed.
• Ethofer et al. (2006) identified regions in the
  right posterior MTG and STS and bilateral
  inferior/middle frontal gyrus that activated more
  when individuals identified affective prosody
  than when identifying the emotional content of
  the spoken words. No distinction was made between
  responses to the different expressed emotions
  studied
• Pourtois et al. (2005) demonstrated an area in
  left MTG that showed heightened activation when
  congruent vocal expressions and facial
  expressions of happiness or fear were
  simultaneously presented, as compared with when
  only one expressive modality was presented

8
The Brain
Whole Brain Atlas Top 100 Brain Structures
http//www.med.harvard.edu/AANLIB/cases/caseM/cas
e.html
9
Loci of Sulci and Gyri

1. Central sulcus
2. Precentral gyrus
3. Postcentral gyrus
4. Precentral sulcus
5. Postcentral sulcus
6. Middle frontal gyrus
7. Superior frontal gyrus
8. Superior parietal lobule
9. Occipital gyri
10. Longitudinal fissure

10
Highly specialized brain function?
11
Hypotheses

• The question remains, therefore, of whether
  specific neural regions are more engaged in the
  processing of some emotions than others.
• Based on the hypothesis that affiliative social
  vocal signals are prevalent throughout our lives
  and serve a fundamental purpose in social binding
  we hypothesize that emotional expressions of
  happiness would preferentially engage parts of
  the temporal cortex and inferior frontal regions
  previously shown to be involved in the processing
  of affective prosody.
• A further question concerning the perception of
  vocal expressions of emotion is how directed
  attention towards or away from the expressed
  emotion affects the associated neural response.
• How to test this?
• Activation for face versus voice most studies
  on face
• What types of emotional speech detected? All?

12
Experimental approach

• FMRI to detect localized regions for processing
  happiness vs anger in speech
• Why those two emotions?
• Reasons
• Control for similar acoustic signal.
• Common/basic. Easier to characterize.
• social utility hypothesis regarding happiness
• Which theory of emotion (jamesian etc)
• What was experimental setup?
• Some factors
• 40 subjects
• Facial stimuli consisted of 16 grayscale images
  of posed expressions of anger and 16 expressions
  of happiness, half of them females.
• Vocal stimuli consisted of short phrases (dates
  and numbers) lasting on average 1 s, spoken with
  either angry or happy prosody.
• Half the participants make their decision on the
  basis of the facial expression, while the other
  half on the basis of the vocal expression
• FMRI scan during happy/angry speech
• With corresponding or opposite facial expression

13
Experiment Issues/problems?

• Brain regions necessarily responding to emotion?
• Quality of audio? Affects results?
• FMRI measures blood flow. Relevant to neural?
• Right handed subjects. Relevant?
• Faces not correlated to the speaker.
• Why normalized audio amplitude?
• Why no semantic emotional content?
• Why short phrases?

14
FRMI
Functional MRI or functional Magnetic Resonance
Imaging (fMRI) is a type of specialized MRI scan.
It measures the haemodynamic response related to
neural activity in the brain or spinal cord of
humans or other animals. It is one of the most
recently developed forms of neuroimaging. Since
the early 1990s, fMRI has come to dominate the
brain mapping field due to its low invasiveness,
lack of radiation exposure, and relatively wide
availability.
15
Experiment - results

• Happy voices elicited significantly more
  activation than angry voices in right anterior
  and posterior middle temporal gyrus (MTG), left
  posterior MTG and right inferior frontal gyrus.
• Attention-dependent results In contrast with
  brain regions showing a main effect of vocal
  emotion regardless of attentional focus, a
  network of brain regions including the left
  insula, left amygdala and hippocampus, and
  rostral ACC responded more to happy voices than
  to angry voices when attending to the voice, but
  showed either no difference or greater activation
  to angry voices than to happy voices when
  attending to the face.
• Do these results support the social binding
  hypothesis?

16
Results - RT and accuracy

• Why longer RT for voice case?
• Why measure RT and accuracy at all?

17
Results
How do these graphs support the stated results?
18
Facial Expressions and Emotion - Paul Ekman
http//en.wikipedia.org/wiki/Paul_Ekman

• Found facial expressions of emotion are not
  culturally determined, but universal across human
  cultures and thus biological in origin.
• How did he determine this?
• Fully true? Examples?
• How does this square with the four approaches
  Darwinian? Jamesian? Social constructivist?
  Appraisal?
• Expressions he found to be universal included
  those indicating anger, disgust, fear, joy,
  sadness, and surprise. Findings on contempt are
  less clear, though there is at least some
  preliminary evidence that this emotion and its
  expression are universally recognized.
• examples of non-universal emotions? of
  culture-dependent expression of same emotion? Of
  individual-dependent expression of same emotion?
• Facial Action Coding System (FACS) taxonomize all
  human facial expression. (Ekman and Friesen)
• 48 action descriptors, which are a contraction or
  relaxation of one or more muscles.
• http//en.wikipedia.org/wiki/Facial_Action_Coding_
  System
• http//www.cs.cmu.edu/afs/cs/project/face/www/facs
  .htm
• Microexpressions detect lying
• How does this phenomenon square with the four
  approaches?

19
FACS action descriptors
1 Inner Brow Raiser -- Frontalis (pars
medialis) 2 Outer Brow Raiser -- Frontalis (pars
lateralis) 4 Brow Lowerer -- Depressor glabellae,
Depressor supercilii, Corrugator supercilii 5
Upper Lid Raiser -- Levator palpebrae
superioris 6 Cheek Raiser -- Orbicularis oculi
(pars orbitalis) 7 Lid Tightener -- Orbicularis
oculi (pars palpebralis) 9 Nose Wrinkler --
Levator labii superioris alaeque nasi 10 Upper
Lip Raiser -- Levator labii superioris, caput
infraorbitalis 11 Nasolabial Deepener --
Zygomaticus minor 12 Lip Corner Puller --
Zygomaticus major 13 Cheek Puffer -- Levator
anguli oris (also known as Caninus) 14 Dimpler --
Buccinator 15 Lip Corner Depressor -- Depressor
anguli oris (also known as Triangularis) 16 Lower
Lip Depressor -- Depressor labii inferioris 17
Chin Raiser -- Mentalis 18 Lip Puckerer --
Incisivii labii superioris and Incisivii labii
inferioris 19 Tongue Out 20 Lip stretcher --
Risorius w/ platysma 21 Neck Tightener --
Platysma 22 Lip Funneler -- Orbicularis oris 23
Lip Tightener -- Orbicularis oris 24 Lip Pressor
-- Orbicularis oris
24 Lip Pressor -- Orbicularis oris 25 Lips part
-- Depressor labii inferioris or relaxation of
Mentalis, or Orbicularis oris 26 Jaw Drop --
Masseter, relaxed Temporalis and internal
pterygoid 27 Mouth Stretch -- Pterygoids,
Digastric 28 Lip Suck -- Orbicularis oris 29 Jaw
Thrust 30 Jaw Sideways 31 Jaw Clencher --
Masseter 32 Lip Bite 33 Cheek Blow 34 Cheek
Puff 35 Cheek Suck 36 Tongue Bulge 37 Lip Wipe 38
Nostril Dilator 39 Nostril Compressor 41 Lid
Droop 42 Slit 43 Eyes Closed -- Relaxation of
Levator palpebrae superioris Orbicularis oculi
(pars palpebralis) 44 Squint 45 Blink --
Relaxation of Levator palpebrae superioris
Orbicularis oculi (pars palpebralis) 46 Wink --
Relaxation of Levator palpebrae superioris
Orbicularis oculi (pars palpebralis)
20
Facial Action Coding System - FACS

• FACS can be used to distinguish two types of
  smiles
• insincere and voluntary Pan American smile
• contraction of zygomatic major alone
• sincere and involuntary Duchenne smile
• contraction of zygomatic major and inferior part
  of orbicularis oculi.

21
Pictures of Facial Affect (resource) Ekman and
Friesen

• 110 pictures of posed facial emotions
• 10 second display to subjects (male and female
  50 mix)
• 70 accurately identified
• Mark each example as one of
• Happy
• Sad
• Fear
• Anger
• Surprise
• Disgust

22
89

• Happy
• Sad
• Fear
• Anger
• Surprise
• Disgust

23
56

• Happy
• Sad
• Fear
• Anger
• Surprise
• Disgust

24
79

• Happy
• Sad
• Fear
• Anger
• Surprise
• Disgust

25
102

• Happy
• Sad
• Fear
• Anger
• Surprise
• Disgust

26
54

• Happy
• Sad
• Fear
• Anger
• Surprise
• Disgust

27
76

• Happy
• Sad
• Fear
• Anger
• Surprise
• Disgust

28
52

• Happy
• Sad
• Fear
• Anger
• Surprise
• Disgust

29
34

• Happy
• Sad
• Fear
• Anger
• Surprise
• Disgust

30
94

• Happy
• Sad
• Fear
• Anger
• Surprise
• Disgust

31
108

• Happy
• Sad
• Fear
• Anger
• Surprise
• Disgust

32
Tally your results!

• Should match most of the time

33
89

• Happy
• Sad
• Fear
• Anger
• Surprise
• Disgust

34
56

• Happy
• Sad
• Fear
• Anger
• Surprise
• Disgust

35
79

• Happy
• Sad
• Fear
• Anger
• Surprise
• Disgust

36
102

• Happy
• Sad
• Fear
• Anger
• Surprise
• Disgust

37
54

• Happy
• Sad
• Fear
• Anger
• Surprise
• Disgust

38
76

• Happy
• Sad
• Fear
• Anger
• Surprise
• Disgust

39
52

• Happy
• Sad
• Fear
• Anger
• Surprise
• Disgust

40
34

• Happy
• Sad
• Fear
• Anger
• Surprise
• Disgust

41
94

• Happy
• Sad
• Fear
• Anger
• Surprise
• Disgust

42
108

• Happy
• Sad
• Fear
• Anger
• Surprise
• Disgust

43
Ekman - Microexpressions

• Play video
• Impact on Darwinian, jamesian, etc?
• What is deception w/r emotion?

44
Poker Tells The Body Language of Poker (1984)
Mike Caro

• Poker players exhibit tells that reveal
  valuable information about their hands or
  intentions to bet
• Some tells are unconscious and directly related
  to the internal emotion (excitement at a good
  hand, disappointment at a bad one)
• Some tells are the result of acting the
  opposite to try to deceive the opponents.
• Tells involve facial expressions, gestures, hand
  positions, body posture, and other behaviors

45
Poker Tells instant reaction
The Body Language of Poker (1984) Mike Caro
46
Poker Tells acting the opposite
47
Poker Tells guarding the hand
48
Expressions in virtual 3D faces FaceGen
Neutral Pose
www.facegen.com
49
Expressions in virtual 3D faces FaceGen Anger
50
Expressions in virtual 3D faces FaceGen
Disgustsmile(open)
51
Expressions in virtual 3D faces FaceGen blend
other expression, phonemes, etc.
52
Next Week

• Hand in either
• A list of possible topics for your course
  project, with a paragraph on each, or (if you
  have already discussed the project with me and
  decided on one)
• A one page project specification with
• Project goal
• What you will do to achieve
• What resources you will need
• What prior work you can draw on