Computation Approaches to Emotional Speech

1
Computation Approaches to Emotional Speech

• Julia Hirschberg
• julia_at_cs.columbia.edu

2
Why Study Emotional Speech?

• Recognition
• Anger/frustration in call centers
• Confidence/uncertainty in online tutoring systems
• Hot spots in meetings
• Generation
• TTS for
• Computer games
• IVR systems
• Other applications Speaker State
• Deception, Charisma, Sleepiness, Interest
• The Love Detector (available for Skype ?)

3
Assessing Health-Related Conditions

• Assessing intoxication levels (Levit et al 01)
• Distinguishing between active and passive coping
  responses in patients with breast cancer (Zei
  Pollermann 02)
• Assessing schizophrenia (Bitouk et al 09)
• Classifying degree of autistic behavior
  (Columbia)
• Suicide notes

4
Hard Questions in Emotion Recognition

• How do we know what emotional speech is?
• Acted speech vs. natural (hand labeled) corpora
• What can we classify?
• Distinguish among multiple classic emotions
• Distinguish
• Valence is it positive or negative?
• Activation how strongly is it felt?
  (sad/despair)
• What features best predict emotions?
• What techniques best to use in classification?

5
Acted Speech LDC Emotional Speech Corpus

• happy
• sad
• angry
• confident
• frustrated
• friendly
• interested

anxious bored encouraging
6
Is Natural Emotion Different? (thanks to Liz
Shriberg)

• Annoyed
• Yes
• Late morning
• Frustrated
• Yes
• No
• No, I am
• no Manila...

• Neutral
• July 30
• Yes
• Disappointed/tired
• No
• Amused/surprised
• No

7
Major Problems for ClassificationDifferent
Valence/Different Activation
8
But.Different Valence/ Same Activation
9
Good Features Can be Hard to Find

• Useful features
• Automatically extracted pitch, intensity, rate,
  VQ
• Hand-labeled, automatically stylized pitch
  contours
• Context
• Lexical information Dictionary of Affect
• But.individual and cultural differences
• Algorithms for classification
• Machine learning (Decision trees, Support Vector
  Machines, Rule induction algorithms, HMMs,)

10
Results Different Emotions, Different Success
Rates
Emotion Baseline Accuracy
angry 69.32 77.27
confident 75.00 75.00
happy 57.39 80.11
interested 69.89 74.43
encouraging 52.27 72.73
sad 61.93 80.11
anxious 55.68 71.59
bored 66.48 78.98
friendly 59.09 73.86
frustrated 59.09 73.86
11
Open Questions

• New features and algorithms
• New types of emotion/speaker state to identify
• New ways of finding/collecting useful data
• New applications of more-or-less successful
  emotion classification
• Interspeech Paralinguistic Challenges

12
This Class

• Goals
• Learn what we know about readings and discussion
  participation
• Learn how to analyze speech, how to design a
  speech experiment, how to classify speaker states
• Try to contribute something new term project
• Practice doing research
• Syllabus
• http//www.cs.columbia.edu/julia/courses/CS6998/s
  yllabus11.htm

13
Readings and Discussion

• Weekly readings
• Everyone prepares/hands in 3 discussion questions
  on each assigned paper or website
• If you read an optional paper, submit questions
  on that as well if you want credit
• Everyone participates in class discussion
• Each week one person leads discussion on one
  paper
• Submit pdf in courseworks shared files

14
Term Project

• Everyone prepares a term project on a topic of
  their choice
• You may work alone or in teams of 2
• Deliverables
• Proposal
• Interim progress report
• Final report
• Short presentation/demo

15
Possible Topics

• Collect audio from children of different ages
  winning and losing a game and see if adults can
  distinguish those who win (happy speech) from
  those who lose (sad speech).
• Create hybrid speech stimuli from tokens uttered
  with different emotions (mixing pitch, loudness,
  duration, speaking rate,...) and see which
  features of emotional speech are most reliably
  associated with emotions.
• Detect different emotions from Cantonese and
  Mandarin speakers and compare performance of an
  automatic program to performance of human judges.
  
• Train Machine Learning algorithms on emotional
  speech corpora and see if you can improve over
  other approaches on the same corpora
• Develop an email reader that detects emotion from
  text and uses the appropriate emotional TTS
  system to read it to the use

16
Important Details

• Read the academic integrity paragraph in the
  syllabus and understand it.
• Do all the readings when they are due, turn in
  all discussion questions by noon on the day of
  class, come to every class

17
Questions?