PsychophysiologyBased Affective Communication for Implicit HumanRobot Interaction

1
Psychophysiology-Based Affective Communication
for Implicit Human-Robot Interaction
Ph.D. Defense

• Pramila Rani, 2005
• October 24, 2005

Committee Dr. Nilanjan Sarkar (Chair) Dr. Mitch
Wilkes, Dr. Richard Shiavi, Dr. Eric Vanman, and
Dr. Michael Goldfarb
2
Some Definitions

• Human-Robot Interaction
• The study of humans, robots and the ways in which
  they influence each other
• Psychophysiology
• Science of understanding the link between
  psychology and physiology
• Affective Communication
• Communication relating to, arising from, or
  influencing feelings or emotions

3
Research Focus

• This dissertation involves developing an
  intuitive affect-sensitive human-robot
  interaction framework where
• robot interacts with a human based on his/her
  probable affective state
• affective states are inferred from the human's
  physiological signals
• robot adapts its behavior in response to the
  human's affective state
• emotion

4
Outline

• Motivation
• Research Hypotheses
• Main Components
• Results
• Discussion
• Conclusion

5
Motivation

• The Robot Invasion
• There is a projected increase of 1,145 in the
  number of personal service robots in use within a
  year
• According to World robotics 2004 report, at the
  end of 2003, about 610,000 autonomous vacuum
  cleaners and lawn-mowing robots were in operation
• In 2004-2007, more than 4 million new units are
  forecasted to be added!!!
• Need for Natural and Intuitive Human-Robot
  Communication
• Unlike industrial robots, personal and
  professional service robots will need to
  communicate more naturally and spontaneously with
  people around
• Robots will be expected to be understanding,
  emphatic and intelligent

6
Motivation

• Attempt to mimic Human-Human Interaction
• More than 70 of communication is non-verbal or
  implicit
• Emotions are a significant part of communication
• 7 percent of the emotional meaning of a message
  is communicated verbally. About 38 by
  paralanguage and 55 via nonverbal channels 1
• Most Significant Channels of Implicit
  Communication in Humans
• Facial Expressions
• Vocal Intonation
• Gestures and Postures
• Physiology

1 Mehrabian, A. (1971). Silent Messages.
Wadsworth, Belmont, California
7
Motivation

• Giving Robots Emotional Intelligence
• Robots should be capable of implicit
  communication with humans
• They should detect human emotions
• They should modify their behavior to adapt to
  human emotions

8
Application Areas
Some Potential Application Areas of
Affect-Sensitive Robots
9
Research Challenges

• HumanCentric Technology
• Affective Robots- Emotion Expression and
  Perception
• Physiology-Based Affective Computing
• Challenges of Affect Recognition
• Robot Control Architecture
• Robots and Real-Time Affective Feedback

10
HumanCentric Technology

• Technological advancement so far has been more
  machine-centric
• Now, new areas of robot application are emerging
  (e.g., battlefield, space, personal assistance,
  search rescue)
• There is a need to synergistically combine
  various capabilities of robotic systems with
  human intelligence
• Most robots lack implicit channel of
  communication with humans
• There is an evident need for technological
  innovation in HRI that permits implicit
  communication between humans and robots so that
  we can begin to build affective or
  emotionally-intelligent robots.

11
Affective Robots

• Two-fold capability of an affective robot
• Perceive emotions in humans,
• Express its own emotions in a manner
  understandable to humans.
• There exist robots that can express their
  emotions using human-like facial expressions and
  affective speech
• Need for real understanding of human emotions
• detecting anxiety, frustration, engagement,
  boredom etc.
• reacting to these emotions
• For intelligent and intuitive human-robot
  interaction, it is imperative that the robot
  should be capable of perceiving human
  psychological states and adapting its behavior
  appropriately to address such a perception.

12
Physiology-Based Affective Computing

• Affect Recognition via Facial Expressions and
  Vocal Intonation
• Work under highly constrained conditions
• Dependent on gender, age, culture
• Under voluntary control, hence manipulable
• Computationally expensive and not designed for
  real-time affect recognition.
• Advantages of using Physiology for Affect
  Recognition
• Largely involuntary
• Reasonably independent of cultural, gender and
  age related biases.
• Continuously available and are not dependent on
  overt emotion expression
• Technological Advancement in Physiological
  Sensing
• Smaller, noninvasive , better sensors
• Wireless communication
• High-speed signal processing and pattern
  recognition capabilities
• Given the strong relationship between physiology
  and affective states, and the continuous and
  involuntary nature of physiological phenomena,
  advanced signal processing and machine learning
  techniques can be effectively employed to
  determine an individual's underlying affective
  states in real-time.

13
Physiology and Affect

• Current Physiology-Based Affect-Recognition
  Systems
• Vyzas et. al., Kim et. Al., Nasoz et.al, Hayakawa
  et. al.
• Limitations of Current Systems
• Distinguish between discrete affective states
• Affect elicitation usually involves audio/visual
  stimuli, or in some cases deliberate emotion
  expression
• Very few systems work online
• No systematic investigation of the relationship
  between a comprehensive set of physiological
  signals, their features and the affective states
• It would be useful to develop an online
  affect-recognition system based on features
  derived from multiple physiological signals, that
  can detect arousal of specific emotions of
  individuals while they are engaged in real-life
  task experiments.

14
Machine Learning

• The data sets are extremely constrained
• Noisy
• Small size
• Missing predictor variables
• High input dimensionality
• Possible redundancy in the input domain
• Machine learning techniques employed by other
  works
• Fuzzy Logic, Neural Network, Hidden Markov
  Models, and Bayesian learning
• Regression Tree based affect-recognition not been
  investigated till now

It would be worthwhile to empirically study the
classification performance, advantages and
disadvantages of few key machine learning
techniques when applied to the domain of affect
recognition using physiological signals.
15
Real-Time Affective Feedback

• Requirements of Robot Control Architecture
• Support channels for Explicit and Implicit
  Communication
• Interpret affective input in the task context
• Adapt Robot functionality to accommodate the
  affective states of the human
• Allows mixed-initiative interaction between the
  human and robot

Till date there is no human-robot interaction
system available in which real-time
physiology-based feedback is utilized by a robot
to interpret the underlying psychological state
of the human and modify or adapt its (robot's)
behavior as a result.
16
Research Hypotheses

• It is possible to detect distinct affective
  states and further differentiate within varying
  levels of each affective state using multiple
  indices derived from physiological signals in
  real-time
• Such a channel of implicit-communication can be
  integrated within a machine's control
  architecture to make it capable of detecting
  human affective states and responding to them
  appropriately
• Such systems are expected to improve human
  performance, while lowering the user's anxiety
  and increasing task challenge.

17
Research Components

• Theoretical
• Psychological states relevant in implicit
  communication
• Physiological signals to be monitored
• Control Architecture to accommodate implicit
  communication
• Task design for training (Phase I) and validation
  (Phase II) phases
• Computational
• Signal conditioning and processing
• Machine learning for affect recognition
• System Development
• Phase I and Phase II
• Experimental
• Phase I Phase II

18
Psychological States Selected

• Anxiety, Engagement, Boredom, Frustration, and
  Anger
• These psychological states play an important role
  in human-machine interaction.
• The affective states identified above were mainly
  chosen from the domain of negative affective
  states since they can be more closely related to
  performance and mental health of humans while
  working with machines.
• Discussion with Psychologists, review of research
  works done in psychophysiology and human factors,
  and preliminary piloting was instrumental in this
  selection.

19
Physiological Signals

• Impedance Cardiogram (ICG)
• Electrocardiogram (PCG)
• Pulseplethysmogram (PPG)
• Phonocardiogram (PCG)
• Electromyogram (EMG)
• Corrugator Supercilii
• Zygomaticus Major
• Upper Trapezius
• Peripheral Temperature
• Electrodermal Activity (EDA)

20
Impedance Cardiogram

• Relationship with Affective States
• Pre-Ejection Period (PEP) is most heavily
  influenced by sympathetic innervation of the
  heart.
• Reduced PEP is a marker of negative affect states
  specifically anxiety
• Features Extracted
• Mean PEP
• Mean IBI

ECG Signal
dZ/dt, where Z ICG Signal
21
ECG and PPG

• Relationship with Affective States
• ECG influenced by frustration, anger and anxiety
• PPG modulated by anxiety, fear of harm
• Negative affect dimension specifically associated
  with increased sympathetic arousal
• Features extracted
• Mean Interbeat Interval (IBI)
• Std. of IBI
• Sympathetic power
• Parasympathetic power
• Ratio of Sympathetic to Parasympathetic power
• Mean amp. of the peak values of the BVP signal
• Standard deviation (Std.) of the peak values of
  the BVP signal
• Mean Pulse Transit Time

ECG Signal
Pulse Transit Time
22
Phonocardiogram (Heart Sound)

• Features extracted
• Mean of the 3rd,4th, and 5th level coefficients
  of the Daubechies wavelet transform of heart
  sound signal
• Standard deviation of the 3rd,4th, and 5th level
  coefficients of the Daubechies wavelet transform
  of heart sound signal

http//www.biologymad.com/HeartExercise/HeartE3.gi
f
23
Electromyogram

• Relationship with Affective States
• Facial displays (frowns, grimaces, smiles etc.)
  of affective reactions are obvious overt
  behaviors associated with expression of emotions
• The Corrugator Supercilii muscles (responsible
  for lowering and contraction of the brows)
  considered as a measure of distress
• EMG activity in the Zygomaticus Major occurs when
  the cheek is drawn back or tightened. This
  activity has been found to increase with
  expression of pleasure.
• Features Extracted
• Mean of EMG activity
• Std. of EMG activity
• Slope. of EMG activity
• Mean Interbeat Interval of blink activity
• Mean amplitude of blink activity
• Mean and Median frequency of Corrugator,
  Zygomaticus and Trapezius

EMG Signal
classes.midlandstech.com/ Bio112/muscles20fac...
24
Electrodermal Activity

• Relationship with Affective States
• Tonic SC can be a useful index of a process
  related to energy mobilization or regulation
• SC response is produced by social stimulation
  that invokes stress, tension, anxiety or
  cognitive reactions.
• Significantly smaller values of SC response
  associated with neutral states than with sadness,
  anger, fear, disgust, and amusement
• Features Extracted
• Mean tonic activity level
• Slope of tonic activity
• Mean amplitude of skin conductance response
  (phasic activity)
• Maximum amplitude of skin conductance response
• Rate of phasic activity

Typical Skin Conductance Response
Skin Conductance Signal
25
Peripheral Temperature

• Relationship with Affective States
• Peripheral temperature is an indirect index of
  peripheral vasoconstriction.
• Skin temperature can vary by 1-2 degrees
  Fahrenheit depending upon the emotional state of
  a person
• In the flight/fight stress response peripheral
  nervous system shunts the blood away from ones
  extremities and into the brain, heart and lungs,
  to aid in optimum performance in order to
  eliminate the acute stress.
• Features Extracted
• Mean
• Slope, and
• Standard deviation of temperature recording

26
Mixed-Initiative Interaction
27
Task Design (Phase I)

• Anagram
• The anagram solving task has been previously
  employed to explore relationships between both
  electrodermal and cardiovascular activity with
  mental anxiety.
• In this task, emotional responses were
  manipulated by presenting the participant with
  anagrams of varying difficulty levels, as
  established through pilot work.
• Affective states such as engagement, boredom,
  anger, frustration and anxiety were induced by
  manipulating the difficulty of anagrams
• All these conditions were well tested during the
  task design and development stage and piloting.

28
Task Design (Phase I)

• Pong
• Pong game has been used in the past by
  researchers to study anxiety, performance, and
  gender differences
• Various parameters of the game were manipulated
  to elicit the required affective responses. These
  included
• ball speed and size,
• paddle speed and size,
• sluggish or over-responsive keyboard,
• random keyboard response.
• The relative difficulties of various trial
  configurations were established through pilot
  work.

29
Task Design (Phase II)

• Pong
• Interactive Pong
• Real-time feedback regarding player anxiety
  provided to machine
• Performance-based game adaptation
• Anxiety-based game adaptation

30
Task Design (Phase II)

• Robot Basketball Game
• A basketball hoop attached to a robotic
  manipulator
• The difficulty of the task varied by controlling
  parameters such as robot arm speed and direction
  of motion.
• Performance Based Game Adaptation
• Anxiety-Based Game Adaptation

31
Computational

• Signal conditioning and processing
• Algorithms for artifact-rejection, adaptive
  thresholding, signal conditioning and
  feature-extraction for various signals
• Wavelet transform, Fourier transform and
  statistical analysis and were extensively used
  in order to perform signal processing
• Machine learning for affect recognition
• A systematic comparison of the strengths and
  weaknesses of four machine learning methods -
  K-Nearest Neighbor, Regression Tree, Bayesian
  Network and Support Vector Machine was performed
  

SVM analysis was done by Mr. Changchun Liu
32
Signal Processing

• Adaptive Thresholding
• An adaptive or continuously changing threshold
  value was used to determine whether candidate for
  peaks qualified to be valid peaks.
• The need for this arose from the fact that for
  signals such as PPG (pulseplethysmogram), the
  average peak amplitude shows a large deviation
  over a given period of time.

33
Adaptive Thresholding

• A moving window was used to determine the
  threshold
• The peaks in a given window were weighed so that
  the most recent peaks had higher weight values
  than the older peaks.
• Where C Scaling factor, wi weight for peak
  (k-i), and Ak-I is the amplitude of peak (k-i).
  The values of wi were such that smaller the
  value of i, greater the value of wi. The values
  of wi, k, and C were determined by Monte Carlo
  Simulations.

34
Fourier Transform

• Powerful signal analysis technique to study the
  frequency components of the physiological signals
• Time series waveforms do not capture
  frequency-related variabilities easily
• Frequency domain analysis has proven valuable in
  linking physiological abnormalities and
  variability to specific frequency bands.
• Fourier Transform of the interbeat interval
  (IBI) derived from ECG

35
Wavelet Transform

• Signals such as PCG are complex and highly
  non-stationary
• FFT analysis has limited analysis capabilities
  for such signals
• Wavelets can be a very powerful tool for
  performing time-frequency analysis of
  non-stationary signals such as heart sounds.
• It allows simultaneous localization in time and
  frequency domain
• It has inbuilt noise filtering

36
Wavelet Transform (cont)
37
Filtering Artifact Rejection

• Filtering of the signal is required to focus on a
  narrow band of electrical energy that is of
  interest
• It removes noise and artifact such as that
  commonly found at 50 or 60 Hz (emitted into the
  recording
• environment by devices such as florescent lights
  , computer power supplies)
• Other elements that need to be filtered out are
  the artifacts caused by limb motions
• Band pass, low pass and high pass were employed
  depending upon the frequency of interest

EMG Signal Filtered in Different Ways
http//www.thoughttechnology.com/pdf/MAR656-0020T
ech20Note20024.pdf
38
Machine learning for affect recognition

• KNN
• description
• BNT
• description
• RT
• description
• SVM
• description

39
System Development

• Phase I System Set-up

40
System Development
Sensors
Task in Progress
Room 1- Experimental Set-up
Sensors
Wearable Sensors
Baselining
41
System Development

• Phase II System Set-up for Pong

42
System Development

• Phase II System Set-up for Robot Basketball

43
Experimental

• Model building and Verification

Model Verification and Analysis
Phase I Experiments
Data Analysis for Model Building
Phase II Experiments
Models for Affective States
(I will make a better figure)
44
Sliding Window Technique

• For New Participants

45
Experimental

• Phase I
• Fifteen participants took part in a 2-month study
  during which each person completed six sessions
  (three sessions of playing Pong and three
  sessions of solving anagrams)
• Phase II
• In the verification experiments for Pong nine
  participants who also took part in Phase I
  experiments volunteered
• Fifteen participants took part in the robot-based
  basketball game. None of them had participated
  in Phase I and four of them were new.

46
Pong

• Experiment Procedure

47
Robot Basketball

• Experiment Procedure

48
Results

• Relationship between physiological signals and
  affective states
• Accuracy of Regression Tree based affect
  recognition
• Comparison between Regression Tree, KNN, Bayesian
  Networks and Support Vector Machines
• Results of Pong game and robot-based basketball
  game with real-time affective feedback

49
Results

• Relationship Between Physiological Signals and
  Affective States

• High Correlations found between physiological
  signals and affective states
• Extent of correlation was different for
  different affective states

50
Results

• Physiological Signals and Affective States

• Person Stereotypy
• The highly correlated physiological indices vary
  from individual to individual

51
Results

• Comparison between Machine Learning Techniques
• A systematic comparison of four machine learning
  methods - K-Nearest Neighbor, Regression Tree,
  Bayesian Network and Support Vector Machine was
  performed
• All the four methods performed competitively
• Support Vector Machine yielded the best
  classification accuracy
• Regression Tree gave the next best classification
  accuracy
• Regression Tree was the most space and time
  efficient method.

52
Results

• Regression Tree-Based Affect Recognition

Classification accuracy
Distinction wrt anxiety
53
Results

• Pong Game with Real-Time Affect Recognition

Increase in Performance
Increase in Challenge
(More results will be added)
Decrease in Anxiety
54
Results

• BB Game with Real-Time Affect Recognition
• Anxiety
• Challenge
• Performance
• Satisfaction Index

55
Discussion
56
Conclusion
57
Future Work
58
Publications

• Dissertation Related Publications
• Rani, P, Sims, J, Brackin, R, and N. Sarkar,
  Online Stress Detection using Psychophysiological
  Signal for Implicit Human-Robot Cooperation, in
  Robotica, Vol. 20, No. 6, pp. 673-686, 2002.
• Rani, P., Sarkar, N., Smith, C., and L. Kirby,
  Anxiety Detecting Robotic Systems Towards
  Implicit Human-Robot Collaboration, in Robotica,
  Vol. 22, No. 1, pp. 85-95, 2004.
• (Under Review) Rani, P., Sarkar, N., Smith, C.,
  A., Adams, J., A., Affective Communication for
  Implicit Human-Machine Interaction, IEEE
  Transactions on Systems, Man, and Cybernetics.
• (Under Review) Rani, P., Sarkar, N., An Approach
  to Human-Robot Interaction Using Affective Cues,
  IEEE Transactions on Robotics.
• Rani, P., Sarkar, N., "Operator Engagement
  Detection and Robot Behavior Adaptation in
  Human-Robot Interaction", IEEE International
  Conference on Robotics and Automation, April
  2005, Barcelona, Spain.
• Rani, P., Sarkar, N., Smith, C., "Affect-Sensitive
  Human-Robot Cooperation Theory and
  Experiments", IEEE International Conference on
  Robotics and Automation, pp 2382-2387, Taiwan,
  September 2003.
• Rani, P., Sarkar, N., "Maintaining Optimal
  Challenge in Computer Games Through Real-Time
  Physiological Feedback ", HCI International, July
  2005, Las Vegas, USA.
• Rani, P., Sarkar, N., Smith, Anxiety Detection
  for Implicit Human-Robot Collaboration, IEEE
  International Conference on Systems, Man
  Cybernetics, Washington D.C., pp 4896-4903,
  October 2003.
• Rani, P., Sarkar, N., "Emotion-Sensitive Robots-
  A New Paradigm for Human-Robot Interaction",
  IEEE-RAS/RSJ International Conference on Humanoid
  Robots (Humanoids 2004), November 2004, Los
  Angeles, USA
• Adams, J, Rani, P, Sarkar, N, Mixed Initiative
  Interaction and Robotic Systems, Workshop on
  Supervisory Control of Learning and
  Adaptive Systems, Nineteenth National Conference
  on Artificial Intelligence (AAAI-04), San Jose,
  CA, July, 2004.
• (Submitted) Liu, C, Rani, P., Sarkar, N.,
  "Comparison of Machine Learning Techniques for
  Affect Detection in Human Robot Interaction,"
  IEEE/RSJ International Conference on Intelligent
  Robots and Systems, August 2005, Canada.
• (Submitted), Rani, P., Sarkar, N., Making Robots
  Emotion-Sensitive - Preliminary Experiments and
  Results,, ROMAN 2005

59
Publications

• Others
• Rani, P., Sarkar, M., Brackin, R., Sarkar, N.,
  "Semi-Autonomous Human-Robot Interaction Using
  EMG-Based Modified Morse Code", Workshop on
  Multi-point interaction in Robotics and Virtual
  Reality, IEEE International Conference on
  Robotics and Automation, New Orleans, USA, April
  2004. (To appear as a chapter in Springer-Verlag
  Tract on Advanced Robotics)
• Erol, D, Rani P, Brackin, RL, Sarkar, MS, and
  Sarkar, N, "Robotic Aid to People with Disability
  By Means of an Innovative Communication
  Paradigm", 9th Mechatronics Forum International
  Conference (Mechatronics 2004) , September 2004,
  Ankara, Turkey
• Chai, P, Rani, P, and N. Sarkar, An innovative
  high-level human-robot interactions for disabled
  persons, IEEE International Conference on
  Robotics and Automation, New Orleans, USA, April
  2004.
• (Submitted), Rani, P., Sarkar, M., EMG-Based
  High Level Human-Robot Interaction System for
  People with Disability, ROMAN 2005

60
Video
61
Special Thanks to my Lab Members Changchun,
Duygun, Bibhrajit, Vishnu
62
Questions?