Title: Designing and Evaluating Life-like Agents as Social Actors
1Designing and Evaluating Life-like Agents as
Social Actors
- Helmut Prendinger
- Dept. of Information and Communication Eng.
- Graduate School of Information Science and
Technology - University of Tokyo
- helmut_at_miv.t.u-tokyo.ac.jp
- http//www.miv.t.u-tokyo.ac.jp/helmut/helmut.html
2Short Bioeducation, experience
- Masters in Logic (1994)
- U. of Salzburg, Austria, Dept. of Logic and
Philosophy of Science - Dynamic modal logic (completeness, decidability)
- Non-degree studies in Psychology, Linguistics,
Literature - Ph.D. in Artificial Intelligence (1998)
- U. of Salzburg, Dept. of Logic and Philosophy of
Science and Dept. of Computer Science U. of
California, Irvine - Incomplete reasoning (deduction, hypothetical
reasoning, EBL) - Post doctoral research
- U. of Tokyo, Ishizuka Lab
- JSPS Fellowship (4/1998-3/2000) Knowledge
compilation, hypothetical reasoning - Mirai Kaitaku project (since 4/2000)
Life-like characters, affective communication
with animated agents, markup languages for
animated agents, emotion recognition
3Social Computingmain objective and task
Social Computing aims to support the tendency of
humans to interact with computers as social
actors. Develop technology that reinforces human
bias towards social interaction by appropriate
feedback in order to improve the communication
between humans and computational devices.
4Social Computingrealization
Most naturally, social computing can be
realized by using life-like characters.
5Life-like Characters at Worksample applications
Tutoring, USC
Knowledge Sharing, ATR
Presentation, U. of Tokyo
Sales, DFKI
Entertainment, MIT
6Life-like Charactersdesiderata
- Life-like characters should be
- emphatic and engaging as tutors
- trustworthy as sales persona
- entertaining and consistent as actors
- stimulating as match-makers
- convincing as presenters
- (in short) social actors
- and competent
- Life-like characters should enable
- effective and natural communication with humans
7Backgroundcomputers as social actors
- Humans are biased to treat computers like real
people - Psychological studies show that people tend to
treat computers as social actors (like other
humans) - Tendency to be nicer in face-to-face
interactions, ... - Animated agents may support this tendency if they
are designed as social actors
Ref. B. Reeves and C. Nass, 1998. The Media
Equation. Cambridge University Press, Cambridge.
8Animated Agents as Social Actorsrequirements for
life-likeness
Features of Life-like Characters
Artificial Emotional Mind
Embodiment
- Synthetic bodies
- Emotional facial display
- Communicative gestures
- Posture
- Affective voice
- Affect-based response
- Personality
- Response adjusted to social context
- social role awareness
- Adaptive behavior
- social intelligence
9Outlinedesigning and evaluating life-like
characters
- The mind of life-like agents
- Emotion, social role awareness, attitude change
- Demo - Casino scenario
- Implementation and character behavior scripting
- Evaluating life-like characters
- Using biosignals to detect user emotions
- Experimental study with character-based quiz game
- Book project - character scripting languages and
applications
10SCREAM System ArchitectureSCRipting
Emotion-based Agent Minds
11Appraisal Modulethe cognitive structure of
emotions
- Evaluates external events according to their
emotional significance for the agent - Outputs emotions
- joy, distress
- happy for, sorry for
- angry at
- resent, gloat
- 22 in total
Ref. A. Ortony, G. Clore, A. Collins, 1988. The
Cognitive Structure of Emotions. Cambridge
University Press, Cambridge.
12Social Filter Moduleemotion expression
modulating factors
- Ekman and Friesens facial Display Rules (69)
- Expression and intensity of emotions is governed
by social and cultural norms - Brown and Levinson (87) on linguistic style
- Linguistic style is determined by social
variables power, distance, imposition of speech
acts
13Agent Modelcharacter profile, affect processing
- Character Profile
- static and dynamic features
- Static features
- personality traits, standards
- Dynamic features
- goals, beliefs, attitudes
- Attitudes (liking/disliking) are an important
source of emotions toward other agents - an agents attitude decides whether it has a
positive or negative emotion (toward another
agent) - happy for resent sorry for gloat
- an agents attitude changes as a result of
communication - dependent on affective interaction history
14Signed Summary Recordcomputing attitude from
affective interaction history
winning emotional states
positive emotions
negative emotions
joy (2)
distress (1)
joy (2)
distress (1)
distress (3)
hope (2)
distress (3)
angry at (2)
good mood(1)
angry at (2)
interaction history
hope (2)
happy for (2)
gloat (1)
Attitude summary value
good mood(1)
??
?
?
gloat (1)
ltemotion, intensitygt pairs
Liking if positive Disliking if negative
happy for (2)
time
Ref. A. Ortony, 1991. Value and emotion. In W.
Kessen, A. Ortony, and F. Craik (eds.),
Memories, Thoughts, and emotions Essays in the
honor of George Mandler. Hillsdale, NJ Erlbaum,
337-353.
15Updating Attitudeweighted update rule
- If a high-intensity emotion of opposite sign
occurs e.g., a liked interlocutor makes the
agent very angry - Agent ignores inconsistent new information
- Agent updates summary value by giving greater
weight to inconsistent information (primacy of
recency, Anderson 65)
?w intensity of (winning) emotion ??, ?
? ,? ?h/r historical/recency weight
?
?
?(Sitn) ??(Sitn?1) ? ?h ?w? (Sitn) ? ?r
?3 (3 ? 0.25) ? (5
? 0.75)
disliking liking h-weight angry
r-weight
- Consequence for future interaction with
interlocutor - Momentary disliking new value is active for
current situation - Essential disliking new value replaces summary
record
16Life-like Agentsmaking them act and speak
- Realization of embodiment
- 2D animation sequences
- Synthetic affective speech
- Technology
- Microsoft Agent package (installed client-side)
- JavaScript based interface in Internet Explorer
- Microsoft Agent package
- Controls to trigger character actions
- Text-to-Speech (TTS) Engine
- Voice recognition
- Multi-modal Presentation Markup Language (MPML)
- Easy-to-use XML-style authoring tool
- Interface with SCREAM system
17Life-like Characters in Interactionsome demos
Comics Scenario
Casino Scenario
Business Scenario
Life-like characters that change their attitude
during interaction
Animated agents that storify tacit corporate
knowledge
Animated comics actors engaging in developing
social relationships
18Casino Scenariolife-like characters with
changing attitude
- Animated advisor (Genie)
- Emotion, personality
- Changes attitude dependent on interaction history
with user - Dealer (James), player (Al)
- Pre-scripted behavior
Genies Character Profile Personality
specification personality_type(genie,agreeableness
,3). personality_type(genie,extraversion,2).
Social variables specification social_power(genie,
user,0,_). social_distance(genie,user,1,_).
Goals wants(genie,user_wins_game,1,_). wants(genie
,user_follows_advice,4,_). Attitude attitude(gen
ie,user,likes,1,init).
- User in the role of player of Black Jack game
Implemented with MPML and SCREAM
19Emotional Arcadvisors dominant emotions
depending on attitude
Round 1
Round 2
Round 3
Round 4
Round 5
advisor has agreeable personality
pos. attitude
pos. attitude
neg. attitude
pos. attitude
pos. attitude
ignores advice
ignores advice
ignores advice
follows advice
ignores advice
user looses
user looses
user looses
user looses
user wins
sorry for (4)
distress (4)
gloat (5)
sorry for (5)
good mood (5)
Internal intensity values
advisor has agreeable personality, is socially
slightly distant to user
sorry for (5)
distress (1)
gloat (2)
sorry for (5)
good mood (5)
Intensity values of expressed emotions
20Implementation
21Agent Scriptingsimple MPML script
lt!--Example MPML script --gt ltmpmlgt ltscene
idintroduction agentsjames,al,spaceboygt
ltseqgt ltspeak agentjamesgtDo you guys
want to play Black Jack?lt/speakgt ltspeak
agentalgtSure.lt/speakgt ltspeak
agentspaceboygtI will join too.lt/speakgt
ltpargt ltspeak agentalgtReady? You got
enough coupons?lt/speakgt ltact
agentspaceboy actapplause/gt lt/pargt
lt/seqgt lt/scenegt lt/mpmlgt
22Mind-Body Interfaceinterface SCREAM MPML
lt!--MPML script showing interface with SCREAM
--gt ltmpmlgt ltconsult target.jamesApplet.a
skResponseComAct(james,al,5)gt lttest
valueresponse25gt ltact agentjames
actpleased/gt ltspeak agentjamesgtI am
so happy to hear that.lt/speakgt lt/testgt
lttest valueresponse26gt ltact
agentjames actdecline/gt ltspeak
agentjamesgtWe can talk about that another
time.lt/speakgt lt/testgt
lt/consultgt lt/mpmlgt
23Alternative Viewsmart characters vs. smart
environments
infers I am happy
environment instructs agent be happy now
acts expresses happiness
perceives game state
tells available behaviors
behavior repository
- Sense-think-act cycle
- Classical AI approach
- Internet softbots search for information on the
web, robots explore their environment - All the intelligence is agent-side
- Annotated environments
- Shift from agent intelligence to environment
intelligence - Semantic web, ubiquitous computing, affordance
theory - Agents and environments can be developed
independently
24Outline revisiteddesigning and evaluating
life-like characters
- The mind of life-like agents
- Emotion, social role awareness, attitude change
- Demo - Casino scenario
- Implementation and character behavior scripting
- Evaluating life-like characters
- Using biosignals to detect user emotions
- Experimental study with character-based quiz game
- Book project - character scripting languages and
applications
25Affective Computingwhy should a computer
recognize user emotions?
- Human-human communication
- Based on efficient grounding mechanisms including
the ability to recognize interlocutors emotions
(frustration, confusion,) - Humans may react appropriately upon detection of
an interlocutors emotion (clarification upon
confusion) - Human-computer communication
- Computers typically lack ability to recognize
user emotions - Ignoring users emotions causes users
frustration - Recognizing and responding to users (often)
negative emotions may improve users interaction
experience
Ref. R. Picard, 1997. Affective Computing. The
MIT Press.
26Emotion Recognitionhow can computers recognize
users emotions?
- Stereotypes
- A typical visitor of a casino wants (to win)
- Communicative modalities
- Facial display (face recognition)
- Prosody (speech analysis)
- Linguistic style (NLU)
- Gestures (gesture recognition)
- Posture (posture recognition)
- Physiological data
- Biosignals
27Physiological Data AssessmentProComp unit
GSR
BVP
- EMG Electromyography
- EEG Electroencephalography
- EKG Electrocardiography
- BVP Blood Volume Pressure
- GSR Galvanic Skin Response
- Respiration
- Temperature
sensors
28Inferring Emotions from Biosignals Langs
2-dimensional emotion model
- Langs two dimensions
- Valence - positive or negative dimension of
feeling - Arousal - degree of intensity of emotional
response - Biometric measures
- Skin conductivity increases with arousal (Picard
97) - Heart rate increases with negatively valenced
emotions - Note
- introverts reach a higher level of emotional
arousal than extroverts
excited
enraged
joyful
Arousal
sad
relaxed
depressed
Valence
some named emotions in the arousal-valence space
Ref. Lang, P. 1995. The emotion probe Studies
of motivation and attention. American
Psychologist 50(5)372385.
29Experimental Studyeffects of a character-based
interface
Junichiro Mori - Experimenter Analyser
- Aim of study
- Show that a character with affective expression
may improve users experience ( reduce
frustration) of a simple quiz game - Method
- Biosignals to measure skin conductance and blood
volume pressure (objective assessment of user
experience) - Questionnaire (users subjective assessment)
- Instruction
- Addition/subtraction task (short-term memory
load) - Solve a series of 30 quizzes correctly and as
fast as possible - Frustration is deliberately caused by delay (in 6
out 30 quizzes) - Subjects
- 20 university students (all male Japanese,
approx. 24 years old) - JPY 1000.- for participation, JPY 5000.- for best
score
30Experimental Setup
31Instructionmathematical quiz game
timer
It is correct. (polite language)
sometimes delay here (6 14 sec.)
- Add 5 numbers and subtract the i-th number (i lt
5) - 1 3 8 5 4 21
- E.g. subtract the 2nd number
- Result 18
- Select the correct answer by clicking the radio
button next to the number - Then the character tells whether answer is
correct
32Two Versions of the Gameaffective vs.
non-affective (independent variables)
Affective Version Non-Affective Version
Description Description
Character expresses happiness (sorriness) for correct (wrong) answer Character shows empathy (when delay occurs) Character expresses affect both verbal and nonverbal Character does not show affective response Character ignores occurrence of delay
Hypotheses Hypotheses
Character may reduce user stress (SC) and decrease negative valence (heart rate) Character has no significant effect on user emotion (SC, heart rate)
33Character Responsesexamples of
affective/non-affective feedback
I am sorry. It is wrong. (hyper-polite language)
I am sorry for the delay. (polite language)
Character apologizes for the delay
Hanging shoulder gesture to express sorriness
non-verbally
Non-affective feedback Wrong. No
non-verbal emotion expression.
Non-affective feedback Character ignores the
occurrence of delay.
34Analyzing Physiological User Data
BVP could not be taken reliably
BVP
user response
agent response
DELAY segment
RESPONSE segment
GSR
delay ends
delay starts
Biograph Software (Thought Technologies)
35Preliminary Findings9 subjects in each version
(data of 2 subjects discarded)
- Hypothesis (design) delay induces frustration in
subjects - All 18 subjects showed significant rise of SC in
DELAY segment - Corresponds to finding in behavioral psychology
(if an individual is prohibited from attaining a
goal, the individual experiences primary
frustration)
- Hypothesis (main) affective agent behavior
reduces user frustration
Non-affective version mean ?0.05 Affective
version mean 0.2 t-test (assuming
unequal variance) t(16)?2.57 p .01
DELAY segment
RESPONSE segment
mean values sf SC (BVP could not be taken
reliably)
Preliminary evaluation suggests that an animated
character expressing emotions and empathy may
undo some of the users frustration.
36Agents Adapting to User Emotionassumes real-time
recognition of user emotions
Dynamic Decision Network (simplified)
ti1
ti
users action
evidence node
agents actions
user model
users traits
users traits
user model
learning
learning
emotional state
emotional state
U
bodily expressions
bodily expressions
QUESTION Given users state at ti,
which agent action will maximize
agents expected utility at ti1, in terms of,
e.g., users learning and emotion?
sensors
sensors
evidence nodes
37Dynamics of User Emotions
user personality
user goals
ti1
ti
succeed by myself
provide help
agreeableness
agents action
do nothing
have fun
users emotional state at ti1
reproach
joy
shame
arousal
users emotional state at ti
bodily expressions
sensors
Ref. Conati, C. 2002. Probabilistic assessment
of users emotions in educational games.
Applied Artificial Intelligence 16(7-8)555575.
down(frowning)
high
high
38Outline revisiteddesigning and evaluating
life-like characters
- The mind of life-like agents
- Emotion, social role awareness, attitude change
- Demo - Casino scenario
- Implementation and character behavior scripting
- Evaluating life-like characters
- Using biosignals to detect user emotions
- Experimental study with character-based quiz game
- Book project - character scripting languages and
applications
39Book Projectcharacter scripting languages and
applications
- Wide dissemination of life-like character
technology requires - standardized ways to represent the behavior of
agents - Book will offer state-of-the-art on XML-based
markup languages and tools - Scripting languages for face animation, body
animation and gestures, emotion expression,
synthetic speech, interaction with environment, - Characters are already used in a wide variety of
applications - Book contains some of the most successful
character-based applications - Synopsis chapters on character design
H. Prendinger, M. Ishizuka (Eds.) Life-like
Characters. Tools, Affective Functions and
Applications Springer Hardcover (in
preparation) useful as Standard/Reference
Book State-of-the-Art in Life-like Agents Course
Book for HCI, HAI, multimedia, life-like
agent applications, scripting languages,
40Conclusion
- Social Computing
- Human-computer interaction as social interaction
- Designing life-like characters as social actors
- Believability-enhancing agent features
- Emotion, personality, social role awareness,
attitude change, familarity change - Casino demo
- Future avenues smart environments (character
annotated environments) - Evaluating life-like characters as social actors
- Experimental study using users biosignals
- Life-like characters affective response may undo
some of the users negative feeling - Future avenues real-time adaptivity of agent
behavior to users emotion, decision-theoretic
approach to agent behavior