AutoTutor: Integrating Learning with Emotions

1
AutoTutor Integrating Learning with Emotions

• Art Graesser, Bethany McDaniel and Sidney DMello

2
Overview

• Project goals (Art)
• Methods of data collection (Bethany)
• Affective states during learning (Sidney)
• Occurrence of affective states
• Inter-judge agreement
• Detection of Ekmans facial actions (Bethany)
• Dialogue acts and emotions (Sidney)

3
AutoTutor Highlights

• Art Graesser (PI)
• Zhiqiang Cai
• Stan Franklin
• Barry Gholson
• Max Louwerse
• Natalie Person
• Roz Picard (MIT)
• Vasile Rus
• Patrick Chipman
• Scotty Craig
• Sidney DMello
• Tanner Jackson
• Brandon King
• Bethany McDaniel
• Jeremiah Sullins
• Kristy Tapp
• Adam Wanderman
• Amy Witherspoon

• Learn by conversation in natural language
• Subject matters
• Computer literacy
• Conceptual physics
• Critical thinking (now)
• Tracks and adapts to cognition, emotions, and
  abilities of learner.
• Improves learning nearly a letter grade compared
  to reading textbooks (.8
  sigma effect size)

4
AutoTutor 1998
(Graesser, Wiemer-Hastings, Wiemer-Hastings,
Kreuz, 1999)
5
(No Transcript)
6
Expectations and misconceptions in Sun Earth
problem

• EXPECTATIONS
• The sun exerts a gravitational force on the
  earth.
• The earth exerts a gravitational force on the
  sun.
• The two forces are a third-law pair.
• The magnitudes of the two forces are the same.
• MISCONCEPTIONS
• Only the larger object exerts a force.
• The force of earth on sun may be less than that
  of sun on earth.

7
Expectation and Misconception-Tailored Dialog
(Pervasive in AutoTutor unskilled human tutors)

• Tutor asks question that requires explanatory
  reasoning
• Student answers with fragments of information,
  distributed over multiple turns
• Tutor analyzes the fragments of the explanation
• Compares to a list of expectations (good
  sentences)
• Compares to a list of misconceptions (bad
  answers)
• Tutor posts goals performs dialog acts (hints,
  prompts) to improve explanation
• Fills in missing expectations (one at a time)
• Corrects expected misconceptions (immediately)
• Tutor handles periodic sub-dialogues
• Student questions
• Student meta-communicative acts
  (e.g., What did you say?)

8
Dialog Moves

• Positive immediate feedback Yeah Right!
• Neutral immediate feedback Okay Uh huh
• Negative immediate feedback No Not quite
• Pump for more information What else?
• Hint What about the earths gravity?
• Prompt for specific information The earth
  exerts a gravitational force on what?
• Assert The earth exerts a gravitational force
  on the sun.
• Correct The smaller object also exerts a force.
  
• Repeat So, once again,
• Summarize So to recap,
• Answer student question

9
Hint-Prompt-Assertion Cycles to Cover One
Expectation
Hint

• Cycle fleshes out one expectation at a time
• Exit cycle when
• LSA-cosine(S, E ) gt T
• S student input
• E expectation
• T threshold

Prompt
Assertion
Hint
Prompt
Assertion
10
How might AutoTutor be Responsive to the
Learners Affect States

• If learner is frustrated, then AutoTutor gives a
  hint.
• If bored, then some engaging razzle dazzle
• If flow/absorbed, then lay low
• If confused, then intelligently manage optimal
  confusion

11
AutoTutor tracking learners emotions
(Memphis team MIT team with Roz Picard)
Confusion
Excitement
Boredom
Flow
Eureka
Frustration
12

• Methods of data collection

13
Visual IBM Blue eyes camera
Posture Body Pressure Measurement System
AutoTutor
Pressure force sensitive mouse and keyboard
AutoTutor text dialog
14
Facial ExpressionsThe IBM Blue Eyes Camera
Red Eye Effect
IBM Blue Eyes Camera
Eyebrow Templates
15
Posture PatternsThe Body Pressure Measurement
System
16
Gold Standard Study

• Session one
• Participants (N28) interact with AutoTutor
• Collect data with sensors
• BPMS
• Blue eyes camera
• AutoTutor logs
• Participants view their video and give ratings
• Session two (one week later)
• Participants view another participants video and
  give affect indications every 20 seconds
• Expert judges (N2) give affect ratings
• Learning measures - 32 multiple choice questions
  (pretest-posttest)

17

Reliability of Emotion Judgments
18
Interrater Reliability Judges

• F(5,135) 38.94, MSe .046, p lt .01
• Trained Judges gt All Raters,
• Self- Peer lt All Raters
• Peer-Judge 1 gt Peer-Judge 2
• Peer-Judge 1 gt Self-Judge 1

19
Interrater ReliabilityEmotions
F(6,162) 23.13, MSe .058, p lt .01 Delight gt
Confusion gt Boredom gt (Flow Frustration
Neutral Surprise)
20
Interrater ReliabilityJudgments Type
21
Number of Observations
22
Interrater ReliabilityTrained Judge Recoding
Overall Kappa .49 N
1133
23
Comparison with other Affect Judgment Efforts
24
Emotion Distribution
25
Conclusions

• Trained judges who are experienced in coding
  facial actions provide affective judgments that
  are more reliable and that match the learners
  self reports than the judgments of untrained
  peers.
• The judgments by peers have very little
  correspondence to the self reports of learners.
• Third, an emotion labeling task is more difficult
  if judges are asked to make emotion ratings at
  regularly polled timestamps, when compared to
  spontaneous judgments.
• Different types of emotions are elicited at
  different judgment points.

26

• Detection of Ekmans Facial Actions

27
Detection of Ekmans Facial Actions

• Facial Action Coding System
• (Ekman, 2003 Ekman Friesen, 1978)
• Each Action Unit (AU) represents muscular
  activity which produces a change in facial
  appearance

Neutral
AU 1 Inner Brow Raise
AU 7 Lid Tightener
28
Discriminability of Action Units

• How well each AU can be detected
• Motion How detectable the change is from the
  Neutral position.
• Edge How clearly defined a line or object on the
  face is in relation to the surrounding area.
• Texture The level of graininess for the general
  area. The degree of variation of the intensity of
  the surface, quantifying properties such as
  smoothness, coarseness and regularity.

29

Grading Discriminability

• 2 expert judges
• trained on the Facial Action Coding System
• Rate Motion, Edge, and Texture on a 1-6 scale
• 1Very Difficult, 6Very Easy
• Averaged the 3 scores for each expert
• Score between 3-18
• 3Very Difficult to detect, 18 Very Easy to
  detect

30
Probabilities of the occurrence of AUs across
emotions and discriminability
31
Probabilities of the occurrence of AUs across
emotions and discriminability
32
Probabilities of the occurrence of AUs across
emotions and discriminability
33
Probabilities of the occurrence of AUs across
emotions and discriminability
34
Facial Action Coding System
Neutral
AU 1 Inner Brow Raise
35
Facial Action Coding System
Neutral
AU 2 Outer Brow Raise
36
Facial Action Coding System
Neutral
AU 4 Brow Lower
37
Facial Action Coding System
Neutral
AU 7 Lid Tightener
38
Facial Action Coding System
Neutral
AU 12 Lip corner puller
39
Facial Action Coding System
Neutral
AU 14 Dimpler
40

Predicting Emotions from Conversational Cues
41
Auto Tutors Text Dialog
Student answers
LSA matches
42
AUTO TUTORS CONVERSATIONAL CHANNELS
TEMPORAL
VERBOSITY
ANS. QUALITY
ADVANCER
FEEDBACK
LOCAL GOOD LOCAL BAD GLOBAL GOOD GLOBAL BAD DEL
LCL GOOD DEL LCL BAD DEL GLG GOOD DEL GLG BAD
PUMP HINT PROMPT CORRECTION ASSERTION SUMMARY
NEGATIVE NEUTRAL NEG NEUTRAL NEUTRAL POS POSITIVE
REAL TIME SUBTOPIC TURN RESPONSE
WORDS CHARS SPEECH ACT
DIRECTNESS
FEEDBACK
43
Collinearity Analysis
44
Multiple Regression Analyses
p lt .10 p lt .05
45
Predictors of Affective States
46
How might AutoTutor be Responsive to the
Learners Affect States

• If learner is frustrated, then AutoTutor gives a
  hint.
• If bored, then some engaging razzle dazzle
• If flow/absorbed, then lay low
• If confused, then intelligently manage optimal
  confusion