Art Graesser

1
Learning with Conversational Agents

• Art Graesser
• Psychology, Computer Science,
• the Institute for Intelligent Systems

2
Learning, Language, and Communication Technologies

• Tutoring
• Language discourse
• Robotic androids

3
Overview

• Why conversational agents?
• AutoTutor
• Learning
• Motivation
• Emotions
• Other learning environments with agents

4
Exploring a Sea of Animated Conversational Agents
SI Agent
Laura
SI Agent
Adele
STEVE
Carmen
AutoTutor
Leonardo
PKD Android
iMAP
BEAT
Casey
iSTART
TLTS
Spark
MRE
5
Why conversational agents?

• Social presence
• The universal interface
• Simulate and model social and conversational
  interactions
• Precise control over the language and discourse
  (unlike humans)
• Understand psychological and pedagogical
  mechanisms by building them

6
Agent Configurations

• Dialogs
• Tutor agent
• Teachable agent
• Trialogs
• Vicarious learning from agent dyad
• Tutor-agent, student-agent, human
• Quadralogs
• Human interacting with student-agent while
  observing an agent dyad

7
What is AutoTutor?

• Art Graesser (PI)
• Zhiqiang Cai
• Don Franceschetti
• Stan Franklin
• Barry Gholson
• Xiangen Hu
• David Lin
• Max Louwerse
• Andrew Olney
• Natalie Person
• Vasile Rus

• Learn by conversation in natural language
• DMello, S.K., Picard, R., Graesser, A.C.
  (2007). Toward an affect-sensitive AutoTutor.
  IEEE Intelligent Systems, 22, 53-61.
• Graesser, A.C., Jackson, G.T., McDaniel, B.
  (2007). AutoTutor holds conversations with
  learners that are responsive to their cognitive
  and emotional states. Educational Technology,
  47, 19-22.
• VanLehn, K., Graesser, A.C., Jackson, G.T.,
  Jordan, P., Olney, A., Rose, C.P. (2007). When
  are tutorial dialogues more effective than
  reading? Cognitive Science, 31, 3-62.

8
AutoTutor
9
Expectations and misconceptions in Sun Earth
problem

• EXPECTATIONS
• The sun exerts a gravitational force on the
  earth.
• The earth exerts a gravitational force on the
  sun.
• The two forces are a third-law pair.
• The magnitudes of the two forces are the same.
• MISCONCEPTIONS
• Only the larger object exerts a force.
• The force of earth on sun is less than that of
  the sun on earth.

10
When a car without headrests on the seats is
struck from behind, the passengers often suffer
neck injuries. Why do passengers get neck
injuries in this situation?
Question
Head
Parameter Controls
Describe what happens
11
What does AutoTutor do?

• Asks questions and presents problems
• Evaluates meaning of the learners answers
• Gives feedback on answers
• Display facial expressions with emotions
• Nods and gestures
• Hints
• Prompts for specific information
• Adds information that is missed
• Corrects bugs and misconceptions
• Answers student questions
• Guides the learner in interactive 3-D simulations

12
(No Transcript)
13
Expectation and Misconception-Tailored Dialog
Pervasive in AutoTutor human tutors

• Tutor asks question that requires explanatory
  reasoning
• Student answers with fragments of information,
  distributed over multiple turns
• Tutor analyzes the fragments of the explanation
• Compares to a list of expected good idea units
• Compares to a list of expected errors and
  misconceptions
• Tutor posts goals performs dialog acts to
  improve explanation
• Fills in missing expected good idea units (one at
  a time)
• Corrects expected errors misconceptions
  (immediately)
• Tutor handles periodic sub-dialogues
• Student questions
• Student meta-communicative acts
  (e.g., What did you say?)

14
5-step Tutoring Frame(Graesser Person, 1994)

• Tutor asks difficult question
• Initial student answer
• Tutor short feedback (, -, neutral)
• Tutor and student collaboratively improve the
  answer via a multi-turn dialogue
• Tutor confirmation of student understanding(often
  by asking students if they understand)

15
Dialog Moves During Steps 2-4

• Positive immediate feedback Yeah Right!
• Neutral immediate feedback Okay Uh huh
• Negative immediate feedback No Not quite
• Pump for more information What else?
• Hint What about the earths gravity?
• Prompt for specific information The earth
  exerts a gravitational force on what?
• Assert The earth exerts a gravitational force
  on the sun.
• Correct The smaller object also exerts a force.
  
• Repeat So, once again,
• Summarize So to recap,
• Answer student question

16
Managing One AutoTutor Turn

• Short feedback on the students previous turn
• Advance the dialog by one or more dialog moves
  that are connected by discourse markers
• End turn with a signal that transfers the floor
  to the student
• Question
• Prompting hand gesture
• Head/gaze signal

17

• What about Learning Gains?

18
LEARNING GAINS OF TUTORS(effect sizes)

• .42 Unskilled human tutors
• (Cohen, Kulik, Kulik, 1982)
• .80 AutoTutor (14 experiments)
• (Graesser and colleagues)
• 1.00 Intelligent tutoring systems
  
• PACT (Anderson, Corbett, Aleven, Koedinger)
• Andes, Atlas (VanLehn)
• Diagnoser (Hunt, Minstrell)
• Sherlock (Lesgold)
• (2.00??) Skilled human tutors
• (Bloom, 1987)

19
Learning Conceptual Physics(VanLehn, Graesser et
al., 2007)

• Four conditions
• Read Nothing
• Read Textbook
• AutoTutor
• Human Tutor

20
How does AutoTutor compare to comparison
conditions on tests of deep comprehension?

• 0.80 sigma compared to pretest, doing nothing,
  and reading the textbook
• 0.22 compared to reading relevant textbook
  segments
• 0.07 compared to reading succinct script
• 0.13 compared to AutoTutor delivering speech acts
  in print
• 0.08 compared to humans in computer-mediated
  conversation
• -0.20 compared to AutoTutor enhanced with
  interactive 3D simulation
• ZONE OF PROXIMAL DEVELOPMENT

21
Force equals mass times acceleration
22

• Increasing motivation during learning
• Rude tutor
• Feedback

23
Example Dialog
Tutor Okay, let's begin. Suppose a runner is
running in a straight line at constant speed, and
the runner throws a pumpkin straight up. Where
will the pumpkin land? Explain why. Student
The pumpkin will land behind the runner Tutor
Well, what can you say about the pumpkin's
vertical velocity in relation to gravity?
Student They are equal Tutor Alright, I am
sure you can get this. The direction in which a
force is being applied to the pumpkin is what?
Student Down Tutor Excellent! Is vertical.
Let's move on. After release the only force on
the pumpkin is gravity, the horizontal velocity
of the pumpkin is not affected. What can you say
about the pumpkin's motion after its release?
Student It is constantly moving forward
24
Example Dialog with RudeTutor
Tutor Okay, let's begin. Suppose a runner is
running in a straight line at constant speed, and
the runner throws a pumpkin straight up. Where
will the pumpkin land? Explain why. Student
The pumpkin will land behind the runner Tutor
What a pathetically incorrect answer! So what can
you say about the pumpkin's vertical velocity in
relation to gravity? Student They are equal
Tutor Ouch, you must have been daydreaming in
your high school football physics course. I
doubt you can get this next question but let me
give it a try. The direction in which a force is
being applied to the pumpkin is what? Student
Down Tutor Let me hold my nose and grant you
this one. Its vertical. Let's move on
25
Content versus Progress Feedback

• Content Feedback
• Qualitative information about the subject matter.
  
• Local Highlighting important words
• Global Summary of ideal answer
• Progress Feedback
• Quantitative assessment of the students
  advancement through the material being covered
• How far the student has come and how much farther
  they have to go?
• Local Progress on individual expectations.
• Global Points scored in session.

26
Screen shot with additions
27
Feedback Experiment(Tanner Jackson dissertation)

• Pretest
• 26 conceptual physics questions (counterbalanced)
• Interact with a version of AutoTutor
• All Feedback
• Progress Only
• Content Only
• No Feedback
• Posttest
• 26 conceptual physics questions (counterbalanced)

28
Adjusted Posttest Performance
29
The Tutor Held my Interest
Rated from 1Strongly Disagree to 6Strongly Agree
30
Conclusions about Feedback Study

• Content Matters Content feedback improved
  learning gains.
• Progress feedback had little to no effect on
  either learning or motivation.
• Negative relationship between liking and learning
• Content feedback provides an increase in learning
  gains, but a decrease in user satisfaction.

31

• Emotions during Learning

32
Emotions and Complex Learning

• Emotions (affective states) are inextricably
  bound to deep learning.
• Negative emotions are
• confusion ? irritation ?
• frustration ? anger ? rage
• Positive emotions are
• engagement ? flow ? delight ? excitement ?
  eureka

33
Theories of Emotions relevant to Complex Learning

• Goal Theory, with achievements and interruptions
  (Dweck, 2002 Mandler, 1976 Stein
  Hernandez,2007)
• Cognitive Disequilibrium Model and confusion
  (Graesser Olde,
  2003 Piaget, 1952)
• Yerkes-Dodson law on performance, arousal, and
  task complexity
• Flow Theory and consciousness (Csikszentmihaly,
  1990)
• Appraisal theory (Frijda, 1986 Laird, 2007
  Scherer, 2006)
• Valence-intensity model, core affective
  framework, folklore (Barrett, 2006 Russell,
  2003)
• Academic risk theory (Meyer Turner, 2006)
• Leppers analysis that promotes challenge,
  control, confidence and curiosity
• Kort, Reilly, Picard Model (2001)
• Multiple zones of affect and cognition

34
Emotions During Learning
Boredom (23)
Delight (4)
Confusion (25)
Surprise (4)
Frustration (16)
Flow (28)
35
(No Transcript)
36
Methods of Identifying Emotions

• Expert observation
• Emote aloud protocols
• After-action judgments by
• Self (learner)
• Peer
• Expert judges
• Teachers
• Sensing channels
• Dialogue history
• Speech parameters
• Facial actions
• Posture

37
Inter-judge Reliability of Trained Judges
38
Emotion Sensors and Channels
Face
Posture
Dialogue
Speech
39
Classification AccuracyOverall
40
Bayesian Diagnosticity
41
Conclusions about Automated Emotion Classification

• Dialogue, face, and posture can classify as good
  as trained judges.
• Fusion of channels typically involves redundancy.
• But superadditivity in the case of boredom
• Different emotions are expressed in different
  channels.

42
How might AutoTutor be Responsive to the
Learners Affect States

• If learner is frustrated, then AutoTutor gives a
  hint.
• If bored, then some engaging razzle dazzle
• If flow/absorbed, then lay low
• If confused, then intelligently manage optimal
  confusion

43
Emotion Sequencing(DMello, Taylor, Graesser)
44

• Beyond
• AutoTutor

45
Memphis Agent Environments
AutoTutor
iSTART
MetaTutor
ARIES
iDRIVE
HURA Advisor
Tutor Agent
46

• ARIES
• Agent
• Trialogs

47
(No Transcript)
48
ARIESAcquiring Research Investigative and
Evaluative Skills

• Scientific reasoning
• Electronic textbook with periodic multiple choice
  questions
• Diagnostic questions
• Critique experiments and newspaper clips
• Game context with aliens
• AutotTutor trialogs with tutor and student agents

49
(No Transcript)
50
(No Transcript)
51
(No Transcript)
52
Our Next Challenge

• How do we design dialogs, trialogs, and
  quadralogs of agents in a science of learning
  that is sensitive to social, motivational,
  affective, and cognitive (SMAC) mechanisms?

53

• ?