error handling Higgins Galatea

1
error handling Higgins / Galatea

• Dialogs on Dialogs Group
• July 2005

2
work by

• Gabriel Skantzeph.d. studentKTH, Stockholm

I am doing research on spoken dialogue systems.
More specifically, I am interested in studying
miscommunication and error handling, but also in
the representation and modelling of utterances
and dialogue, as well as conducting experiments
with users.

• and co-authors J. Edlund, D. House, R. Carlson

3
3 papers

• Higgins
• Higgins a spoken dialogue system for
  investigating error handling techniques, Edlund,
  Skantze, Carlson 2004
• Galatea
• GALATEA A Discourse Modeller Supporting
  Concept-Level Error Handling in Spoken Dialog
  Systems, Skantze 2005
• Prosody Clarifications
• The Effects of Prosodic Features on the
  Interpretation of Clarification Ellipses, Edlund,
  House, Skantze 2004

4
1st paper

• Higgins
• Higgins a spoken dialogue system for
  investigating error handling techniques, Edlund,
  Skantze, Carlson 2004
• Galatea
• GALATEA A Discourse Modeller Supporting
  Concept-Level Error Handling in Spoken Dialog
  Systems, Skantze 2005
• Prosody Clarifications
• The Effects of Prosodic Features on the
  Interpretation of Clarification Ellipses, Edlund,
  House, Skantze 2004

5
Higgins

• practical goal of Higgins project
• build a collaborative dialog system in which
  error handling ideas can be tested empirically
• error handling issues, plus
• incremental dialogue processing
• on-line prosodic feature extraction
• robust interpretation
• flexible generation and output

6
domain

• pedestrian city navigation and guiding
• user gives system a destination
• system guides user by giving verbal instructions
• complex
• large variety of error types
• semantic structures can be quite complex
• reference resolution
• domain can be extended even further

7
architecture

• follow-up from Adapt
• everything is XML
• domain objects
• utterance semantics
• discourse model
• database content
• system output (before surface)
• 3D city model

8
research issues

• early detection and correction
• late detection
• incrementality
• error recovery

9
early detection and correction

• KTH LVCSR output likely to contain errors ?
• robust interpretation Pickering
• some syntactic analysis is needed
• e.g. relations between objects
• but handles insertions and non-agreement phrases
• humans - good at early detection (woz)

10
late detection and correction

• discourse modeller (GALATEA)
• joins several results from Pickering into a
  discourse model
• adds grounding information
• can be manipulated later
• remove concepts which turn out not to be grounded

11
incrementality

• end-pointers cause trouble
• even more so in this domain

better
12
incrementality 2

• all components support incremental processing
• several issues
• when to barge in? (semantic content and prosody)
• longer-than-utterance units interpreter or
  dialog manager?
• rapid and unobtrusive feedback challenge for
  synthesis

13
error recovery

• signaling non-understandings
• decreased experience of task success
• slower recovery
• ask other task-related question

14
2nd paper

• Higgins
• Higgins a spoken dialogue system for
  investigating error handling techniques, Edlund,
  Skantze, Carlson 2004
• Galatea
• GALATEA A Discourse Modeller Supporting
  Concept-Level Error Handling in Spoken Dialog
  Systems, Skantze 2005
• Prosody Clarifications
• The Effects of Prosodic Features on the
  Interpretation of Clarification Ellipses, Edlund,
  House, Skantze 2004

15
GALATEA

• a discourse modeller for conversational spoken
  dialog systems
• builds a discourse model (what has been said
  during the discourse)
• resolution of ellipses anaphora
• tracks the grounding status
• who said what when (plus confidence information)
• can be used for concept-level error handling

16
should do grounding at concept level

• explicit and implicit verification on whole
  utterance can be tedious and unnatural
• 45 of clarifications in BNC are fragmentary /
  elliptical

17
should do grounding at concept level

• Traum (1994) utterance level computational
  model of grounding
• Larsson (2002) issue-level computational model
  of grounding in Issue-Based DM
• Rieser (2004), Schlangen (2004) systems capable
  of fragmentary clarification requests, but models
  do not handle user reactions
• systems should keep grounding information at the
  concept level
• like RavenClaw? ?

18
semantic representation

• rooted unordered trees of semantic concepts
• nodes attr-value pairs, objects, relations,
  properties

19
semantic representation

• enhanced with meta-information
• confidence
• communicative acts
• info is new / given

20
ellipsis resolution

• transforms ellipsis into full propositions
• rule based
• 10 rules
• domain-specific

21
anaphora resolution

• keeps a list of entities (talked about)
• assigns ids
• when given entities are added to the discourse,
  look up the antecedent
• if found, unification (and move to the top of the
  entity list)
• unification also allows entities to be referred
  to in new ways
• how does this fare and compare?

22
grounding status

• who added the concept?
• in which turn?
• how confident?
• may be used by the action manager
• for instance remove all items with high grounding
  status when referring to an entity

23
updating grounding status
24
late error detection

• discover inconsistencies in discourse model
• look at grounding status to see where error may
  be
• concept can be removed

25
future

• methods for automatic tuning of strategy
  selection
• extend to track confidence and grounding status
  at different levels
• evaluate
• how people respond to incorrect confirmations,
  and how can that information be used to update
  grounding status
• error recovery after non-understandings
• other domains

26
3rd paper

• Higgins
• Higgins a spoken dialogue system for
  investigating error handling techniques, Edlund,
  Skantze, Carlson 2004
• Galatea
• GALATEA A Discourse Modeller Supporting
  Concept-Level Error Handling in Spoken Dialog
  Systems, Skantze 2005
• Prosody Clarifications
• The Effects of Prosodic Features on the
  Interpretation of Clarification Ellipses, Edlund,
  House, Skantze 2004

27
prosody in clarifications

• effects of prosodic features on interpretation of
  elliptical clarifications
• U Further ahead on the right I see a red
  building
• S Red (?)
• vary prosodic features
• study impact on users understanding of the
  systems intention

28
motivation

• long (whole utterance) confirmations are not good
• tedious, unnatural
• BNC corpus 45 of clarifications are elliptical
• short confirmations
• make dialog more efficient by focusing on the
  actual problematic fragments
• however
• interpretation depends on context and prosody

29
3 readings

• U Further ahead on the right I see a red
  building
• S Red (?)
• Ok, red all positive
• Do you really mean red? What do you mean by red?
  positive perception, negative understanding
• Did you say red? positive contact, negative
  perception

30
stimuli

• 3 test words red, blue, yellow
• di-phone voice (MBROLA)
• manipulated
• peak position mid, early, late / 100ms
• peak height 130Hz / 160 Hz
• vowel duration normal, long / 100ms

31
subjects design

• 8 speakers 2f / 6m, 2nn / 6n
• introduced to Higgins
• listen to all 42 (only once) random order
• 3 options
• Okay, X
• Did you really mean X?
• Did you say X?

32
results

• no effects for
• color, subject, duration
• significant effects for
• peak position, peak height, their interaction

33
results

• Statement early, low peak
• Question late, high peak
• Clear division between did you mean and did
  you say

34
food for thought

• how about English?
• red
• red?
• red!?
• how many ways can you say it?

35
conclusion

• strong relationship between intonation and
  meaning
• statement early, low peak
• question late, high peak
• clear division between did you mean and did
  you say

36
the end