Spoken Dialog Management for an Astronaut

1
S.D.M.A.P.A.

• Spoken Dialog Management for an Astronauts
  Procedure Assistant
• Presented by Dan Bohus (dbohus_at_cs.cmu.edu)
• Collaborators Gregory Aist, RIALIST Group
• NASA Ames/RIACS Summer Student Research Program
• August 2002

2
RIALIST Group _at_ NASA Ames

• Location NASA Ames / Moffett Field / Mountain
  View, CA (CMU West)
• People
• Jim Hieronymous
• John Dowding
• Beth-Ann Hockey
• Greg Aist
• 7 summer students
• Projects
• Personal Satellite Assistant
• Mobile Robots
• Intelligent Procedure Assistant

3
Motivation

• Crew members on International Space Station (ISS)
  have to perform a wide variety of procedural
  tasks
• ISS maintenance and repair
• Operations (i.e. EVA Suit Checkout)
• Setup and running scientific experiments
• Medical Exams

4
Typical setup

• Paper documentation
• 2 or more astronauts involved one reads, one
  executes the procedure

5
Proposed alternative

• Build a spoken dialog system for procedural task
  guidance and support

• Create more crew-time!
• Used also for ground training
• Potential for increased safety

6
Outline

• Motivation
• The Intelligent Procedure Assistant
• A Dialog Management Component for IPA
• Demo
• Future work

7
The Intelligent Procedure Assistant

• Multi-modal spoken dialog system providing crew
  support and training for the execution of
  procedural tasks
• Guide astronauts through procedural tasks
• Provide information access (diagrams, docs)
• Act as a training tool
• Facilitate task delegation
• Provide support for voice annotations, intercom

8
IPA Version 0 June 2002

• RIALIST IPA Version 0
• Load procedure
• Next
• Previous
• Drill-down
• Describe remaining steps
• Increase volume
• Decrease volume
• Say again

9
Outline

• Motivation
• The Intelligent Procedure Assistant
• A Dialog Management Component for the IPA
• Demo
• Future work

10
Contributions Dialog Management Resources
U
D
SI
SpeechRecognition
Semanticizer(parsing)
Dialog Management
Resources/Agents
LanguageGeneration
Synthesis
SO
NL
S
11
Contributions Dialog Management Resources
1
2
3
12
1. Resources

• Procedures represented in XML (DTD, XSLT)
• Title, Description.
• Steps Basic Steps, Conditional Steps, Messages,
  Images
• Messages Notes, Cautions
• Images Sub-images
• Water Sampling Procedure
• 28 steps, 150 sub-steps, 18 conditional steps,
  1 image with 5 sub-images
• HTML version (using XSLT)

13
2. Dialog Task Development

• In RavenClaw, a dialog task is implemented as a
  hierarchy of dialog agents (C classes)

IPA
Welcome
LoadProcedure
Do Procedures
DoProcedure1
DoProcedure2
Intro
Closing
14
2. Dialog Task Development (cont)

• 52 dialog task agents (types) were implemented
• Loading procedures by name
• Guidance
• Step-by-step execution
• Drill-down
• Branching
• Presenting notes / cautions / images
• Information access
• Talking on/about images, objects
• Navigational behavior
• History / Trace through procedure
• Next, Previous step/substep
• Start over

15
3. Conversational Skills Enhancements

• Timeouts
• System reacts after a timeout period
• User Barge-in
• Requests for repeat
• Requests for help
• generic help!
• what can I say ?
• Context establishment
• where are we ?

16
Outline

• Motivation
• The Intelligent Procedure Assistant
• A Dialog Management Component for IPA
• Demo
• Future work

17
Demo
Guidance
Information Access
Volume control
18
Outline

• Motivation
• The Intelligent Procedure Assistant
• A Dialog Management Component for IPA
• Demo
• Future work
• IPA System (RIALIST Group)
• Advances in dialog management

19
Future work for IPA system (RIALIST Group)

• IPA system functionality goals
• Guide astronauts through procedural tasks
• Provide information access
• Act as a training tool
• Facilitate task delegation
• Provide support for voice annotations, intercom
• Iterative improvement process
• Other features, including those suggested by JSC
  training personnel and astronauts

20
Timeline

• Reliable prototype by the end of September
• Guidance/Information Access
• First fieldable system by March 2003

21
Summer _at_ NASA is cool !
22
Being back is also cool !

• 2 new systems to be developed based on the
  Communicator SDS setup / RavenClaw DM
• RoomLine system
• Provide conference room reservation services
  within SCS
• Ridy Lie
• BusLine system
• Provide access to bus line information
• Project Lets go! (Maxine/Alan)?
• Brian Langner, Antoine Raux
• Hope you can call them sometime before Christmas
  !!!