Human Interface Objectives Spring 2006

1
Human Interface ObjectivesSpring 2006

• Thomas K Harris Alexander Rudnicky

Boeing Treasure Hunt Year 2 Kickoff
2
State of the System

• We currently have a working and integrated
  multi-agent, speech and tablet human interface
  system.
• Task and object types are the bare minimum for a
  working treasure hunt system.
• GUI and speech systems are working, but not
  robust and probably not user-friendly.

3
Implementation
AGENT1
Ravenclaw Bashful
HUMAN INTERFACE
Helios
Sphinx II
Phoenix
OpTrader
Backend
Map GUI
AGENT2
Helios
Rosetta
Swift
Ravenclaw Clyde
4
Task and Object Types

• Current objects
• Home
• Robot Names
• Treasure is only implied by search
• Current tasks
• Search
• Follow
• Relative and absolute translations
• Reporting locations

5
Spoken Language

• Natural language
• A limited coverage of the domain
• Fairly good speech recognition for native
  speakers of American English
• Semi-robust to speech errors and syntactic
  variations Ward
• Sophisticated error prevention and recovery
  dialogs Bohus Rudnicky
• Flexible turn-taking (in the works) Raux

6
Map GUI

• Written in Java
• Is a Galaxy Server
• Can currently be used to
• Identify and address robots
• Register robots on a global map
• Track robots movements
• Deliver waypoints for a search task

7
Spring 2006 Objectives

• Manage human coordination as a team member
  involved in plays
• Incorporate additional modes of feedback from the
  robots
• video or pictures
• evolving maps
• Develop new grounding strategies with the
  introduction of map annotation
• Expand the interfaces set of types and functions
• expanded language
• GUI improvements

8
Coordination

• Human team members should participate in tasks,
  rather than simply direct action.
• Much of the infrastructure is already there in
  the STP framework.
• The OpTrader needs to become a bidder for tasks
  as well as an auctioneer?
• Language needs to be introduced to allow the
  human the ability to accept and reject tasks, and
  to report task status.

9
Feedback

• We would like to incorporate additional sources
  of information from the robots.
• Video streams or pictures
• Identification, classification, grounding
• Real-time mapping
• Addresses the need to search novel environments
• Shared map annotations

10
Grounding

• We wish to be able to identify possibly novel yet
  relevant objects encountered during an
  exploration.
• The human and robot team members must establish a
  common understanding of the relationship between
  named (spoken and/or glyphic) concept and their
  real-world referents.
• This common understanding will be achieved
  through dialog strategies/algorithms that we will
  begin to develop this Spring.

11
Types and Functions

• We will continue to develop an ontology of
  objects and actions for the treasure hunt domain.
  Driven by
• The working scenarios evolving complexity
• The need to support the growing capabilities of
  the system
• We will continue to improve the usability of the
  GUI and spoken language interfaces. Driven by
• Regular field meetings

12
February

• A play actually exists with a human role. The
  human is informed of its role and is assumed to
  perform that role.
• The human interface receives a copy of the global
  map.
• Names for areas, points, and objects can be
  introduced and referred to.
• Additional robots, 2 of each kind

13
March

• The human may decline to perform tasks.
• Mapping during exploration is done and shared
  with the human interface.
• Found treasure is indicated on the shared map.
• Start identifying some kind of hazard.

14
April

• The human can indicate the completion or failure
  to perform some task.
• The human can request images of areas and
  objects.
• Hazards must be verified by the human.

15
Potential Papers

• ROMAN Robot and Human Communication
• March 15
• Possible topic Human-Robot Teams Dias, Harris,
  
• Interspeech
• April 7
• Possible topic flexible turn-taking in
  multi-modal multi-participant dialogue Harris,
  Raux,
• SEMDIAL Semantics and Pragmatics of Dialogues
• May 12
• Possible topic robot grounding with speech and
  map dialogue Harris, Rudnicky
• Possible topic quantifying words into market
  actions Jones, Harris,