Title: Faculty: Manuela Veloso, Anthony Stentz, Alex Rudnicky
1Dynamic Human-Robot Teams Engaged in Complex
Adversarial Tasks Using Language-Based
Communication
- Faculty Manuela Veloso, Anthony Stentz, Alex
Rudnicky - Brett Browning, M. Bernardine Dias
- Students Thomas Harris, Brenna Argall, Gil
Jones - Satanjeev Banerjee
Carnegie Mellon School of Computer
Science Boeing Human-Robot Teams Project
2Project Goals
- Treasure hunt with two or more teams of humans
and robots competing to locate target objects as
they explore an unknown space - Research Goals/Questions
- Specify team members roles and capabilities to
perform tasks - Rapidly form ad-hoc heterogeneous teams of robots
and humans - Robots and humans executing synchronized action
as a team, while communicating via speech - Improve team performance with experience
3Alignment with Boeings Objectives
- Our project develops component technologies
relevant to - FCS Force multiplication for human-robot teams
- NASA Code T Emphasis on robots assisting and
augmenting humans in complex tasks - Space activities (Other than Code T)
- Automating launch sites
- Automating escape systems
- Automating maintenance inspections/repair
- Ice inspections
- Aircraft maintenance
- Automating maintenance inspections/repair
- Ice inspections
- Automated baggage handlers
In collaboration with Phillip Koons, Boeing
4STP Overview
- Skills for low-level control, Tactics for
single robot, Plays for teamwork
APPLICABLE offense DONE aborted !offense ROLE 1
pass 3 mark best_opponent ROLE 2 block ROLE
3 pos_for_pass R B 1000 0 receive_pass
shoot A ROLE 4 defend_lane
5STP Implementation
- Play selection from playbook
- Dynamic role assignment
- Active tactic determines transition
- Execution monitoring
- Reward, and adaptation
Selection
Monitoring, Adaptation
Execution
Robot 1 Tactic
Robot 1 Tactic
Robot N Tactic
6Relevant Research Interests
- Methods for pickup teams
- Extend play-based coordination to distributed
execution - Effective human-robot teamwork with pickup teams
7TraderBots Overview
- Robots are organized as an economy
- Team mission is to maximize production and
minimize costs - Robots exchange virtual money for tasks to
maximize individual profit - System is designed to align local and global
profit maximization
8TraderBots Implementation
9Relevant Research Interests
- Human-multi-robot interaction
- Role assignment for highly heterogeneous teams
- Improving robustness and adaptivity
10Speech Overview
- Communication in mobile environments
- Natural spoken language-based interaction
- Supporting high semantic content
- Communication in multi-participant domains
- Human(s) and robots(s) interacting on the same
channel
11Speech Implementation
- Sphinx ASR engine
- Speaker-independent recognition
- Speaker-adaptive capability
- Phoenix semantic parser
- Concept extraction
- RavenClaw dialogue engine
- Task-based representations
- Full mixed-initiative dialogue
- Festival/Theta synthesis engine
- Rosetta generation engine
- Portable/Wearable platforms
Comms Channel
. . .
12Relevant Research Interests
- Developing conversational algorithms for
human-robot polylogue - Communicating navigational information between
humans and robots in unstructured environments - Cooperative grounding of objects, locations, and
tasks in novel environments
13Current Status
- Wireless network-based (UDP) communication
protocol for human-robot interaction designed and
implemented - Closed loop integration between speech-based
command system, a Segway robot, and a Pioneer
robot accomplished - 2-D kinematic simulation tool for testing
interface to Pioneer robots - Video demonstration of speech-based command of a
Segway robot and a Pioneer robot - Plan for year 1 scenario and relevant
technology development and integration to
accomplish this scenario
14Treasure Hunt Scenario
15Basic Scenario
- 2 Segways, 2 Pioneers, and 1 human
- Discover and return to base as much treasure as
possible within a 20 minute period - Treasure will be identified by landmarks
currently used by the Segway soccer team
Mapper (M) Seeker (S) Handler (H) Deliverer (D) Leader (L)
Humans X X X
Segways X X X
Pioneers X X X
Gators X X X
AIBOs X
16Detailed Scenario
- Goal
- To combine TraderBots for negotiation and role
assignment with plays for synchronization - Low-level software remains independent
- Challenges
- How do we decide which Play to adopt?
- How do we handle sub-teams?
- How do we generate leaders?
17Step 1 Form Sub-Teams
Lets form a sub-team
Fixed sub-teams for May 2005
18Step 2 Command to Search
Team A, search area 1
- Build map
19Step 3 Explore and Search
- Follow
- Go to search area
- Build map
20Step 4 Search
- Follow and search for treasure
- Execute search pattern
- Build map
21Step 5 Found!
We found it!
We are at ltx,ygt
22Step 6 Recovery
Human aided loading
23Step 7 Return
- Follow
- Unload
- Return home
- Unload
Human aided unloading
24Post Year 1 Goals
- Failure detection and recovery
- Outdoor environments
- Adversaries
- Larger more diverse teams
- Adaptation and learning