Task Learning by Instruction: Benefits and Challenges for Intelligent Interactive Systems

1
Task Learning by InstructionBenefits and
Challenges for Intelligent Interactive Systems

• Jim Blythe, Prateek Tandon and Mandar Tillu
• USC Information Sciences Institute

2
Overview

• Complex tasks in intelligent assistants
• Overview of task learning by instruction in
  Tailor
• Integration with other task-related capabilities
• Challenges and new approaches in specifying tasks

3
Handling complex tasks in an intelligent assistant
Medication Regime
Travel
Assistant
UI
Task models
Domain models
Contracts
Vacation Activities
4
Specialized capabilities for complex tasks
Application
User invocation Advice/ Preferences Explanation
Monitoring Trouble-shooting Autonomy Opportunism/
Synergy .
Assistant
UI
Plan creation Learning/ Instructability Sharing
tasks and goals
Task models
Domain models
5
Specialized capabilities for complex tasks
Application
User invocation Advice/ Preferences Explanation
Monitoring Trouble-shooting Autonomy Opportunism/
Synergy .
Assistant
UI
Plan creation Learning/ Instructability Sharing
tasks and goals
Task models
Domain models
6
Modifying a procedure with TailorTask Learning
by Instruction

• User enters text instruction, e.g. You dont
  need authorization when the cost is below 2000.
• Tailor interprets by
• (1) recognizing type of modification,
• (2) mapping terms to its task and world
  ontology,
• (3) analyzing the proposed modification.

defProcedure PurchaseEquip cue do (purchase
obj_id) precond (and (user user) (pSelId
obj obj_id)) body seq do
(find_bids obj bid) context (use_form
form obj) do (complete_form form
obj) select (and (total_price bid
price) (lt price
2000)) do (get_authorization obj
user)
To purchase equipment 1. find two bids 2.
fill out form 10C 3. request authorization
from two managers 4. if authorization granted
place order with purchasing
7
Modifying a procedure with Tailor compared with
learning from observation/demonstration

• Users directly describe the desired change
• Users describe steps without taking them
• Users can ignore implementation details

• Benefits
• Challenges

• Map user instruction to procedure knowledge using
  context (e.g. other intentions)
• Navigating a set of complex alternatives
• Follow-up questions to resolve ambiguity and help
  fix introduced problems

Blythe, IUI 05 AAAI 05
8
User describes a new taske.g. list hotels
within miles of meeting
9
The first step
List each hotel name
10
Aggregating DP approach to find compositions of
tasks

• User instruction you dont need form B when the
  seller is in the US
• Tailor finds matching expression (and (vendor
  selection v) (country v US))
• Shown as when the country of the vendor of the
  selected laptop is the US

Synonym from WordNet seller
Variable and description from GPA
laptop
Vendor
0selection (GPA) the selected laptop
company
0 1(Vendor seln v)
Object from user instruction
HasDisplay
ComputerTotalPrice, ShippingCost, RAM,
Country
display
country
DialCode
0 1(HasDisplay seln d)
Size
0US (constant) 1
number
match
0 1(Price seln p), .. 2 (and (HasDisplay
seln d) (Size d s)), ..
2(and (Vendor seln v) (Country v
US))
lt, gt,
11
Recent improvements extend DP with dynamic
grouping of partial matches

• Group partial expressions by datatype and set of
  instruction words matched
• invariant wrt. propagation and matching
• Propagate groups, not individual expressions
• Exponential reduction in propagations, and in
  iterations required to find a match

Vendor
laptop
company
Country
0selection (GPA) the selected laptop
0 1(Vendor lapt Ø)
country
selection e laptop Ø
0US (constant) 1
(Vendor laptop Ø) e company seller
match
2(and cmpny seller (Country ? US))
e country seller, US
12
The last step and resulting procedure
13
Overview

• Task learning in desktop systems
• Overview of task learning by instruction
• Integration with other task-related capabilities
• Challenges and new approaches in specifying tasks

14
Challenges highlighted by Tailor prototype

• Users unsure of the next step
• More context-dependent menus and wizards
• Fewer options, although there are several valid
  alternatives
• Many alternative input sets for primitive
  procedures
• Full list is overwhelming be selective and
  summarize
• Constraints on inputs of primitive procedures
• Procedure hard to understand
• Combining tasks built by users inappropriate
  information
• Hide substeps
• Hide parameters
• Allow drill-down

15
Task-related capabilities communication and
reasoning
Communicating with user
Reasoning
Plan creation Monitoring Learning by
observation Opportunism Collaboration
Invocation Explanation Advice/ preferences
Learning by instruction/ demo Autonomy
16
Context-related guidance
Select modification
Add step
Add input
Done
In order to find available hotels near the
meeting with input a meeting
No steps currently defined
Add a step
17
Hiding steps and parameters
Describe step to add
Hide parameters focus on distinct actions
get hotels
OK
In order to find available hotels near the
meeting with input a meeting
get hotel chains with frequent flyer miles for
American
Options
get hotel chains with frequent flyer miles for
American
get hotel chains with frequent flyer miles for
United
get hotel chains with frequent flyer miles for
Delta
get hotels from onlineReservationz
18
Heuristic show external actions (e.g. info
access), hide internal book-keeping
Describe step to add
list hotel if distance lt 2
OK
Highlight new inputs that are automatically
suggested
In order to find available hotels near the
meeting with input a meeting, a street and
a zip
get hotels from onlineReservationz for each
hotel get Yahoo distance if distance
lt 2 add the hotel to the list
Options
Options
Show steps that determine information access
19
Show more detail about steps when the user
explores the alternatives
Describe step to add
list hotel if distance lt 2
OK
In order to find available hotels near the
meeting with input a meeting, a street and
a zip
Show parameters when the user asks to see
alternatives
get hotels from onlineReservationz for each
hotel get Yahoo distance if distance
lt 2 add the hotel to the list
Options
Options
distance between the street and zip, and the
hotel street and zip
options that dont use the hotel street and zip
options that dont use new input variables
Explore only through the inputs, not how they are
used
20
Reduce search space using constraints on inputs

• Given 3 locations
• YahooDistance(street1, zip1, street2, zip2)
  81 alternatives
• Given location tuple(street, zip)
• YahooDistance(loc1, loc2) 9 alternatives
• Given output is a distance metric symmetric,
  d(X,X) 0
• YahooDistance(loc1, loc2) 3 alternatives

21
Constraints may be reusedfrom task ontology
Action
Information-generating action
Symmetric only consider one ordering
constraints
Information-generating action output includes a
distance between two inputs A and B
Identity d(A,A) 0, so ignore B A.
YahooDistance(Aloc, Bloc) -gt distance_miles
loc compound(street, zipcode)
22
High-level organization of outputs
Select modification
Done
Output is created in list form because the
constructor is embedded in an iteration
In order to find available hotels near the
meeting given a meeting, a street and zip
get hotels from onlineReservationz for each
hotel get Yahoo distance if distance
lt 2 add the hotel to the list Output
is the list of hotels
Add step before this
Add test
Remove this step
Change step inputs
Mark step as output
Show details
23
Result simpler description and control
Select modification
Done
In order to find available hotels near the
meeting with input a meeting, a street and
zip
get hotels from onlineReservationz for each
hotel get Yahoo distance if distance
lt 2 add the hotel to the list Output
is the list of hotels
Instead of
24
Current work

• Creating primitive procedures while creating
  top-level procedure, not before.
• Using examples from previously-build procedures
  to select steps, bindings and follow-ups
• Constraints representation, input and use

25
Summary

• Learning complex activities from instruction.
  Similar issues in modules that communicate about
  tasks
• Many potential matches for an instruction
• Describing learned tasks find the right level
  of initial detail and allow drill-down
• Utilize constraints on possible procedure inputs