Learning Agents Center

1
IT 803 Spring 2004 Mixed-Initiative Intelligent
Systems Prof. G. Tecuci
COLLAGEN Mixed-Initiative Interaction with a
Collaborative Agent
Dorin Marcu 02-09-2004
Learning Agents Center Computer Science
Department George Mason University
2
Presentation Outline

• A brief introduction to COLLAGEN
• An air travel application built with COLLAGEN
• A more detailed look at COLLAGEN
• architecture
• communication
• shared awareness
• tasks
• control
• evaluation
• Summary of the main ideas in COLLAGEN
• Lessons learned
• References

3
Research Motivation

• current systems lack support for the users
  problem solving process during extended periods
  of time
• the order in which actions must be performed by
  the user and the system is often inflexible
• its hard to recover from mistakes
• each system has its own interaction conventions

4
Explored Solution

• support the problem-solving level of
  human-computer interaction via a collaborative
  software agent
• develop a new paradigm for human-computer
  interaction which explicitly supports the users
  problem-solving process based on current theories
  of collaborative discourse

5
Assumption
A human-computer interface based on familiar
human discourse rules and conventions will be
easier for people to learn and use than one that
is not.
6
Collaborative Interface Agents
From (Rich and Sidner, 1998), page 317
7
COLLAGEN A Collaboration Manager

• A collaboration manager is a software component
  that
• mediates the interaction between a software
  interface agent and a user
• keeps track of the linguistic and attentional
  state of a discourse (similarly to a discourse
  manager)
• keeps track of the collaborative intentions of
  the participants

8
COLLAGEN A Collaboration Manager (contd.)

• A collaboration manager is less than a fully
  automated planning system because
• it does not by itself decide what the agent
  should do or say next (though it may provide some
  candidates)
• it provides a representation for recording the
  decisions that the agent has made and
  communicated.

9
Definition of Mixed-Initiative in COLLAGEN
Mixed-initiative reasoning in COLLAGEN can be
defined as a discourse-based collaboration
between a human user and an interface agent that
attempt to achieve shared goals by decomposing
them into sub-goals and primary actions for which
they have complementary solving capabilities.
10
Characteristics of Mixed-Initiative in COLLAGEN

• one human one agent, discourse-oriented
  collaboration
• partial, task-oriented mixed-initiative system
• some tasks have fixed-initiative flags,
  specifying which participants can solve them
• the human participant has precedence over the
  agent, and can ignore the contributions of the
  agent
• initiative taking
• proposing a goal
• proposing a recipe for solving a goal
• solving a (part of a) recipe
• delegation

11
Main Features of COLLAGEN

• both participants know and intend that all their
  actions are observed
• reporting communication (I have done x)
• direct observation

12
Main Features of COLLAGEN (contd.)

• the mixed-initiative capabilities of the agent
  arise from the interplay of two sources
• application-independent algorithms and data
  structures in COLLAGEN
• application-specific code and libraries in the
  agent
• a library of recipes that specify the typical
  steps and constraints for achieving certain goals
• arbitrary pattern-action rules

13
Main Features of COLLAGEN (contd.)

• supports mixed-initiative by
• interpreting discourse acts
• maintaining a model of the achieved and expected
  tasks and goals of the user and agent
• the user makes the final decision for the
  problem-solving process and can ignore the
  contributions of the agent

14
Main Features of COLLAGEN (contd.)

• the interaction model is based on a formal
  representation of the mutual beliefs about the
  goals and actions to be performed, and the
  capabilities, intentions, and commitments of the
  participants

15
Presentation Outline

• A brief introduction to COLLAGEN
• An air travel application built with COLLAGEN
• A more detailed look at COLLAGEN
• architecture
• communication
• shared awareness
• tasks
• control
• evaluation
• Summary of the main ideas in COLLAGEN
• Lessons learned
• References

16
Air Travel Planning System
You are a Boston-based sales representative
planning a trip to visit customers in Dallas,
Denver, and San Francisco next week. You would
prefer to leave on Wednesday morning, but can
leave on Tuesday night if necessary. Your
customer in Denver is only available between 11
a.m. and 3 p.m. on Thursday. You would prefer to
fly as much as possible on American Airlines,
because you have almost enough frequent-flier
miles to qualify for a free trip this summer.
You absolutely must be home by 5 p.m. on Friday
to attend your sons piano recital.
17
Air Travel Planner Interface
From (Rich and Sidner, 1998), page 319
18
Air Travel Planner Interface (contd.)
From (Rich and Sidner, 1998), page 320
19
Traditional Use of Planner

• Seven visitors and staff members were asked to
  solve this and similar problems and their
  behavior was recorded via informal notes and the
  logging facilities built into the application
• A typical problem solving session lasted about 15
  minutes and entailed about 150 user actions
  (mouse clicks).

20
User Problems with the Traditional Planner

• Various forms of getting stuck and getting lost
• trouble knowing what to try next when the trip
  has been over- or under-constrained
• trouble keeping track of which combinations of
  routes and constraints were already examined
• workflow interruptions caused by the use of
  applications functions (e.g. Snapshot save
  context)

21
Collaborative Interaction Example
(Rich and Sidner, 1998), page 323
22
Collaborative Interaction Example (contd.)
(Rich and Sidner, 1998), page 323
23
Presentation Outline

• A brief introduction to COLLAGEN
• An air travel application built with COLLAGEN
• A more detailed look at COLLAGEN
• architecture
• communication
• shared awareness
• tasks
• control
• evaluation
• Summary of the main ideas in COLLAGEN
• Lessons learned
• References

24
The Architecture of COLLAGEN
(Rich and Sidner, 1998), page 335
25
The Interface Agent
COLLAGEN does not provide tools for building a
complete agent different types of agents can be
supported (rule-based expert systems, neural
nets, or a completely ad hoc collection of
code) COLLAGEN provides a generic framework for
recording the decisions made and communicated by
the agent (and the user), but not for making them.
26
Question
Why would such a generic framework be useful?
Effective?
27
Answer
Why would such a generic framework be useful?
Effective? Because it will allow the reuse of
components for - building collaborative agents -
application-independent discourse manager
28
The Execution Cycle

• a communication or observation event arrives at
  the discourse interpretation module
• the discourse interpretation module updates the
  discourse state
• a new agenda of expected communication and
  manipulation acts is computed by the discourse
  generation module
• the agent may decide to select an entry in this
  new agenda for immediate execution (according to
  the agents initiative strategy)
• the user communication menu is updated with all
  the communication actions in the agenda for which
  the actor is either unspecified or the user

29
The Default Agent Initiative Strategy
The default agent implementation that is included
in COLLAGEN always chooses to perform the highest
priority action in the current agenda for which
the actor is either unspecified or itself. The
priorities are manually associated with
actions. Previous experiments also used pair-wise
comparison rules. (Rich, 2002)
30
The Architecture Issue

• COLLAGEN has a generic framework for developing
  interface agents that can collaborate with human
  users to solve shared goals.
• The generic framework must be customized for a
  specific application by developing
  application-dependent
• recipes
• action models
• methods for performing actions and for observing
  actions performed by the user
• collaboration behavior

31
Presentation Outline

• A brief introduction to COLLAGEN
• An air travel application built with COLLAGEN
• A more detailed look at COLLAGEN
• architecture
• communication
• tasks
• shared awareness
• control
• evaluation
• Summary of the main ideas in COLLAGEN
• Lessons learned
• References

32
Window Sharing

• both the user and the agent have a dedicated
  home window that is used for communication
  between them
• each home window contains an identifying face
  and has an associated cursor
• the home windows are serviced by separate
  processes to support asynchronous
  mixed-initiative interaction

(Rich and Sidner, 1998), page 321
33
Types of Agent Communication

• printing English text in the agents home window
• acting on the applications interface with its
  cursor while approved and observed by the user
• The user can ignore the suggestions proposed by
  the agent.

(Rich and Sidner, 1997), page 290
34
Types of User Communication

• selecting from a menu of communications expected
  by the discourse interpretation algorithm (that
  were generated from the underlying model of
  discourse)
• acting directly on the application with his/her
  cursor (the agent always observes the users
  actions through a generic layer in the
  application that mirrors semantic actions into
  the input buffer of the agent process)

(Rich and Sidner, 1998), page 325
(Rich and Sidner, 1998), page 319
35
The Communication Issue

• COLLAGEN is based on a discourse theory of
  collaboration in which
• the agent communicates by
• generating natural language text or
  user-selectable menus/buttons
• performing direct actions in the interface that
  the user can observe
• the user communicates by
• selecting agent generated menus/buttons
• performing direct actions in the interface that
  the agent can observe

36
Presentation Outline

• A brief introduction to COLLAGEN
• An air travel application built with COLLAGEN
• A more detailed look at COLLAGEN
• architecture
• communication
• shared awareness
• tasks
• control
• evaluation
• Summary of the main ideas in COLLAGEN
• Lessons learned
• References

37
Collaborative Discourse Theory
Collaboration is a process in which two or more
participants coordinate their actions toward
achieving shared goals. Most collaborations
between humans involves communication. Discourse
is a technical term for an extended communication
between two or more participants in a shared
context, such as a collaboration.
38
Collaborative Discourse Theory (contd.)

• Three interrelated types of collaborative
  discourse structures (Grosz and Sidner, 1986)
• intentional structure (formalized as partial
  SharedPlans)
• linguistic structure (includes the hierarchical
  grouping of actions into segments)
• attentional structure (captured by a focus stack
  of segments)

39
Intentional Structure for the Air Travel Planner

• the user knows the constraints on travel
• the agent has access to a data base of all
  possible flights
• both participants have a common goal (to find an
  itinerary that satisfies the constraints)

40
Intentional Structure for the Air Travel Planner
(contd.)

• both participants
• have agreed on a sequence of actions (a recipe)
  to accomplish the common goal (e.g., choose a
  route, specify some constraints on each leg,
  etc.)
• are capable of performing their assigned actions
• intend to do their assigned actions
• are committed to the overall success of the
  collaboration (not just the successful completion
  of their own parts).

41
SharedPlans
SharedPlans is a formal representation of the
participants mutual beliefs about the goals and
actions to be performed, and of their
capabilities, intentions, and commitments.
42
Representation of SharedPlans in COLLAGEN
(Rich and Sidner, 1998), page 331
43
Features of SharedPlans

• partial (due to incomplete knowledge)
  communication is required to fully specify them
• recursive
• their planning and execution is usually
  interleaved for each participant and among
  participants

44
SharedPlans in COLLAGEN
COLLAGEN provides a generic framework only for
recording the order in which planning and
execution occur. COLLAGEN does not currently
provide a generic framework for execution
(interleaving planning and execution).
45
Discussion
No generally accepted domain-independent theory
of how people manage the interleaving of planning
and execution. Best candidate the BDI
(belief/desire/intention) frameworks.
46
Discourse Segments
There is general agreement that discourse in
human-human interactions has a natural
hierarchical structure, the elements of which are
called segments. A segment is a contiguous
sequence of communicative actions that serve some
purpose (e.g. to achieve shared knowledge of some
fact).
47
Example of Discourse Segments
(Rich and Sidner, 1998), page 329
48
Focus Stack
The focus stack contains discourse segments in
the order in which they are created during the
natural flow of a collaborative discourse. It
captures the shifting focus of attention in a
discourse. New segments and sub-segments are
created, pushed onto the focus stack, completed,
and then popped off the stack as the SharedPlan
unfolds in the conversation.
49
Example of Focus Stack
(Rich and Sidner, 1998), page 331
50
Discourse State in COLLAGEN
(Rich and Sidner, 1998), page 335
51
DiscourseState Representation
52
Discourse State Representation (contd.)

• The discourse state representation contains
• the plan tree, which is an approximate
  representation of a partial SharedPlan)
• the focus stack
• the history list, which contains top level
  segments that have been completed and removed
  from the focus stack

53
Discourse Interpretation in COLLAGEN
(Rich and Sidner, 1998), page 335
54
Discourse Interpretation
The discourse interpretation module determines
how the current direct communication or observed
manipulation action can be viewed as contributing
to the current discourse purpose (of the top
segment in the focus stack).
55
Recognized Acts

• The discourse interpretation module recognizes
  acts that
• directly achieve the current purpose
• correspond to one of the steps in a recipe for
  the current purpose
• identify the recipe to be used to achieve the
  current purpose
• identify who should perform the current purpose
  or a step in the current recipe
• identify an unspecified parameter of the current
  purpose or a step in the current recipe.

56
Questions
What types of acts that may occur are not
recognized by the discourse interpretation
module? What can be done in those cases?
57
Answers

• What types of acts that may occur are not
  recognized by the discourse interpretation
  module? What can be done in those cases?
• interruptions (a segment that does not contribute
  to its parent)
• one participant decides to address a new goal
• the participants pursue multiple goal in the
  same time
• focus stack management
• incomplete recipe
• let the user manage the focus stack

58
Questions
Discourse interpretation looks similar to plan
recognition, which is known to be exponential in
the worst case. Is discourse interpretation
necessarily exponential? What can be done to
avoid this?
59
Answers

• Discourse Interpretation is generally not
  exponential because
• the participants can try to make sure that their
  intentions can be understood without a large
  cognitive search
• the participants can ask clarification questions
  to minimize search
• the search is performed through the steps of the
  current recipe or all known recipes for the
  current segment

60
Discourse Generation in COLLAGEN
(Rich and Sidner, 1998), page 335
61
Discourse Generation
Discourse generation is the inverse of discourse
interpretation. It produces a prioritized agenda
of (possibly partially specified) actions which
would contribute to the current discourse segment
from the focus stack, and its associated
SharedPlan.
62
Dynamic Generation of Agent Responses

• Develop a plug-in architecture for reusable
  components that generate agent responses based on
  the last event (communication or action) and the
  current discourse status.
• Develop
• application-independent
• application-dependent
• role-dependent (e.g. assistant vs. tutor)
• plug-ins for the developed architecture.

63
Plug-in Architecture
(Rich et all., 2002), page 784
64
Plug-ins for Generating Agent Responses

• A plug-in for generating agent responses is an
  object that is associated with an internal state
  and an algorithm for generating responses based
  on its state, the discourse state and the event
  received by the discourse interpreter module.
• A plug-in
• can access and modify the discourse state.
• can access the recipes library.
• has a priority associated with all the responses
  it generates

65
The Plug-in Interface
update(event) Updates the private state of the
plug-in (if any) and modifies the discourse state
(if necessary) for example by adding a goal or
binding a parameter. generate(event) Returns a
(possibly empty) list of agenda items which would
be an immediate response to event. visit(node) R
eturns a (possibly empty) list of agenda items
which would contribute to the goal of node.
66
Application-Independent Plug-ins (6)
Execute(goal) (default priority 0) If goal is
primitive and all of its parameters are bound and
its first parameter is bound to the agent, then
return goal. ProposeShouldUser(goal) (default
priority 0) If goal is primitive and its first
parameter is bound to the user, then return an
utterance of the form Please perform/say
goal. AskWho(goal) (default priority 0) If
goal is primitive and its first parameter is
unbound, then return an utterance of the form
Who should perform/say goal?
67
Role-Dependent Plug-ins
TeachStep(goal) (priority 100) If goal is
teachable, then return an utterance of the form
The first/next step is goal. TeachInitiative(go
al) (priority 130) If the student model
indicates that the student should already know
how to achieve goal, then return You take it
from here. PositiveFeedback(action) (priority
160) If action was live and expected, then return
an utterance such as Great, Nice, Right.
68
Algorithm for Choosing Responses

• add to the agenda the results in the order
  returned, of invoking the generate method of each
  plug-in on the received event, in the default
  plug-in order
• add to the agenda the results in the order
  returned, of invoking the visit method of each
  plug-in, in the default plug-in order on the
  live, expected nodes (visited in breadth-first
  order)
• do a post-pass to reorder the agenda based on
  any additional explicit preferences

69
Segmented Interaction History

• provides the user with a structured guide to
  his/her problem solving process
• can be used to retry, re-apply or undo past
  actions
• displayed at the users request

(Rich and Sidner, 1998), page 336
70
Producing the Segmented Interaction History

• Starting from the current discourse state
• list the purpose of each top level segment on
  the history list starting with the oldest
• recursively list the purpose of each closed
  sub-segment followed by the purpose and elements
  of each open sub-segment, starting with the top
  level open segment (the base segment of the focus
  stack)
• for each open sub-segment list the unexecuted
  recipe steps (if any) in the plan tree for that
  segments purpose, starting with the most deeply
  nested

71
Question
What are some advantages or disadvantages of
COLLAGENs segmented interaction history?
72
Answer

• Some advantages or disadvantages of COLLAGENs
  segmented interaction history
• is structured (interaction histories are
  typically flat)
• reflects the users actual problem solving
  process
• requires a structured discourse (interaction)
  model and its use in the problem solving process
  (interpretation, etc.)

73
The Shared Awareness Issue

• Shared awareness in COLLAGEN is maintained by the
  agent through the discourse state, which is
  accessible to the user via the segmented
  interaction history.
• It contains descriptions of the
• past interactions
• current stack of execution reflecting the
  goals being solved.
• It is updated based on
• direct manipulation actions performed by the
  user or the agent
• user selections of agent generated menus/buttons

74
Presentation Outline

• A brief introduction to COLLAGEN
• An air travel application built with COLLAGEN
• A more detailed look at COLLAGEN
• architecture
• communication
• shared awareness
• tasks
• control
• evaluation
• Summary of the main ideas in COLLAGEN
• Lessons learned
• References

75
Agent Customization Through Task Models
The goal is to develop a formal model of the
collaborative tasks being performed by the agent
and the user. Since the agent does not have to
rely on the task model alone for its decision
making, the model only needs to be complete
enough to support communication and collaboration
with the user. The task model in COLLAGEN
contains recipes and primitive actions.
76
The Recipe Library in COLLAGEN
(Rich and Sidner, 1998), page 335
77
Recipes

• a recipe is an application specific resource
  used to derive a sequence of steps to achieve a
  given goal (the objective of the recipe)

(Garland at all., 2001), page 45
78
Recipes (contd.)

• It is easy to represent recipes with a fixed
  number of steps in a simple recipe formalism.
• More complicated recipes require step structures
  that depend on some parameters of the objective
  (e.g. the top level recipes for scheduling a trip
  working forwards or backwards).

79
Question
How can a recipe with variable structure be
specified?
80
Answer

• How can a recipe with variable structure be
  specified?
• In COLLAGEN, a general-purpose procedural
  approach, called recipe generators, was
  developed
• recipes with variable structure are represented
  as procedures which, given an objective, return a
  recipe
• a predicate can also be associated with a recipe
  to test whether it is still applicable as it is
  being executed

81
Manual Development of the Task Model

• initial versions of a task model are inferred
  from a small number of examples that show the
  most common solutions to key domain tasks
• the model is generalized to cover additional
  examples that demonstrate solutions involving
  alternate orderings for actions, optional
  behavior, or alternate task decompositions
  (defining additional examples may force the
  expert to re-conceptualize the entire domain
  necessitating reworking many previous examples)
• correct and complete learned models may need to
  be tweaked by the expert for other reasons (e.g.
  organization of a complete and accurate task
  model may be inappropriate for a collaborative
  agent)
• verify the behavior of the model using the
  collection of past examples

82
Learning The Task Model From Examples

• programming by demonstration
• incremental learning from examples
• allows updating past examples used in learning
• regression testing with all examples

83
Learning The Task Model From Examples

• Step one (optional) provide an initial list of
  primitive and non-primitive acts
• Non-Primitives MakeMeal
• Primitives Boil, CookPasta, PrepareSauce,
  ServeDinner

84
Learning The Task Model From Examples (contd.)

• Step two define un-annotated example
• GetPasta
• Boil
• CookPasta
• PrepareSauce
• ServeDinner

85
Learning The Task Model From Examples (contd.)

• Step three (partially) annotate example
• MakeMeal
• PreparePasta
• GetPasta
• Boil
• CookPasta
• PrepareSauce
• ServeDinner

86
Learning The Task Model From Examples (contd.)

• Step four learn (partial) recipe from example
• nonprimitive act PreparePasta
• primitive act GetPasta
• recipe Boil_CookPasta_GetPasta achieves
  PreparePasta
• steps Boil boil
• CookPasta cookPasta
• GetPasta getPasta
• constraints getPasta precedes boil
• boil precedes cookPasta

87
Learning The Task Model From Examples (contd.)

• Step five define additional example(s)
• MakeMeal
• GoToKitchen(kitchen2)
• PrepareSauce
• Boil(water3)
• CookClams(clams8,water3)
• MakeClamSauce(clams8)
• PreparePasta
• Boil(water3)
• CookPasta(linguini11,water3)
• ServeDinner(kitchen2)

88
Learning The Task Model From Examples (contd.)

• Step six refine learned recipe
• nonprimitive act PreparePasta
• parameter Water water
• primitive act GetPasta
• recipe Boil_CookPasta achieves PreparePasta
• steps Boil boil
• CookPasta cookPasta
• optional GetPasta getPasta
• bindings achieves.water cookPasta.water
• constraints boil.water cookPasta.water
• cookPasta.water water3
• cookPasta.pasta linguini11
• getPasta precedes boil
• boil precedes cookPasta

89
Revive Viewer
(Garland et all., 2001), page 49
90
The Learning Problem
Input (partially annotated example) Output
recipe(s) refined examples
(Garland et all., 2001), page 46
91
Experimental Results
The impact of annotations on the number of needed
examples O ordering E equalities P
propagators All includes optional
(Garland et all., 2001), page 47
92
Recipe Library Statistics

• 8 recipes defined in terms of 15 different goal
  or action types for the air planner system (the
  authors estimate that the final number should be
  about twice as much) (Rich and Sidner, 1998)
• 29 recipes, 67 recipe steps, 36 primitive acts
  and 29 non-primitive acts for the Symbol Editor
  application (Garland el all., 2001)
• 8 recipes, 19, recipe steps, 13 primitive acts
  and 4 non-primitive acts for the artificial
  cooking world model (Garland el all., 2001)

93
The Task Issue
Collaboration is modeled in COLLAGEN through
recipes and primary actions. At task modeling
time, some tasks can be designated to be
performed by the user or by the agent, the rest
being open. Example of user-designated tasks
specifying route or constraints. Example of
agent-designated tasks search data-base for
flight info. Open tasks can be negotiated based
on user preferences (e.g. asking the user who
should perform a task) and roles (tutoring vs.
assisting).
94
Presentation Outline

• A brief introduction to COLLAGEN
• An air travel application built with COLLAGEN
• A more detailed look at COLLAGEN
• architecture
• communication
• shared awareness
• tasks
• control
• evaluation
• Summary of the main ideas in COLLAGEN
• Lessons learned
• References

95
Control Principles

• a recipe needs to be identified for the current
  goal (identification of a recipe may be achieved
  by asking the other participant)
• a goal or action may be performed when all of
  its parameters are known and all of its
  predecessors (in the current recipe) have been
  achieved
• a goal or action may be performed by any
  participant who is capable, unless a specific
  participant has been specified

96
Global vs. Local Initiative

• most work is done on local initiative and its
  aspects
• turn taking and conversational control
• interruptions, and
• grounding (how the current speaker indicates
  that she has heard and understood the content of
  the previous speakers turn)
• true mixed-initiative systems must address both
  global and local initiative

97
Global Initiative

• application-independent based on the current
  discourse state and the agenda computed from it,
  a COLLAGEN-based agent has an explicit choice of
  relevant things to say at most points in a
  conversation
• application-dependent the agent can choose
  between answering a question, performing an
  interface action or some other behavior,
  according to the specifics of the application
  domain

98
Negotiation as Global Initiative

• negotiation is the ability to resolve
  differences in beliefs that are relevant to some
  shared goal
• through negotiation, collaborators with
  differing points of view about the goals and
  recipes they undertake, as well as the state of
  the world at any point in time, can reach an
  agreement
• currently, COLLAGEN does not incorporate a
  negotiation facility

99
Local Initiative

• COLLANGEN does not provide a general algorithm
  for turn taking because of the limitations of the
  current theories of local initiative in
  conversation
• Ad-Hoc mechanism developed (detailed in the
  following slides)
• provide the user with means to relinquish
  control to the agent
• provide the agent with means to get the users
  attention when the agent does not have control

100
Relinquish User Control
The user relinquishes the control when he/she
does not want to contribute any further to the
current SharedPlan and instead would like to see
what the agent can contribute. The agent was
designed to interpret an OK selected by the
user (other than as an answer to a direct Yes/No
question) as an signal of relinquishing control.
101
Request Users Attention
The agent was designed with the ability to wave
its cursor hand when it has something important
to contribute to the conversation. The resulting
behavior is considered very humanlike and
affecting.
102
History-based Transformations

• History-based transformations are used to alter
  the course of the problem solving process by
  reusing past actions (without modifying the
  history).
• History-based transformation types (at segment
  level)
• stop
• revisit
• retry
• undo
• reply
• The availability of such transformations is
  application-dependent (the extent to which the
  application provides methods to alter its
  internal state)

103
(Rich and Sidner, 1998), page 340
104
The Stop Transformation

• motivation
• the user wants to let the agent know that she/he
  is no longer working towards the current goal
• the agent has misunderstood what the current
  goal is
• actions
• pops the current segment off the focus stack
  without changing the application state
• if the purpose of the popped segment contributes
  to its parent, the appropriate un-bindings are
  also performed in the plan tree
• applicability
• only to open segments

105
The Retry Transformation

• motivation
• the user wants to return to working on an
  earlier goal achieved or not and try
  achieving it in a different way
• actions
• stop and reset the stack and plan tree to their
  states at the start of the selected segment
• the recipe becomes unbound
• applicability
• to any type of segments

106
The Revisit Transformation

• motivation
• the user wants to pick up where he/she left off
  working on an earlier goal
• actions
• stop and reset the stack and plan tree to their
  states at the end of the selected segment
• the recipe is preserved
• applicability
• only to closed segments

107
The Replay Transformation

• motivation
• the user wants to reuse earlier work in the
  current context
• actions
• all of the application acts in the selected
  segment are put together into one (possibly
  hierarchical) recipe which is then executed by
  the agent in the current context

108
Questions
What happens if some of the acts may not be valid
in the current context? To what type of segments
is the Replay transformation applicable?
109
Answers
What happens if some of the acts may not be valid
in the current context? Depending on the
specific details of the agents interface to the
application, such errors may need to be handled
by application-specific code in the agent, or may
be taken care of by the applications existing
API or graphical interface. To what type of
segments is the Replay transformation
applicable? Open and close segments. Why?
110
The Control Issue

• The control of initiative in COLLAGEN is partly
  managed
• in a generic way by the application-independent
  collaborative algorithms
• in an application-dependent way by the
  related-designations in the recipes
• in a role-dependent way by the discourse
  generation plug-ins.
• The agent can negotiate with the user by asking
  who to perform a non-designated task, or by
  asking whether the agent-generated solution is
  acceptable to the user.
• The user can relinquish control by asking the
  agent to perform a task (where possible).
• Through history-based transformations, the user
  has direct control over the problem solving
  process.

111
Presentation Outline

• A brief introduction to COLLAGEN
• An air travel application built with COLLAGEN
• A more detailed look at COLLAGEN
• architecture
• communication
• shared awareness
• tasks
• control
• evaluation
• Summary of the main ideas in COLLAGEN
• Lessons learned
• References

112
Evaluation
No experimental evaluation to assess any
strengths/weaknesses of the mixed-initiative
approach vs. the classic approach (direct
manipulation of the GUI-based applications. My
assessment since the user has the final decision
on the performed steps and the communication
model is straight-forward and natural, the
collaborative approach should lead to better and
faster problem-solving episodes.
113
Presentation Outline

• A brief introduction to COLLAGEN
• An air travel application built with COLLAGEN
• A more detailed look at COLLAGEN
• architecture
• communication
• shared awareness
• tasks
• control
• evaluation
• Summary of the main ideas in COLLAGEN
• Lessons learned
• References

114
Key Ideas in COLLAGEN

• multidisciplinary research in user interface,
  linguistics, and artificial intelligence
• collaboration between one user and one interface
  agent that is similar to human-human
  collaboration)
• guided and structured interaction based on
  formal model of collaboration (SharedPlans)
• separation of application-independent structures
  (collaboration theory, general framework) from
  application-dependent structures (recipes, agent
  customization, plug-ins)
• supports a users problem solving process by
  relating current actions to the global context
  and history of the interaction (discourse
  interpretation, discourse generation, discourse
  status, recipe library)
• oriented toward applications with direct
  manipulation of interfaces

115
Presentation Outline

• A brief introduction to COLLAGEN
• An air travel application built with COLLAGEN
• A more detailed look at COLLAGEN
• architecture
• communication
• shared awareness
• tasks
• control
• evaluation
• Summary of the main ideas in COLLAGEN
• Lessons learned
• References

116
Lessons Learned

• using a formal task model for guiding
  collaboration and mixed-initiative interaction is
  effective (either SharedPlans or GOMS-like
  models)
• using a menu-driven interaction is simpler that
  natural language processing (although it is not
  clear if it is generally sufficient)
• using plug-ins to customize behavior seems to
  be an effective approach
• recognizing user actions and performing agent
  actions is application dependent, and might be
  challenging with applications that are not
  restricted to direct manipulation of GUIs
• more advanced user/agent models are necessary
  for different roles (e.g. tutoring) and for
  multi-human multi-agent collaborations

117
Presentation Outline

• A brief introduction to COLLAGEN
• An air travel application built with COLLAGEN
• A more detailed look at COLLAGEN
• architecture
• communication
• shared awareness
• tasks
• control
• evaluation
• Summary of the main ideas in COLLAGEN
• Lessons learned
• References

118
Papers Used For Presentation
Charles Rich and Candace L. Sidner, COLLAGEN A
Collaboration Manager for Software Interface
Agents, User Modeling and User-Adapted
Interaction 8(3), 1998, Kluwer Academic
Publishers, pp 315-350. Charles Rich and Candace
L. Sidner, COLLAGEN When Agents Collaborate
with People, in Proceedings of the First
International Conference on Autonomous Agents,
February 1997, Marina Del Rey, California, ACM
Press, pp 284-291. Charles Rich, Neal Lesh, Jeff
Rickel and Andrew Garland, A Plug-in
Architecture for Generating Collaborative Agent
Responses, in Proceedings of the First
International Joint Conference on Autonomous
Agents and Multiagent Systems AAMAS02, July
15-19, 2002, Bologna, Italy, ACM Press, pp
782-789. Andrew Garland, Kathy Ryall and Charles
Rich, Learning Hierarchical Task Models by
Defining and Refining Examples, in Proceedings
of the International Conference on Knowledge
Capture K-CAP01, October 22-23, 2001,
Victoria, British Columbia, Canada, ACM Press, pp
44-51.
119
Other Interesting Related Papers
Andrew Garland, Neal Lesh and Charles Rich,
Responding to and Recovering from Mistakes
during Collaboration, in Proceedings of the
Workshop on Mixed-Initiative Intelligent Systems,
the 18th International Joint Conference on
Artificial Intelligence IJCAI03, August 9,
2003, Acapulco, Mexico, AAAI Press, pp
59-64. Jacob Eisenstein and Charles Rich,
Agents and GUIs from Task Models, in
Proceedings of the 7th International Conference
on Intelligent User Interfaces IUI02, January
13-16, 2002, San Francisco, ACM Press, pp
47-54. Charles Rich, Window Sharing with
Collaborative Interface Agents, ACM SIGCHI
Bulletin, Volume 28 Issue 1, January 1996, ACM
Press, pp 70-78. Barbara J. Grosz and Candace L.
Sidner, Attention, Intentions, and the Structure
of Discourse. Computational Linguistics 12(3),
1986, pp 175204. Jim R. Davies, Abigail S.
Gertner, Neal Lesh, Charles Rich, Candace L.
Sidner and Jeff Rickel, Incorporating Tutorial
Strategies Into an Intelligent Assistant, in
Proceedings of the 6th International Conference
on Intelligent User Interfaces IUI01, January
14-17, 2001, Santa Fe, New Mexico, ACM Press, pp
53-56.