Intelligent Automation in Collaborative Systems

1
Intelligent Automation in Collaborative Systems

• Joshua Introne and Richard Alterman
• COOP06
• 12/05/2006

2
Overview

• Automation
• Approaches
• Groupware
• Case study
• VesselWorld
• Representational problems for the system
• Representational problems for the users
• Adding interface structure
• Adding automation
• Conclusions Future Work

3
Automation
4
Why add automation?

• Human activity is inherently situation dependant
  (Suchman, 1987).
• The interface can only effectively support a
  limited set of functionality at once.
• How to get enough functionality in?
• Interface clutter, creeping feature-itis
• Automation helps solve a design problem.
• Allows the designer to provide additional
  functionality, to be delivered at runtime when
  the need arises.

5
Automation Issues

• User knowledge acquisition
• Automation delivery
• Inference

6
Knowledge Acquisition Strategies

• Try to infer the users needs by monitoring what
  they do (i.e. via plan recognition).
• Possible in some direct manipulation (Hutchins,
  Hollan Norman, 1985) environments.

7
Knowledge Acquisition Strategies

• Introduce a formal language so that the user
  communicates directly with an agent.

8
Knowledge Acquisition Strategies

• Introduce natural language so that the user can
  communicate with the system.

9
Knowledge Acquisition Strategies

• Keyhole plan recognition (Kautz, 1991 Bauer,
  1999)
• Hard depends upon existing interface structure
  domain
• Adding collaborative agents with formal languages
  (e.g. Rich and Sidner, 1998)
• Increased user effort user manages two tasks
  instead of one
• Natural language (e.g. Litman Allen, 1998).
• Hard does not scale to complex applications (two
  task problem).

There is tension between user effort at runtime
for designer effort at design time.
10
Collaboration Groupware

• Hard to coordinate activities
• General purpose (e.g. chat) tools offer no
  support for organizing activity.
• Structure helps
• E-mail templates (e.g. Information Lens (Malone,
  1987))
• Artifacts in workflow systems (e.g. Ariadne
  (Schmidt Simone, 1996)
• Maps charts (e.g. Mercator projection chart
  (Hutchins, 1995))
• Stop sign (Alterman, et. al, 2001).
• Ubiquitous shared calendars, bug-tracking
  systems, classified ads, stock-market ticker

11
Groupware Automation

• Structured artifacts address users
  representational needs.
• Also provide the system with structured access to
  coordinating information at runtime.
• Shared, semi-structured artifacts help solve the
  knowledge acquisition problem

12
Case Study
13
VesselWorld

• Three participants at remote workstations
• Each player is a ships captain
• Goal is to remove toxic waste from a harbor
• Communication occurs through a chat window
• Users have private (non-shared) marker facility
• Activity is turn-based

14
VesselWorld

• Each ship has different capabilities, and can
  only see for a limited range.
• At runtime, no one has complete information about
  the domain.
• Toxic wastes require various coordination
  strategies.
• Operators have different capabilities.

15
Automation Goal

• Planning is laborious and error prone
• Forgotten plan steps.
• Plans out of synchrony.
• Too many possible plans at once.
• Goal Provide automated planning facility
• Automatically generate synchronized plans for
  users.
• Infer users goals and make them visible to
  others.

16
Knowledge Acquisition

• To infer plans, system needs state information.
• Where each waste is, equipment requirements, etc.
• Where each user is, where they are going
• System has some information
• Users planning information
• Access to interface actions and chat
• But no access to structured information about
  toxic waste barrels!

17
Representational problems for the system
18
Representational problems for the system
19
Representational problems for the users
20
Representational problems for the users

• Users explicitly introduce a shorthand for
  referring to wastes (c.f. Clark Wilkes-Gibbs,
  1986).
• Frequent forgotten wastes, misaligned common
  ground.
• One group created a protocol to align private
  representations
• One user would request a marker check
• All other users would enter their private markers
  into chat
• Discrepancies were resolved

21
Introducing a structured representation
The Object List
22
(No Transcript)
23
Object List created during chat
24
User Performance with the Object List
25
Using information from the Object List

• Object List makes two kinds of information
  available
• A set of references to objects that users use in
  chat
• P(UtteranceJoint Lift) 64
• P(UtteranceSingle Lift) 42
• A list of user-maintained information about the
  current state of the world.
• The Object List is a model of the users runtime
  context.

26
Intent Inference

• Static Bayesian Network predicts user intentions
  from Object List information.
• Easy to implement plenty of open-source
  implementations.

27
Evaluating Utility of Object List for Knowledge
Acquisition

• How does information from the Object List compare
  to perfect information about the world?
• Four information conditions Perfect Information,
  Perfect Information Chat, Object List
  Information, Object List Chat
• Two metrics
• Correct Goal Rate Percentage of lifts
  accurately predicted
• False Positive Rate Percentage of predictions
  that were wrong

28
Empirical Results
29
Introducing automation

• Shared, WYSIWIS component.
• Users can request single or joint plans.

30
Evaluating Automation

• 40-hour study with four teams
• 2 hrs. of training time
• Additional 10 hrs. of play time
• One group with the automated component, one group
  without

31
Results

• The automation was used.
• One quarter of all plan steps were submitted
  through the tool.
• Surveys indicated that users liked the component
  (average 5.3 out of 7).
• Performance improved.
• Joint errors were reduced by 45.
• Reduced cognitive effort.
• Plan steps submitted twice as fast for
  automatically generated plans.
• Automation was not used for awareness
• No decrease in communication
• Users did not report using it for awareness

32
Summary Future Work
33
Summary

• Automation can bridge the gap from design-time to
  run-time.
• A problem is acquiring enough run-time
  information.
• In groupware, structured representations can help
  people coordinate.
• Groupware can also make automation easier by
  making coordinating information available to the
  system.
• The case-study presented demonstrates how this
  can be done.

34
Future work

• Some unanswered questions
• What kind of problems can be solved with
  automation?
• What kind of structures can be leveraged?
• Why wasnt awareness information useful?
• How should automation be delivered in groupware?

35
Backup slides
36
Approach
37
Adding Structure

• Few existing design level techniques
• DCog, Activity Theory, Cognitive Task Analysis
  provide insight but no explicit design
  recommendations
• Reference Structure Analysis (Alterman, et al.,
  2001 Feinman Alterman, 2004)
• Analysis of referencing activity to predict
  structural needs
• Validated in user studies (Feinman, 2006).

38
Methodology

• Log complete interaction data with a base system
• THYME groupware toolkit provides logging and
  replay (Landsman, 2005).
• Analyze data using RSA
• Add shared structured representations
• Leverage structure to add automation

39
Chat Information
40
Utility of reference information
41
Automation Delivery
42
Automation Delivery

• How does the system know when to provide
  automated behavior?
• Estimate cost of interruption (Horvitz, 1999)
• Identify users location in a task hierarchy
  (Adamczyk Bailey, 2004)
• Hard inference problems similar to keyhole plan
  recognition.

43
What about automation delivery?

• Critical issues in automation delivery are
  interruption, and user-control.
• In VesselWorld, automation was on-demand, and did
  not interrupt.
• Planning support worked.
• But users did not report using the automated
  component to stay aware of one another.
• WHY?

44
Activity in VesselWorld

• In VesselWorld, activity is hierarchically
  organized
• Plans have subplans
• Often deeply nested
• Planning errors occur more frequently during more
  complex plans
• There is a need for awareness of higher level
  plans

45
Coupling

• Coordinating actors move continuously in and out
  of phases of coupling (Salvador et al., 1996
  see also Dourish Bellotti, 1992).
• Research in multi-tasking suggests that moving
  between tasks is an opportunity for loss of
  context (Miyata Norman, 1986 Kirch, 2000).