(step s2 (find-loc alt-hold-button = ?loc)) (step s

1
The Apex ArchitectureMultitasking Under
Time-Pressure and Uncertainty

• Michael Freed
• NASA Ames Research Center

2
Cognitive Architecture Applications at NASA

• Evaluating cockpit designs
• Planning crewed missions
• Artificial participants in
• training/eval simulations
• Tech impact assessment
• Problem
• Demand for human simulation
• models exceeds what modelers
• can supply with current tools

3
Cognitive Architectures as Practical Tools

• What capabilities/resources does it provide above
• those in a general-purpose programming language?
• How usable are those capabilities? How much
  time,
• expertise, and inventiveness does someone need
  to
• build an agent that meets given requirements?
• How good are its software qualities? E.g.
  stability,
• maintainability, extensibility (ilities)

4
Outline

• Overview of Apex architecture and applications
• Capabilities incorporated into the architecture
• - Procedure-based reactive planner
• - Multitask management
• Usability-related efforts

5
Apex applications
Realistic simulations of human pilots and
air traffic controllers Interface
evaluation based on CPM-GOMS Autonomous
robots for reconnaisance and surveillance
6
(No Transcript)
7
Capabilities What we need
Many domains of practical interest are demanding
in the sense that a skilled agent must

• Cope with uncertainty
• Cant completely know or predict world state
• Actions may fail or produce undesirable
  side-effects
• Actual resource requirements determined during
  execution
• New, urgent tasks can arise at any time
• Cope with time-pressure
• Cant deliberate endlessly
• Options for resolving uncertainty constrained
• Conform to SOPs (not just goal-pursuit)

8
Capabilities Procedure-based reactive
planner
Procedure-based reactive planners (E.g. RAPS,
ESL, PRS) select action based on stored,
generalized procedures.

• Cope with time pressure by
• Using stored, generalized procedures
• Avoiding expensive decision processes (search)
• Cope with uncertainty by
• Deciding action at the last moment when maximum
• situation information available
• Integrated contingency handling
• Not necessarily sound or complete

9
Procedure representation in Apex

• concurrency

(procedure (index (hold-altitude using mcp))
(profile right-hand) (step s1 (clear
right-hand)) (step s2 (find-loc
alt-hold-button gt ?loc)) (step s3
(press-button ?loc right-hand) (waitfor
(empty right-hand) (location
alt-hold-button ?loc))) (step end (terminate)
(waitfor (illuminated alt-hold-button))
(step aux1 (restart ?self) (waitfor
(resumed ?self))))
10
Procedure representation in Apex

• concurrency
• reactivity / cl-control

(procedure (index (hold-altitude using mcp))
(profile right-hand) (step s1 (clear
right-hand)) (step s2 (find-loc
alt-hold-button gt ?loc)) (step s3
(press-button ?loc right-hand) (waitfor
(empty right-hand) (location
alt-hold-button ?loc))) (step end (terminate)
(waitfor (illuminated alt-hold-button))
(step aux1 (restart ?self) (waitfor
(resumed ?self))))
11
Procedure representation in Apex

• concurrency
• reactivity / cl-control
• hierarchy / selection

(procedure (index (hold-altitude using mcp))
(profile right-hand) (step s1 (clear
right-hand)) (step s2 (find-loc
alt-hold-button gt ?loc)) (step s3
(press-button ?loc right-hand) (waitfor
(empty right-hand) (location
alt-hold-button ?loc))) (step end (terminate)
(waitfor (illuminated alt-hold-button))
(step aux1 (restart ?self) (waitfor
(resumed ?self))))
12
Procedure representation in Apex

• concurrency
• reactivity / cl-control
• hierarchy / selection
• contingency-handling

(procedure (index (hold-altitude using mcp))
(profile right-hand) (step s1 (clear
right-hand)) (step s2 (find-loc
alt-hold-button gt ?loc)) (step s3
(press-button ?loc right-hand) (waitfor
(empty right-hand) (location
alt-hold-button ?loc))) (step end (terminate)
(waitfor (illuminated alt-hold-button))
(step aux1 (restart ?self) (waitfor
(resumed ?self))))
13
Procedure representation in Apex

• concurrency
• reactivity / cl-control
• hierarchy / selection
• contingency-handling
• multitask management

(procedure (index (hold-altitude using mcp))
(profile right-hand) (step s1 (clear
right-hand)) (step s2 (find-loc
alt-hold-button gt ?loc)) (step s3
(press-button ?loc right-hand) (waitfor
(empty right-hand) (location
alt-hold-button ?loc))) (step end (terminate)
(waitfor (illuminated alt-hold-button))
(step aux1 (restart ?self) (waitfor
(resumed ?self))))
14
Execution-time action-selection using stored
procedures

• Execution-time action selection
• Copes well with uncertainty
• Reduced deliberation
• Stored procedures
• Natural representation of SOPs
• Easy to construct, debug, evaluate
• Fast decisions
• Can adapt to statistical structure of env by
  repeated use
• Can adapt environment to make procedures more
  effective
• Easy to communicate to other agents
• - Not necessarily sound or complete
• Successful procedure-based behavior requires
  multitasking

15
Multitask management examples

• Delay answering phone until finished typing
  sentence
• heuristic prefer to delay interrupt until
  good stopping point
• Pull over to side of road before studying map
• Drive back onto road (but dont drive to start
  point)
• Do something useful when stopped at a red light

Multitask management execution-time coordination
of interacting tasks based on task-specific
characteristics and general heuristics.
16
Challenge
Support creation of agents with human-level
ability to employ diverse multitask management
tactics

• General heuristics underlying tactics
• architecture mechanisms
• Task-specific knowledge
• specialized representation elements
• task representation methodology

17
Multitasking in Apex

• Concurrency control
• Interruption, resumption, interleaving
• Efficient use of resources

18
Concurrency Control PDL idioms
Race (procedure (index (do-it)) (step s1
(do-A) (step s2 (do-B) (step s3 (do-C)
(waitfor ?s1) (waitfor ?s2)) (step s4
(terminate) (waitfor ?s3)))
Synchronize (procedure (index (do-it))
(step s1 (do-A)) (step s2 (do-B)
(waitfor (started ?s1))) (step s3 (terminate)
(waitfor ?s1 ?s2)))
Converge (procedure (index (do-it)) (step
s1 (do-A) (step s2 (do-B) (step s3 (do-C)
(waitfor ?s1 ?s2) (step s4 (terminate)
(waitfor ?s3)))
19
Interruption and resumption
Determining if tasks conflict

• Resource requirements declared in procedure
• (profile (ltresourcegt tolerance) )
• (hold-resource ltresourcegt) tolerance)
• (release-resource ltresourcegt tolerance)
• Some tasks tolerate brief interruptions
• Conflict exists between tasks A and B if
• A and B both require resource R, and
• Expected Duration (A) gt Tolerance (B)
• or Expected Duration (B) gt Tolerance (A)

20
Interruption and resumption
Resolving task conflicts

• Compute priority from task attributes
  situational factors
• urgency (U) measure of time until deadline
• importance (I) expected cost of missing deadline
• subjective workload (S) measure of task crowding
• interrupt cost (IC) expected cost of
  interruption
• In low workload, prefer urgent tasks (avoid
  unnecessary loss)
• In high workload, prefer important tasks (avoid
  most
• expensive losses)

21
Interruption and resumption
Resolving task conflicts

• Resource conflicts resolved using priority
  heuristic
• Priority information set with step-level PDL
  clauses
• (step
• (priority lturgencygt ltimportancegt)
• (interrupt-cost ltcostgt)

22
Interruption and resumption Transition
behaviors
(procedure (index (fly-cruise-leg using
manual-control)) (step s1 (maintain-altitude)
(interrupt-cost 5)) ... (step s12
(handoff-to-pilot-not-flying) (priority
(importance 10) (urgency 10))) (waitfor
(interrupted ?self))) (step s13
(monitor-pilot-not-flying) (waitfor
(completed ?s12))) (step s14 (request-role-pilot
-flying) (waitfor (resumed ?self)))
...)
23
Efficient use of resources

• Combine redundant tasks
• (merge ltconditiongt lttask patterngt)
• Exploit slack in procedure resource use
• Automatic detection of slack intervals
• Integrated dispatch scheduling
• Concurrent recursive decomposition gt tasks
• Priority-based allocation gt schedule

24
Multitask Management Summary

• Multitasking ability founded on tactical
  knowledge derived from general heuristics
• Reactive planners can be extended to execute
  these heuristics in uncertain/time-pressured
  environments
• Specifically, extensions for concurrency control,
  interruption handling and resource management
  facilitate use of multitasking tactics
• Understanding of what needs to be represented and
  what notation is best for this purpose are
  improving as new Apex applications are developed

25
Motivation and Learning Work in progress

• Intent inference for multitasking agents
• use procedural knowledge in reverse
• distinguish separate threads of behavior
• initial approach in Freed and Dahlman, 2002
• Learning aggregate properties of procedures
• learning duration and resource characteristics
• will enable better dispatch scheduling
• other statistics useful for handling
  contingencies
• such as failure and interruption

26
Reducing time/expertise to construct models

• Usability efforts
• High-level language

27
Behavior representation usability
Intuitive (step s3 (stop) (waitfor (shape ?x
light) (color ?x red))) conjunctive
preconditions mutually constraining Expressive
(step s4 (slow) (waitfor (color ?x green) then
(color ?x yellow)) able to express temporal
relations between preconditions Compact
abbreviated form for sequential
procedures Evolves as users needs become better
understood Support tools exist for authoring and
debugging
28
Reducing time/expertise to construct models

• Usability efforts
• High-level language
• Visualization/debug tools

29
Visualizing and Debugging

• Sherpa
• Simulation traces
• Physical environment
• PERT charts
• VISTA
• Enhanced Sim Trace
• (2001 CMU student project)

30
Reducing time/expertise to construct models

• Usability efforts
• High-level language
• Visualization/debug tools
• World modeling

31

• Specifying a Physical World Model

Time-consuming and error-prone when done by hand
32
Specifying a Physical World Model
Easy with drag-and-drop interface
CMU 2002 student project
33
Reducing time/expertise to construct models

• Usability efforts
• High-level language
• Visualization/debug tools
• World modeling
• Interoperability

34
Interoperability
Apex interops with Riptide X-Plane DOMS Mozilla A
MBR (HLA) Reusable elements Apex API Sim
compatibility Com support
Apex pilot flies an F-16 over NASA Ames
35
Reducing time/expertise to construct models

• Usability efforts
• High-level language
• Visualization/debug tools
• World modeling
• Interoperability
• Reusable components

36
Resuable building-blocks
(procedure (index (fast-move-click-R-hand-on-mou
se target ?target)) (step c1
(initiate-move-cursor ?target)) (step m1
(move-cursor ?target) (waitfor ?c1)) (step c2
(attend-target ?target)) (step c3
(initiate-eye-movement ?target) (waitfor ?c2))
(step m2 (eye-movement ?target) (waitfor ?c3))
(step p1 (perceive-target-complex ?target))
(step c4 (verify-target-position ?target)
(waitfor ?c3 ?p1)) (step c5 (initiate-click
?target) (waitfor ?c4 ?m1)) (step m3
(mouse-down ?target) (waitfor ?m1 ?c5)) (step
m4 (mouse-up ?target) (waitfor ?m3)) (step t
(terminate) (waitfor ?m4 ?rvr1 ?rvr2)))
37
Reducing time/expertise to construct models

• Usability efforts
• High-level language
• Visualization/debug tools
• World modeling
• Interoperability
• Reusable components
• Distributed development

38
Distributed Development

• Why needed
• Library of building blocks
• Resource models
• Software interfaces
• Visualization tools
• Making it happen
• Software infrastructure
• Educational outreach
• Web-based support

Apex collaborative web site (CMU HCI project
2001)
39
Apex Modeling Tool

• Apex System
• Apex architecture (autonomy)
• Human resource architecture
• Reusable procedure lib
• World model widget libs
• Simulation engine
• GUI-based vis/debug tools
• GUI-based world dev tool
• Interoperability framework
• Manuals, tutorials,

Download system from http//human-factors.arc.nas
a.gov/apex/ User Support apex-help_at_eos.arc.nasa.
gov
40
Apex 2.3
Apex is available at human-factors.arc.nasa.gov/
apex
41
How PDL specifies interleaving

• CPM-GOMS requires that actions from templates
  earlier
• in a template sequence have priority in resource
  conflicts
• over activities from later templates.
• Initial approach used the priority clause
• determines how to resolve resource conflict
• globally scoped to allow comparison of any tasks
• (step s2 (turn off alarm)
• (waitfor (on alarm))
• (priority 5))

42
How PDL specifies interleaving
Global scope of priority value assignments
undesirable for CPM-GOMS representations
(procedure (procedure (index (get ?amt from
atm)) (index (init atm transaction)) (step
s1 (init atm transaction) (step s1 (insert
card) (priority 3000)) (priority 3200))
(step s2 (withdraw ?amt)) (step s2 (enter
password) (priority 2000))
(priority 3100)) (step s3 (end atm
transaction) (step s3 (terminate) (priority
1000)) (waitfor ?s2))) (step s4
(terminate) (waitfor ?s3)))
Requires modeler to anticipate decomposition
Complex procedure to set priority values
43
How PDL specifies interleaving
Solution alternative mechanisms/syntax for
resolving resource conflicts that is dynamically
scoped
(procedure (procedure (index (get ?amt from
atm)) (index (init atm transaction)) (step
s1 (init atm transaction) (step s1 (insert
card) (rank 1)) (rank 1)) (step s2
(withdraw ?amt)) (step s2 (enter
password) (rank 2)) (rank
2)) (step s3 (end atm transaction) (step s3
(terminate) (rank 3))
(waitfor ?s2))) (step s4 (terminate) (waitfor
?s3)))
44
How PDL specifies interleaving
Abbreviation to further simplify syntax
(procedure ranked (procedure ranked (index
(get ?amt from atm)) (index (init atm
transaction)) (step s1 (init atm transaction)
(step s1 (insert card) (step s2 (withdraw
?amt)) (step s2 (enter password) (step s3
(end atm transaction) (step s3 (terminate)
(step s4 (terminate) (waitfor ?s3)))
(waitfor ?s2)))
45
FAST MOVE-CLICK
- from Gray and Boehm-Davis (2000)
46
Interleaving Templates
0
0
new-cursor-location
new-cursor-location
290
290
perceive- target
perceive- target
50
50
50
50
50
50
START initiate-move- cursor
50
attend- target
initiate- POG
verify- target pos
50
START initiate-move- cursor
attend- target
initiate- POG
verify- target pos
initiate- mouseDn
initiate- mouseDn
590 (FL)
182 (FL)
100
100
100
mouseDn
100
mouse Up
move- cursor
mouseDn
mouse Up
move- cursor
FAST M/C
FAST M/C
30
30
POG
POG
last vision action in the previous operator
47
Autonomous Helicopter ApplicationProject Lead
Matt Whalley (Army)Autonomy Lead Michael Freed
(NASA/IHMC)
Intelligent recon/surveillance Goal maximize
information return Task decide where to go
next
Ariel (Yamaha RMAX) Payload 66 lbs. Flight
duration 60 min
48
Recon/surveillance target selection

• Utility of target selection
• Set of target sites s ? S
• Observation time-cost
• distance
• flight dynamics
• required observe action
• Criterial events ei,s ? Es
• baseline p(ei,s ) and mods
• cumulative cost(ei,s ,t)

• Approach multimethod
• search for small target sets
• for larger sets, method depends
• on uniformity/clumpiness
• spatial clumps
• cost and deadline clumps
• Research Area resource mgt.
• Target selection is special case
• of allocating unary resources
• under time-pressure/uncertainty.

Best choice min opportunity cost
49
Apex Architecture
Execution-time action-selection using stored-plans
Exec-time Action-Selection High uncertainty
tolerance - Little time to deliberate Stored
Plans Fast decisions - Not sound or complete
Adapted to stat struc of env Mutual evolution
with env SOP compliance Easy to construct,
debug Amenable to expert evaluatn More
predictable behavior