A Hierarchical Approach to Modelbased Reactive Planning in Large State Spaces

1
A Hierarchical Approach to Model-based Reactive
Planning in Large State Spaces
Brian C. Williams
Joint with Seung H. Chung
2
Outline

• Model-based programming
• A Simple model-based executive (Livingstone)
• The need for model-based reactive planning
• The Burton model-based reactive planner

3

• Polar Lander Leading Diagnosis
• Legs deployed during descent.
• Noise spike on leg sensors latched by software
  monitors.
• Laser altimeter registers 50ft.
• Begins polling leg monitors to determine touch
  down.
• Latched noise spike read as touchdown.
• Engine shutdown at 50ft.

• Mars Mission Failures, 2000
• Climate Orbiter
• Polar Lander

Objective Embedded languages that reason from
hardware models. (Reactive Model-based
Programming)
4
Model-based Programs Interact Directly with State

• Model-based programs interact with plant state
• Read state
• Write state

• Embedded programs interact withplant sensors and
  actuators
• Read sensors
• Set actuators

Problem Programmer must must map between state
and sensors/actuators.
Solution Model-based executive maps between
state and sensors/actuators.
5
(No Transcript)
6
Orbital Insertion Example
Turn camera off and engine on
EngineA
EngineB
Science Camera
7
Model-based Program Evolves Hidden State

• Reactive Model-based Programming Language
• Asserts state
• Queries state
• Executes conditionally
• Preempts
• Iterates
• Executes concurrently

Programmer specifiesabstract state evolutions
Temporal planner
Programmer specifies plant model

• Model specifies
• Mode transitions
• Mode behavior

goals
State
Model-based Executive
Command
Observations
8
Model-based Executive Reasons from Plant Model
Goal Achieve Thrust
Temporal planner
State Goals
State Estimates
Model-based Executive
Commands
Observations
9
Model-based Executive Reasons from Plant Model
Goal Achieve Thrust
Temporal planner
goals
State
Model-based Executive
Command
Observations
10
Outline

• Model-based programming
• A Simple model-based executive (Livingstone)
• The need for model-based reactive planning
• The Burton model-based reactive planner

11
A simple model-based executive (Livingstone)
commanded NASAs Deep Space One probe
Started January 1996 Launch October 15th,
1998 Remote Agent Experiment May, 1999
courtesy NASA JPL
12
Livingstone Williams Nayak, AAAI96
State goals
State estimate
Model
Mode Reconfiguration
Mode Estimation
Flight System Control
Command
RT Control Layer
Observations
13
Thrust
Reconfigure modes to meet goals
Estimate current likely Modes
State estimate
State goals
Model
Mode Selection
Mode Estimation
Command
Observations
Flight System Control
RT Control Layer
14
Mode Selection Select a least cost set of
allowed component modes that entail the current
goal, and are consistent
Mode Estimation Select a most likely set of
component mode transitions that are consistent
with the model and observations
State estimate
State goals
Model
Mode Selection
Mode Estimation
arg max Pt(m) s.t. M(m) O(m) is consistent
arg min Ct(m) s.t. M(m) entails G(m) s.t.
M(m) is consistent
Command
Observations
Flight System Control
RT Control Layer
15
OpSat arg min f(x) s.t. C(x) is satisfiable
D(x) is unsatisfiable
State estimate
State goals
Model
Mode Selection
Mode Estimation
arg max Pt(m) s.t. M(m) O(m) is satisfiable
arg min Ct(m) s.t. M(m) entails G(m) s.t.
M(m) is satisfiable
Command
Observations
Flight System Control
RT Control Layer
16
Outline

• Model-based programming
• A simple model-based executive (Livingstone)
• The need for model-based reactive planning
• The Burton model-based reactive planner

17
DS 1 Attitude Control System

• Livingstone reconfigured modes using one step
  commands. But How does the flight computer
  really open a valve?
• Requires turning on device drivers
• Requires repairing bus controllers
• Sending commands
• Powering down devices . . .

18
How do we reconfigure a valve?
Remote Terminal
Driver
Valve
Bus Control
Computer
Remote Terminal
Driver
Valve

• Device modes are changed through indirect
  commanding.
• Communication paths are established by
  reconfiguring other devices.
• The task of reconfiguring devices in the proper
  order generalizes state-space planning to handle
  indirect effects.
• to achieve reactivity the all possible plans for
  all possible goal states should be pre-compiled
  (a generalization of universal plans).
• To achieve compactness we decompose these
  universal plans according to a goal/sub-goal
  hierarchy.

19
Model-based Execution Reactive PlanningBurton
Williams Nayak, IJCAI97
State goals
State estimate
Model
Mode Selection
Mode Estimation
Command
Observations
Flight System Control
RT Control Layer
20
Example Driver Valve Command Sequence
Valve vlv
Valve Driver dr
Goal No thrust
vcmdin
dcmdin
Commands Driver State Valve State ME dr
off, vlv open MS dr off, vlv
closed MRP dcmdin on ME dr on, vlv
open MRP dcmdin close ME dr reset failure,
vlv open MRP dcmdin reset ME dr on,
vlv open MRP dcmdin off ME dr off,
vlv open
21
Model-based Reactive Planning Execution

• Limitation of configuration management
• Reactive Planning
• Model compilation
• Reversible Planning
• Constructing Hierarchical Policies
• Execution

22
To achieve reactivity we eliminate all forms of
search.
23
Model-based Reactive Planning

• Achieved by
• Eliminate Indirect Control . . .
  through Compilation
• Eliminate Search for Goal Ordering
  . . . through Reversibility and Serialization
• Eliminate Search to find Suitable Transitions
  . . . by Constructing Hierarchical
  Polices

24
Model-based Reactive Planning

• Achieved by
• Eliminate Indirect Control . . .
  through Compilation
• Eliminate Search for Goal Ordering
  . . . through Reversibility and Serialization
• Eliminate Search to find Suitable Transitions
  . . . by Constructing Hierarchical
  Polices

25
To Handle Indirect Control . . .
dcmdout vcmdin
flowin
vcmdin
dcmdin
flowout
26
. . . Compile Out Constraints
dcmdout vcmdin
dcmdin dcmdout
inflow outflow
inflow outflow
dcmdin reset
stuck open
open
on
resettable
dcmdin off
dcmdin on
vcmdin close
vcmdin open
dcmdin off
stuck closed
closed
off
failed
flowin
vcmdin
dcmdin
27
. . . Compile Out Constraints
dcmdin reset
stuck open
open
on
resettable
dcmdin off
dcmdin on
driver on
driver on
dcmdin off
dcmdin close
dcmdin open
stuck closed
closed
off
failed
dcmdin
28
To Compile Out Constraints

• Eliminate intermediate variables.
• Transitions are conditioned on mode and control
  variables
• Generate transitions as prime implicates Fi
  Þ next(yi ei)where Fi is a conjunction of
  mode and control variable assignments.
• Prime implicates for transitions enumerated using
  OpSAT
• 40 seconds on SPARC 20 for 12,000 clause
  spacecraft model.

29
Model-based Reactive Planning

• Achieved by
• Eliminate Indirect Effects . . .
  through Compilation
• Eliminate Search for Goal Ordering
  . . . through Reversibility and Serialization
• Eliminate Search to find Suitable Transitions
  . . . by Constructing Hierarchical
  Polices

30
Why Search is Needed

• 1) An achieved goal can be clobbered by a
  subsequent goal.

• Achieve Valve goal before Driver goal

31

• Note Component schematics tend not to have loops

Remote Terminal
Valve
Driver
Bus Control
Computer
Remote Terminal
Valve
Driver

• Define Causal Graph G of compiled transition
  system S
• vertices are state variables.
• edge from vi to vj if vjs transition is
  conditioned on vi.

• Requirement The causal graph is acyclic.

• Work conjunctive goals upstream from outputs to
  inputs

32
Solution

• The only variables used to set some variable (y7)
  is its ancestors,
• ? y7 can be changed without affecting its
  descendants.

• Safe to achieve goals in an upstream order.
• Simple check
• Number causal graph depth first
• achieve goals in order of increasing depth first
  number.

33
Why Search is Needed

• 2) Two goals can compete for the same variable in
  their subgoals.

Switch
1
Latch1
data
2
Latch2

• Example
• Latch1 and Latch2 compete for the position of
  Switch if achieved concurrently.

34

• Sibling goals (7,4) may both need shared
  ancestors.

Unaffected
Not Shared
13
10
5
11
6
12
7
2
8
3
Shared
9
4
1

• Solution Solve one goal before starting next
  sibling (Serialization).
• Feature Generates first control action of plan
  first!

35
Why Search is Needed

• 3) A state transition of a subgoal variable has
  irreversible effect.

Switch
1
Latch1
data
2
Latch2

• Example
• Assume Switch can be used once,
• Then Latch1 must be latched before Latch2.

• But irreversible effects arent desirable for
  reactive planners
• Dont allow irreversible actions
• . . . Except to repair failure modes

36
Solution Mark Allowed Transitions/Assignments
1
2
3
Driver
Valve
dcmdin
dcmdin open

• Mark all control variable assignments allowed

37
Solution Mark Allowed Transitions/Assignments
1
2
3
Driver
Valve
dcmdin
dcmdin reset
stuck open
open
on
resettable
dcmdin off
dcmdin on
driver on
driver on
dcmdin off
dcmdin close
dcmdin open
stuck closed
closed
off
failed

• Mark all control variable assignments allowed

• For each mode variable v, in decreasing order of
  DF number
• Select each transition of v, whose guard has
  only allowed assignments.

38
Solution Mark Allowed Transitions/Assignments
1
2
3
Driver
Valve
dcmdin
dcmdin reset
stuck open
open
on
resettable
dcmdin off
dcmdin on
driver on
driver on
dcmdin off
dcmdin close
dcmdin open
stuck closed
closed
off
failed

• Mark all control variable assignments allowed

• For each mode variable v, in decreasing order of
  DF number
• Select each transition of v, whose guard has
  only allowed assignments.

• Given current assignment v I for v

• Find strongly connected component of selected
  transitions that contains I.

• Mark assignments and transitions in SCC allowed.

39
Solution Mark Allowed Transitions/Assignments
1
2
3
Driver
Valve
dcmdin
stuck open
open
on
dcmdin off
dcmdin on
driver on
driver on
dcmdin close
dcmdin open
stuck closed
closed
off

• Mark all control variable assignments allowed

• For each mode variable v, in decreasing order of
  DF number
• Select each transition of v, whose guard has
  only allowed assignments.

• Given current assignment v I for v

• Find strongly connected component of selected
  transitions that contains I.

• Mark assignments and transitions in SCC allowed.

40
Solution Mark Allowed Transitions/Assignments
1
2
3
Driver
Valve
dcmdin
stuck open
open
on
dcmdin off
dcmdin on
driver on
driver on
dcmdin close
dcmdin open
stuck closed
closed
off

• Mark all control variable assignments allowed

• For each mode variable v, in decreasing order of
  DF number
• Select each transition of v, whose guard has
  only allowed assignments.

• Given current assignment v I for v

• Find strongly connected component of selected
  transitions that contains I.

• Mark assignments and transitions in SCC allowed.

41
Solution Mark Allowed Transitions/Assignments
1
2
3
Driver
Valve
dcmdin
open
on
dcmdin off
dcmdin on
driver on
driver on
dcmdin close
dcmdin open
closed
off

• Mark all control variable assignments allowed

• For each mode variable v, in decreasing order of
  DF number
• Select each transition of v, whose guard has
  only allowed assignments.

• Given current assignment v I for v

• Find strongly connected component of selected
  transitions that contains I.

• Mark assignments and transitions in SCC allowed.

42
Model-based Reactive Planning

• Achieved by
• Eliminate Indirect Effects . . .
  through Compilation
• Eliminate Search for Goal Ordering
  . . . through Reversibility and Serialization
• Eliminate Search to find Suitable Transitions
  . . . by Constructing Hierarchical
  Polices

43
Solution
open
driver on
driver on
cmd close
cmd open
closed

• Convert automata into hierarchical policies, one
  per automaton

• Policy selects first transition towards achieving
  each automata goal state, given current state.
• Policy maps goals to subgoals and commands, in
  proper order
• Ensures only reversible transitions are taken,by
  only using transitions marked allowed.

44
Plan by passing sub-goals up causal graph
Goal Driver off, Valve closed
Current Driver off, Valve open
1
Valve
Driver
2
Goal
Goal
Current
Current
Open
On
Closed
Off
idle
driver on cmd close
idle
cmd off
Open
On
driver on cmd open
idle
cmd on
idle
Closed
Off
fail
fail
cmd reset
cmd off
Stuck
Resettable
45
Plan by passing sub-goals up causal graph
Goal Driver off, Valve closed
Current Driver off, Valve open
1
Valve
Driver
2
Goal
Goal
Current
Current
Open
On
Closed
Off
idle
driver on cmd close
idle
cmd off
Open
On
driver on cmd open
idle
cmd on
idle
Closed
Off
fail
fail
cmd reset
cmd off
Stuck
Resettable
46
Plan by passing sub-goals up causal graph
Goal Driver off, Valve closed
Current Driver off, Valve open
1
Valve
Driver
2
Send cmd on
Goal
Goal
Current
Current
Open
On
Closed
Off
idle
driver on cmd close
idle
cmd off
Open
On
driver on cmd open
idle
cmd on
idle
Closed
Off
fail
fail
cmd reset
cmd off
Stuck
Resettable
47
Plan by passing sub-goals up causal graph
Goal Driver off, Valve closed
Failed Resettable
Current Driver resettable, Valve open
1
2
Valve
Driver
Goal
Goal
Current
Current
Open
On
Closed
Off
idle
idle
cmd off
driver on cmd close
Open
On
driver on cmd open
idle
idle
cmd on
Closed
Off
fail
fail
cmd reset
cmd off
Stuck
Resettable
48
Plan by passing sub-goals up causal graph
Goal Driver off, Valve closed
Current Driver resettable, Valve open
1
Valve
Driver
2
Goal
Goal
Current
Current
Open
On
Closed
Off
idle
idle
cmd off
driver on cmd close
Open
On
driver on cmd open
idle
cmd on
idle
Closed
Off
fail
fail
cmd reset
cmd off
Stuck
Resettable
49
Plan by passing sub-goals up causal graph
Goal Driver off, Valve closed
Current Driver resettable, Valve open
1
Valve
Driver
2
Send cmd reset
Goal
Goal
Current
Current
Open
On
Closed
Off
idle
idle
cmd off
driver on cmd close
Open
On
driver on cmd open
idle
cmd on
idle
Closed
Off
fail
fail
cmd reset
cmd off
Stuck
Resettable
50
Plan by passing sub-goals up causal graph
Goal Driver off, Valve closed
Current Driver on, Valve open
1
Valve
Driver
2
Send cmd close
Goal
Goal
Current
Current
Open
On
Closed
Off
idle
driver on cmd close
idle
cmd off
Open
On
driver on cmd open
idle
cmd on
idle
Closed
Off
fail
fail
cmd reset
cmd off
Stuck
Resettable
51
Plan by passing sub-goals up causal graph
Goal Driver off, Valve closed
Current Driver on, Valve closed
1
Valve
Driver
2
Send cmd off
Goal
Goal
Current
Current
Open
On
Closed
Off
idle
driver on cmd close
idle
cmd off
Open
On
driver on cmd open
idle
cmd on
idle
Closed
Off
fail
fail
cmd reset
cmd off
Stuck
Resettable
52
Plan by passing sub-goals up causal graph
Success
Goal Driver off, Valve closed
Current Driver off, Valve closed
1
Valve
Driver
2
Goal
Goal
Current
Current
Open
On
Closed
Off
idle
driver on cmd close
idle
cmd off
Open
On
driver on cmd open
idle
cmd on
idle
Closed
Off
fail
fail
cmd reset
cmd off
Stuck
Resettable
53
Hierarchical, Model-based Reactive Planning

• Compile-time Analysis
• Compile-out interactions
• Confirm schematics are loop free.
• Depth first number variables.
• Periodic, Run-time Analysis
• Given initial state
• Identify allowed transitions and assignments
• Given autonomous jump to failure state
• Identify allowed transitions and assignments
• Run-time Plan Execution
• Work conjunctive goals from outputs to inputs.
• Achieve goals serially.
• Only perform reversible transitions.
• Lookup control actions and sub-goals in policies

54
Complexity of Reactive Planning

• Worst Case per action Depth Sub-goal branch
  factor
• Average Cost per action Sub-goal branch factor

Valve1 open
Valve2 open
Driver1 off
Driver2 off
Driver1 on
Driver2 on
CU on
CU on
CU on
CU on
CU on
CU on
55
What If Plan is Not Serializable?

• What if causal graph G contains cycles?
• Solution
• Isolate the cyclic components (compute SCCs)

• compose each cycle into a single component.

• New causal graph G is acyclic,
• Goals of G are serializable

56
Composing Cyclic Components
Transmitter
Amplifier
on
on
A off
A off
T on
cmdA off
cmdT off
cmdT on
cmdA on
off
off
57
Policy for Composed Components

• Problem Composition grows exponential in space
  usage.
• Solution Use BDD encoding (in progress).

cmdA off
onT onA
onT offA
cmdA on
cmdT off
cmdT on
cmdA off
offT offA
offT onA
58
Model-based Reactive Planning

• Compile away constraints from the model
• Compile away cyclic components
• Plan serially pursuing causal graph upstream
• Generate actions using hierarchical policies
• Only performs reversible actions
• Responds to failure at each step
• Average cost per step subgoal branching factor

59
Current Demonstration Testbeds

• Air Force Tech Sat 21 flight
• NASA NMP ST-7 Phase A
• NASA Mercury Messengeron ground.
• MIT Spheres on Space Station
• NASA Robonaut, X-37, ISPP
• Multi-Rover Testbed
• Simulated Air Vehicles

60
Model-based Programming of Embedded Systems

• To survive decades embedded systems orchestrate
  complex regulatory and immune systems.
• Future systems will be programmed with
  models,describing themselves and their
  environments.
• Runtime kernels will be agile, deducing and
  planning by solving optimization problems with
  propositional constraints.
• Model-based reactive planners respond quickly to
  failure, while using compile-time analysis of
  structure to respond quickly and concisely to
  indirect effects.