Optimal Control and Stability of Stochastic Hybrid Systems

1
Optimal Control and Stability of Stochastic
Hybrid Systems

• Alessandro Abate, Shankar Sastry
• University of California at Berkeley

2
What are Hybrid Systems?

• Dynamical systems with interacting continuous and
  discrete dynamics

3
Deterministic Hybrid Systems

• A set of discrete states
• Associated with each discrete state k
• invariant set (domain) an open subset Uk of Rn
• dynamics an ODE (or a control system) on Uk
• guards subsets of Uk, or temporal switches
• Associated with each guards,
• discrete transition to (more than) one new
  discrete state
• continuous transition (reset condition)
• Hybrid executions (St,Xt), t?0

4
Stochastic Hybrid Systems (1)

• A set of discrete states and open domains
• Dynamics inside each domain governed by a SDE
• Stop upon hitting domain boundary
• Boundary of each domain is partitioned into
  guards
• Deterministically jump to a new discrete state
  according to the guard
• Reset randomly in the new domain

5
Stochastic Hybrid Systems (1)
6
Stochastic Hybrid Systems (2)

• A set of discrete states and open domains
• Dynamics inside each domain governed by an ODE
• Jump upon hitting domain boundary, or with a
  condition in time
• Switch to a new discrete state according to a
  Transition Probability Matrix
• Reset inside the new domain corresponding to a
  reset function

7
Stochastic Hybrid Systems (2)
8
General Stochastic Hybrid Systems

• State variables
• continuous variable X
• discrete variable S
• System dynamics
• continuous dynamics
• discrete dynamics

X(t) Ordinary or stochastic differential
equations
9
Interdependent Dynamics

• Discrete dynamics ? continuous dynamics
• Continuous dynamics ? discrete dynamics

dX(t)/dtf(X,S,t)?(X,S,t)dW(t)

• Different continuous dynamics for different S
• Reset of X depends on the discrete transitions

Transition matrix P pij(X)ij1,,K

• Transition probabilities depend on X
• Time between jumps depends on X

10
Optimal Control for Stochastic HS

• Our Approach dynamics according to ODEs,
  underlying Markov Chain
• Each state i has a reward ri
• An input ui can be applied, at some cost gi(ui),
  to reach the boundary with time hi(ui)ltT
• Temporal transitions (every T ), spatial if u is
  applied
• Consider an NT Time Horizon case
• Object Maximize Expectation of

11
Optimal Control for Stochastic HS

• 2 Analyses Infinite Time Horizon, and Finite.
• Infinite Time Horizon E(R) can be expressed in a
  deterministic, closed form
• Dependence on the steady-state distribution p of
  the MC

12
Optimal Control for Stochastic HS

• Finite Time Horizon Bellman-like Approach -gt
  computational complexity
• But, assuming we are in steady state,
• Main Result the finite-time horizon analysis has
  the same optimum as the infinite-time horizon
  one.
• This implies mathematically, no dependence on
  final cost
• practically, drastic decrease in
  complexity.
• Algorithms for the selection of the optimal
  control have been proposed.

13
Motivational Example

• Productivity Allocation
• Product pi, manufacture cost ci, price ri
• Market demand oi .
• Choice of production increase (control) w
• production time hi(w), 0lthi(w)ltT
• additional cost gi(w), cltgi(w).
• State i production of pi
• Switching probabilities oi /Ss os.

14
Stability of Stochastic HS

• Dynamics ODEs, possibly nonlinear (flows have
  bounded Lipschitz constant)
• Underlying Markov Chain
• Temporal transitions (every T time)
• Single Equilibrium q shared among all domains
• Reset maps with bounded Lip constant

15
Stability of Stochastic HS

• Stability in Probability q is (asymptotically)
  stable in prob. if, for every D, q e D, there
  exists a region E, included in D, s.t. the hybrid
  flow starting in any point in D will end up
  evolving in E, as time goes to infinity, with
  Probability 1.
• Assumptions
• n Domains
• Vector fields fi -gt flows ji
• Reset maps Rij
• Steady-state distribution p p1,..., pn

16
Stability of Stochastic HS

• Theorem
• Define
• n Pi1,..,nLip(jiT)pi
• m Pi,j1,..,nLip(Rij )pi Pij
• If n m lt1, then equilibrium q is stable in
  probability (sufficient condition).
• Extension to switches in time according to
  exponential distributions.

17
Stability of Stochastic HS

• Simulation HS with 5 nodes, linear vector
  fields, reset maps are the identity, jumps at
  fixed times.

18
Applicative Example

• Stocks Pricing Market has fixed number of
  equities (n), with an equilibrium price
• X of stockholders willing to buy 1 title
• Y of operators willing to sell 1 title

3 regions Equilibrium, Overpricing,
Depreciation
19
Applicative Example

• Every time T, one transaction can be made.
• Model of the Market?
• Given starting domain (status of the market) and
  equities value, prediction of the long-term
  dynamics of the stocks prices

20
Future Work

• Optimal Control
• More complex dynamics, general reset maps
• Further applications, most likely in biology
• Reward w.r.t. the systems equilibrium gt
  relation with the Stability results.
• Stability
• Extension to more equilibria per domain
• Investigate other kinds of stability.