Context-Dependent Network Agents

1
Context-Dependent Network Agents
Context-Dependence
Learning
Objective Improve agility and robustness
(survivability) of large dynamic networks through
agents that are
Collaboration
Agent Network

• widely distributed
• context-dependent
• semi-autonomous
• collaborative

• multi-modal
• self-improving
• local in sensing influence
• multi-objective

Sensing Control
Concept
Physical Network

• Specific technology goals of funded effort
• distributed computation and control
• applications of synchronized sampling
• collaboration techniques
• Accomplishments
• distributed rolling horizon strategies
• protocols for dynamic collaboration
• Markov modeling and multi-mode learning
• context-dependent FACTS and PSS controls
• dependable secure protective relaying
  strategies
• distributed power flow
• evaluation of reactive power control in
  deregulated markets
• remote-access real-time control emulator
• Next steps
• context detection and identification
• integrated CDNA strategies
• CDNA modeling, simulation and validation
• extensions to computer, traffic, biological, and
  C2 networks

• multi-objective hybrid strategies
• learning, diagnostics and adaptation
• real-time infrastructure

PI Bruce H. KroghDept. of ECE, Carnegie Mellon
University5000 Forbes AvenuePittsburgh, PA
15213-3890ph. 1 412 268 2472 fax -3890e-mail
krogh_at_ece.cmu.edu Contract Number
WO8333-05 Start and duration of funded effort
Jan. 1, 1999 through Dec. 31, 2003
Goals/Progress/Directions
Programmatic Information
SKIP
2
CDNA Progress by Subtask

• Task 1 Agent Templates and Modules
• Subtask 1.1 CDN Agent Template
• Implemented and studied structures for realizing
  various agent behaviors and capabilities for
  specific network control scenarios (CMU, RPI,
  TAMU, UM)
• Subtask 1.2 Module Specifications
• Created initial design of control agent modules
  separated from power system simulation program
  for real-time system emulation (RPI, UIUC)
• Subtask 1.3 Interface Design
• Designed and implemented input-output interfaces
  between MATLAB power system simulator and
  real-time system emulator (RPI, UIUC)
• Subtask 1.4 Tools for CDN Agent Construction
• Evaluated various algorithmic tools for
  constructing agent capabilities (CMU, RPI, TAMU,
  UM)
• Task 2 Restructured Power System Modeling
• Subtask 2.1 Definition of Operating Modes
• Examples of operating modes for TCSC control for
  voltage and stability transients (RPI, TAMU)
• Use of Markov decision models for defining
  operating modes (CMU)
• New method for evaluating safe transient-stability
  operating regimes (CMU)

3
CDNA Progress by Subtask (cont'd.)

• Subtask 2.2 Online Identification of Operating
  Modes
• Analysis of data from phasor measurement units
  (PMU) to identify signatures of disturbances
  (RPI)
• Use of neural nets and synchronized sampling for
  improved dependability/security (TAMU)
• Subtask 2.3 Decomposition and Aggregation
• Convergence results for new distributed load flow
  computations (UM)
• Task 3 Agent Coordination and Learning
• Subtask 3.1 Development of Collaboration
  Strategies
• New system structures for distributed model
  predictive control (CMU)
• Neighbor-coordination schemes in C-Nets
  (collaborative networks) (CMU)
• Subtask 3.2 Learning Algorithms for Coordination
• Game-theoretic formulations and learning for
  multi-agent control (CMU)
• Multi-objective coordination (TAMU)
• Subtask 3.3 Local Control Strategies
• Multimode control of Markov decision processes
  (CMU)
• Congestion control strategies for voltage and
  stability transients (RPI, TAMU)
• Impacts of deregulation market on reactive
  control (TAMU)
• Subtask 3.4 Robust Hybrid Dynamics
• Conditions for certainty equivalents in switching
  control strategies for Markov decision processes
  (CMU)

4
CDNA Progress by Subtask (cont'd.)

• Task 4 Real Time Infrastructure
• Subtask 4.1 Real-time Environment
• Completed demonstration system (Telelab) for
  remote access to the Simplex infrastructure and
  MATLAB Power System Toolbox (RPI, UIUC)
• Subtask 4.2 Robustness Features
• Implemented Telelab for multiple users to
  download and test code (UIUC)
• Task 5 Tests and Demonstrations
• Subtask 5.1 Demonstration Scenarios
• Multi-area scenarios for TCSC voltage and
  transient stability (RPI, TAMU)
• Scenarios for distributed multi-agent model
  predictive control (CMU)
• Multi-area load flow computation scenario (UM)
• Subtask 5.2 Application Simulator
• MATLAB Power System Toolbox-Simplex Telelab (RPI,
  UIUC)
• Subtask 5.3 Visualization Tools
• Evaluated power system simulation tools for
  protective relaying scenario presentation (MATLAB
  and EUROSTAG) (TAMU)
• Task 6 The Virtual Institute
• Subtask 6.1 Customization of Lire
• Upgraded LIRE for faster access and e-mail
  notification services (CMU)
• Subtask 6.2 Computer-Based Collaboration
• Web-based distribution of project reports and
  results (All participants)

5
Overview of CDNA Accomplishments
- collaboration techniques - distributed
computation
multi-objective hybrid strategies
real-time infrastructure
- learning - diagnostics - adaptation
Agent Network
distributed control
applications of synchronized sampling
Physical Network
6
Specific CDNA Accomplishments
remote-access real-time control emulator
distributed- rolling horizon strategies- power
flow
B
context-dependent PSS controls
evaluation of reactive power control in
deregulated markets
A
context-dependent FACTS controls
dependable secure protective relaying strategies
- Markov modeling - multi-mode learning
C
protocols for dynamiccollaboration
7
CDNA Universities and Principal Investigators
8
Current CDNA Collaborations

4. RPI
economic markets decentralized comp. biological
networks
1. U of Minn
Telelab-MATLAB real-time emulator
ABC system switching control implementation
2. CMU
agents arch. collaboration
5. UIUC
distributed power flow
ABC system voltage studies synchronous data
modeling
stability region comp.
3. TAMU
9
Selected Results
10
Transmission System Security Analysis using
Network Agents

• Security analysis is done by running power flows
• We are seeking methods of solving distributed
  power flows using agents (computer systems) in
  multiple control systems
• We would like to eliminate the idea of a
  security center approach.

University of Minnesota
11

• ISO
• Trends
• Getting larger
• Standard data formats
• Less functionality in regional systems

12

• Networked Control Systems
• Region can be any size
• Can extend to any number of regions
• Regions retain original functionality
• Aggregate has same functionality as large area
  control system

13
Poor Results from Multiple Processors solving One
Power Flow

• Divide power system into several areas
• Solve each area on a separate processor
• Communicate results of each processor with other
  processors
• Communication time is greater than time saved by
  using multiple processors
• Try to minimize data that must be sent between
  processors

University of Minnesota
14
Security Analysis and multiple processors

• Security analysis requires solving multiple power
  flows, one for each contingency case
• When calculation on one case is completed, start
  communication
• While communication is being done, start
  calculation on next case

University of Minnesota
15
Multiple processors solving multiple outage cases
calculation overlaps communications
Greatly increased speed
University of Minnesota
16
Methods Tested

• Gauss-Seidel Method
• Filtered Solution
• Block Border Gauss Method
• Conjugate Gradient Methods
• Reduced Orthogonal Subspaces
• Diakoptics

University of Minnesota
17
Results of research at the University of
Minnesota

• Communication is the bottleneck
• Methods with only neighbor to neighbor
  communications require too many iterations to
  solve
• Methods that exchange sensitivities require
  fewer iterations ? Some entity must calculate the
  sensitivities
• We have reduced the sensitivity data that must be
  exchanged to a minimum without sacrificing speed

return
University of Minnesota
18
COORDINATION OF DISTRIBUTED, AUTONOMOUS
AGENTS Sarosh Talukdar, Eduardo Camponogara,
Haoyu Zhou

• ACCOMPLISHMENTS
• Extension of Model Predictive Control (the
  Rolling Horizon Strategy)
• to serve as a coordination framework for
  autonomous, distributed
• agents.
• Development of a test-bed for coordination and
  learning strategies
• in networks of stationary and mobile,
  autonomous, distributed
• agents

Carnegie Mellon University
19
EXTENSION OF MODEL PREDICTIVE CONTROL TO
AUTONOMOUS, DISTRIBUTED AGENTS

• The communication links between agents define a
  set of overlapping
• neighborhoods.

• Neighbors of an agent adjacent agents
• For each agent, the systems variables are
  divided into three sets
• X proximate variables (those variables the agent
  can sense or control)
• Y neighborhood variables (those variables the
  agents neighbors can
• sense or control)
• Z remote variables (all other variables)

Carnegie Mellon University
20
SUFFICIENT CONDITIONS (for the successfull
extension of model predictive control to
distributed, autonomous agents)
If

• the overall-system-problem is feasible
• the overall-system-problem is convex
• the overall-system-problem is decomposed into
  sub-problems for
• the agents, such that each sub-problem matches
  its agent exactly
• (Z is empty for each agent)
• each agent uses an iterative, interior point
  method to solve its
• sub-problem
• each agent communicates the results of each
  iteration to its neighbors
• the agents in each neighborhood work serially
  (one after the other)

Then
the agents iterations will converge to an
optimal solution of the overall-system-problem
Question are these conditions necessary?
Carnegie Mellon University
21
COORDINATION HEURISTICS

• There are at least two families of heuristics by
  which the conditions
• on
• exact matchings of agents to sub-problems
• problem-convexity
• communication frequency within each neighborhood
• serial work within each neighborhood
• can be demonstrated to be unnecessary for
  representative networks.
• These families are based on
• tightening the resource constraints by the
  inclusion of
• resource margins
• learning models by which each agent can predict
  the
• actions of its neighbors

These heuristics allow the agents to work
asynchronously (in parallel, each at its own
speed) on realistic (non-convex) control tasks.
Carnegie Mellon University
22
Carnegie Mellon University
23
Carnegie Mellon University
24
Context Dependent Switching and Learning
An application in the deregulated power market

• Contexts different buyers and sellers
  (decision-makers) with the same
• Objective to develop bidding strategies for
  their own profits.
• So many uncertainties for a decision-maker, G1,
  for example,
• Unobservable infinitely many possible
  combinations of bidding from G2, L1, L2.
• Transmission line capacity variations.

Carnegie Mellon University
25

• Switching
• Using finite number of modes to describe the
  infinitely many possibilities.
• Designing optimal strategy for each mode and
  switching between these optimal strategies.
• Learning
• Performance measurement for the switching among
  the current set of strategies.
• When the performance is not satisfactory, a new
  mode will be identified and corresponding optimal
  strategy will be designed.

Carnegie Mellon University
26
Multi-Mode Markov Decision Process Model

• Markov System Xk, k 0, 1, , state space S.
• System Mode ?k, k 0, 1, , ? 1, 2, , ?.
• Action set U and action subset U(s) ? U for
  each s?S.
• A (stationary) policy L is a mapping from S to U
  such that L(s) ? U(s) for each s?S.
• At epoch k, after an action u ? U(Xk) is applied,
• Transition to s with probability
• Reward incurred
• Mode jumps to ?k1.
• Objective find optimal policy sequence L0, L1,
  to maximize performance

Carnegie Mellon University
27
Switching Based on Certainty Equivalence (CE)

• Let L? be the optimal policy when ?k is a
  constant ???.
• Suppose ?k is a Markov process with transition
  matrix Q.
• CE Switching Strategy Apply L? when ?k?.
• When is the CE strategy optimal?
• I - Q ? ? (1 - ?) B /(2 ?A), A and B
  computable
• How well does CE switching do in general?
• JCE - J? ? 2 ?A I - Q? / (1 - ?)2
• JCE performance under CE switching
• J optimal performance.

Carnegie Mellon University
28
CE Strategy for Unobservable Modes

• CE switching Use the MLE of ?k
• When is CE switching optimal?
• 2(1 - max? p(?))? (1 - ?) B /(2 ?A)
• How well does CE switching do?
• JCE - J? ? 4?A(1- max?p(?)) / (1 - ?)2
• JCE performance under CE switching
• J optimal performance.

Carnegie Mellon University
29
Simple Example

• Stationary policies for each mode
• If G2 always bids 14, 19 and 25, G1 bids 10,
  15 and 20.
• Case 1
• G2 bids randomly with prob. 0.2, 0.2, 0.8
• G1s optimal bidding strategy Always bid 20 -
  CE strategy!
• Case 2
• G2 bids randomly with prob. 0.3, 0.3, 0.4
• G1s optimal bidding strategy Always bid 15 -
  not a CE strategy!

return
Carnegie Mellon University
30
Research at TAMU Objectives

• Survivability and Protection of the system for
• Transient voltage, angle, oscillation, long term
  voltage stability crises.
• Overflow problems.
• Protective relaying
• Responsibility evaluations of
• Loop flow Problems.
• Market Efficiency for
• Generation Dispatch Problems.
• All the problems are coupled, objectives are
  sometimes
• conflicting.

Texas AM University
31
CDNA Interpretations

• Detection agents to detect transient angle
  stability, voltage stability, oscillation, long
  term voltage crises using acceleration angle
  velocity, line flows, voltage profiles, etc.
  (Security Margin Monitoring)
• Stabilizing agents, congestion control agents,
  auction agents, protection agents, performance
  control agents (to compromise different
  objectives.)
• Need to know contexts to switch among agents for
  survivability, protection and market efficiency.
• CDNA activates the needed control for best
  performance.

Texas AM University
32
Accomplishments

• Transient angle stabilizing controls using TCSC,
  SMES, SVC, Braking resistors.
• Stabilizing controls for transient voltage
  stability using TCSC.
• Generation dispatch using auction agents.
• Flow Decompositions of bilateral trades for
  responsibility evaluations.
• Demonstrate interactions between market policies
  and reactive power controls Stable financial
  systems imposed on a stable engineering system
  may cause overall instability. Bad incentives and
  misconceptions.

SKIP
Texas AM University
33
Work to be done in 2000

• Detection Agents for Transient voltage, angle,
  long term voltage stability crises.
• Security Margin Monitoring for long term and
  short voltage problems.
• Responsibility evaluations of loop flows using
  Flow Decompositions.
• Congestion controls using FACTs.
• Demonstrate the use of Protection Relays as
  Structural Controls to avoid cascading failures..

Texas AM University
34
Some Highlights

• Key misconception on reactive power controls are
  identified.
• We demonstrate using a simple BPA system why
  these concepts are wrong.
• Six questions are clarified using simulations.
• Pricing based on these wrong concepts may lead to
  system instability.
• Financial incentives should be based on solid
  engineering foundations.

Texas AM University
35
Q1 Is Voltage Control Effect of Generators Local
by nature? What are the impacts on reducing the
Var reserve?

• No. It is system-wide. And reduced Var reserve
  will have system-wide impact.
• Reduced reserve will also cause voltage transient
  stability, which collapsed in seconds.

Q2 How ULTC affects Voltage Stability?

• In many cases they harm the security.

Texas AM University
36
Q4 What is the impact of Real Loads on Voltage
Stability?
Q3 Can Intensive Use of Smart Shunt Banks at
Load Areas Replace Dynamic VAR Reserves of
Generators?

• No, definitely not.

• They have substantial impacts.

Texas AM University
37
Q5 How do Load Characteristics Impact on
Security Margin?

• They have substantial impact.

Q6 Will a stable financial system imposed on a
stable engineering system destabilize the whole
system?

• Yes, definitely. Wrong incentives and
  misconception of reactive power system can
  destabilize the whole system.
• Interactions between financial system and
  engineering systems need to be investigated.

Texas AM University
38
Conclusions

• Not all VARs are created equal.
• Misconceptions on voltage stability are
  demonstrated.
• New findings will enable us to accurately
  evaluate reactive power provisions from
  generators and other devices in a deregulated
  power market, such as power pool market,
  bilateral trade market or compatible market.

Texas AM University
39
Protective Relaying

• Identified needs
• Reduce dependency on setting inaccuracy
• Improve selectivity between permanent and
  temporary faults
• Improve security/dependability
• Introduce Coordination between Control and
  Protection

Texas AM University
40
Protective Relaying

• Defined New Protective Relaying Agents
• 1 Neural Net (NN) Algorithm for Fault detection
  and classification
• 2 Synchronized Sampling (SS) algorithm for
  fault location
• 3 Coordination Between NN and SS
• 4 Coordination Between NN, SS and Control

Texas AM University
41
Protective Relaying

• Developed Context Dependent Approach
• Learning (training) for NN Agents
• Line Model (on-line parameter estimation) for
  Synchronized Sampling Agents

Texas AM University
42
Protective Relaying

• Introduced New Performance Benefits
• Better relaying (dependability/security)
• Better reclosing (recognition of permanent vs
  temporary faults)
• Better control (preventing cascading outages)

Texas AM University
43
Protective Relaying

• Introduced New Evaluation Approach
• Definition of future use of modeling and
  simulation tools (local and system events)
• Use of Matlab customized software for evaluation
  of individual protective relaying agents
• Use of Eurostag software for evaluation of
  system-wide interaction among agents

return
Texas AM University
44
Real-Time System Emulation

• Based on inputs from UIUC, developed a
  preliminary power system simulator with external
  control from a remote computer
• Simulator is MATLAB based communication protocol
  is SOCKET
• Demonstrated with the ABC system external
  control switch between several control options

Rensselaer Polytechnic Institute
45
Power System Dynamic Monitoring

• Worked with ISO-NE, NYISO, and NYPA to obtain
  monitored power system disturbance transient data
  from about a dozen Dynamic Recording Devices
  (from several vendors)
• Developed a rule-based Event Identifier for
  classifying system disturbances next step will
  be the development of an advanced identifier
  using detection filters
• In the process of using data obtained from
  several different monitors for the same event to
  analyze interarea oscillations

Rensselaer Polytechnic Institute
46
Control Design

• Proposed alternative controller structure using
  remote measurements as feedback signals
• Controller structure to handle communication
  delay
• Further development of linear matrix inequality
  techniques to control systems with parametric
  dependence

Rensselaer Polytechnic Institute
47
Scenarios

• Introduce two contingency scenarios for the ABC
  system, in addition to the normal operating
  condition, and design controls for the
  contingencies will continue to develop
  additional scenarios for the system

return
Rensselaer Polytechnic Institute
48
Real Time Infrastructure

• How to support the deployment of control agents
  in real time reliably without shutting down the
  normal operations is an important concern.
• Telelab integrates WWW service with a fault
  tolerant dynamic real time architecture, the
  Simplex architecture. Telelab architecture gives
  you the ability
• to add or replace application software components
  on the fly without shutting down its operation.
• to protect the system operation and the integrity
  of equipment from bugs that could be introduced
  by changes.

University of Illinois at Urbana Champaign
49
Telelab Remote Lab Interface
CORBA A/V Streams
LynxOS
Simplex
annotated, pre-recorded
presentation (e.g. HTML)
(in case of communication
failures)
CORBA
A/V Streams
Win98/NT
Win98/NT
important
important
important
important
important
important
important
important
important
important
important
important
important
important
important
important
Demo available at www-drii.cs.uiuc.edu
SKIP
University of Illinois at Urbana Champaign
50
Next Step A Sample Power Network

University of Illinois at Urbana Champaign
51
Sympathetic Relay Tripping A Model Problem

• Background Short circuit and temporary overload
  are very different. But they are treated as if
  they are the same problem due to the lack of
  coordination. Local response could lead to
  cascading failures that bring down a large
  portion of network.
• Coordination Context
• Triggering event a relay open followed by
  neighboring relay open
• Event network SS sample locate the fault,
  inform overload relays to hold and related nodes
  to do power rerouting/load shedding
• Data network continue monitoring and report the
  overload situation on the relays in the holding
  mode
• Control network agents change from normal
  control to overload management to bring the
  relays from holding mode to normal mode

University of Illinois at Urbana Champaign
52
Sympathetic Relay Tripping Model Problem -
contd

• For each chain of events there should be a
  coordinated response.
• Value of information for each fault event there
  can be two solutions. The CDNA solution using
  information or throwing resources at it. This
  allows us to compute the resource equivalence of
  CDNA.
• Research on cascaded failures (network system
  instability) do we have parallels in Internet or
  other forms of network reactions, where a
  coordinated response could have prevented
  cascading failures. Will any ideas in Internet
  congestion control useful for power networks?

University of Illinois at Urbana Champaign
53
A Sample Power Network Failure

overload

• G1 or G2 could become unstable unless
  controllers are switched
• The open of the overload lines could propagate
  the failure to the entire region
• We need to stabilize G1 and G2 controllers and
  re-adjust G3,4,5, 6 and normalize the overloaded
  lines quickly

University of Illinois at Urbana Champaign
54
CDNA Simulation

• Contingency management and agent based control
  testing cases to implement agent and context
  management.
• Getting RPIs sample system implemented.
• Getting CMUs agent sample system implemented.

return
University of Illinois at Urbana Champaign