Uncertainty in Automation: Anomaly Detection in Event-Based Systems

1
Uncertainty in AutomationAnomaly Detection in
Event-Based Systems

• Dawn Tilbury
• Linday Allen (PhD) and John Broderick
• University of Michigan

2
Outline

• Example problem inconsistent logical behavior
• Developed solution Anomaly detection
• Model generation using observed data
• Performance assessment of models using known
  good and bad behavior
• On-line anomaly detection
• Industrial Application
• Academic assumptions meet industry realities
• Resolution and results
• Lessons learned

3
Example from our testbed
Correct, typical behavior
Incorrect behavior
PartReady
Part1
Release Pallet
PartReady
Part1
LoadPart1

• No model of entire systems correct behavior
• Manual inspection required to find this anomaly
• Laborious, offline

4
Approach Anomaly detection using model
generation

• Goal diagnosis of system level event-based
  faults in mfg systems without use of pre-existing
  formal model

• Method
• Generate models based on training data
• Detect anomalies on-line by comparing traces to
  models
• Advise the operator when anomaly occurs

5
Knowns Unknowns

• Known
• Resources in system
• Robots
• CNC machines
• Pallets
• Measurable
• OPC tag changes
• Communication events between controllers

• Unknown
• Formal model of the systemCould be constructed
  but is time-consuming and error prone
• Logic control codeWritten by different people at
  different times in different languages
• Correct event orderMany different orders may be
  acceptable

6
Solution approach
Given resource information and strings of ok
events
Given a new string, determine whether the models
accept it (weight by model performance)
Given some ok and not ok strings, compute the
performance of each model
Create a set of models that can generate these
strings
If not, where is the anomaly
7
Anomaly Detection Method

• Inputs
• Streams of events from system in operation
• Resource information, including mapping of events
  to resources
• Outputs
• Set of models that represent the system behavior
• Model performance on training data
• On-line detection
• Score for each string anomalous or not

8
Machining Cell Physical Set-Up
Entry
Hand-off
Exit
Reject

• Problem G2 will have raw parts and at least one
  CNC available, but G2 incorrectly waits
• Resources
• Gantries, CNCs, buffer at hand-off
• Events PLC data recorded via Ford data
  collection system

9
Data collection set-up

• Data from each machine gantry
• Bits include Cycle End, Good/Bad Cycle, Wait
  Aux, Blocked, and Starved
• PLC message generated each time particular bit
  changes occur
• Approx. 11,000 parts worth of data

IT System
FunctionBlock
FunctionBlock
DrivingLogic
Driving Logic
PLC
PLC
(270,000 PLC messages)
10
Identified Inconsistencies

• What we thought we would get
• Well-defined strings of events
• Events that acquire/release resources recorded
• Unique mapping of PLC bits to events
• Many strings, starting from the initial state,
  labeled as good or bad

• What we got
• Not every event triggers a message ? multi-bit
  change (order is uncertain)
• Not all resource events captured in data
  collection
• Some bits used for multiple purposes
• One huge log file with no defined beginning

11
Resolution of Inconsistencies
Academic Assumptions Industry Realities Resolution
1 Resource events available Some events filtered in data collection I Logic changed
2 String of ordered events Multiple bit changes per message possible A Heuristic decision algorithm
3 Consistent bit-meaning mapping Inconsistent bit-meaning mapping I, A Logic changed, pre-process data
4 Event streams start in initial state System runs continuously A Nec. condition to create stream
5 Separate, labeled streams Continuous, unlabeled stream A Splitting, labeling algorithm
12
Ford data
Word 18 bits 8-10 give the CNC ID
Gantry waiting word 19 bit 9 is high
13
Lessons learned

• Sometimes you can adapt/improve your method to
  handle given uncertainties
• Multiple models when system model unknown
• Multiple bit changes ? uncertain event order
• Initial state unknown
• Advise operator instead of closing the loop
• Sometimes you have to decrease the uncertainty by
  improving the system
• Consistent bit/event mapping
• Unobservable events for acquiring resources

14
Future work