A Complex Event Recognition Architecture

1
A Complex Event Recognition Architecture

• Will Fitzgerald
• Kalamazoo College
• R. James Firby
• I/NET, Inc.

2

• A Complex Event Recognition Architecture
• Protecting us from the Metal Horde!
• Will Fitzgerald
• R. James Firby

3
What is

• A Complex Event?
• Complex events are hierarchical, discrete,
  time-stamped structures inferred from
  multi-channel, asynchronous signals.

4
What is

• A Complex Event Recognition Architecture?
• A description or implementation of typical
  patterns and recognition algorithms for complex
  events.

5
A real example

• Water Recovery System at NASA's Johnson Space
  Center
• Four complex subsystems,
• About 200 sensors and actuators,
• Each subsystem asynchronously signals data.

6
Detecting Safe Mode

• When a problem is detected internally, the Water
  Recovery System attempts to go into safe mode,
  which occurs when the four subsystems are safed.
• Safing of the four subsystems happen
  asynchronously.
• Safing detection for each subsystem differs
  from one another.
• On recognizing that the WRS has gone into safe
  mode, signal an event that all subsystems have
  been safed.

7
Another example

• To get directions to a location on the on-board
  map, the user says
• Go here and
• Taps the display location
• within 200 ms.

(CNN photo)
8
Parsing the world

• Dynamic Predictive Memory Architecture (DPMA)
• KR and Semantic parsing
• Task execution and dialogue management
• complex, dynamic environments
• Do similar techniques apply to
• multi-channel, asynchronous sensors?
• multi-modal interface input?

9
A Complex Event Recognition Architecture

• What assumptions are reasonable to make about the
  form of input data?
• What useful general patterns are there in the
  data?
• What recognition algorithms do we need?

10
NLP Assumptions

• Input to Natural Language Processing systems are
  typically assumed to be
• Discrete events of one type (words)
• Single channel
• Totally ordered by position duration irrelevant

11
More generally

• Events of various types
• Over multiple channels and asynchronous
• Duration of event often important
• Hierarchical model still useful

12
Assumptions about Events

• Discrete Individually distinict, non-continuous
  data (could be discretized).
• Time-stamped Event carries the start and end
  times (defining the event duration, which could
  be instanteneous).
• Typed Events form distinct types (e.g., words
  vs. taps).
• Structured Event may internal, hierarchical
  structure (complex).

13
Standard Event Patterns

• Are there patterns of events which are
  particularly useful to identify?
• Are there recognition algorithms to identify
  those patterns?
• Yes.
• ONE and BINDING
• IN-ORDER, ALL, ONE-OF
• Allen patterns
• WITHIN and WITHOUT

14
ONE and BINDING patterns

• ONE The simple pattern of looking for a single
  event (of a particular type).
• BINDING ONE pattern plus collecting and
  constraining state.

• Essentially event-driven programming the
  stimulus in S-R.
• ON-CLICK
• A ONE pattern if just looking for the click
• A BINDING pattern if x,y coordinates are
  significant.

15
IN-ORDER patterns

• Events will occur in order
• That is, saying two events, A and B, occur in
  order, the start time of B is ? the end time of
  A.
• (IN-ORDER A B C D)
• First an event of type A, then B, etc.

16
IN-ORDER as NLP

• Combined with BINDING and signaling of
  subpatterns this is essentially a classic natural
  language processing pattern.

S ? NP VP NP ? DET N VP ? V NP
The boy saw the girl. S NP DET theN boy
VP V saw NP DET the N
girl
17
ALL Patterns

• Events will all occur, but in any order
• With this, we leave (our) standard NLP
  approaches.
• For example, user will choose from all of the
  sets of options.
• For example, all subsystems will be safed, but
  in any order.

18
ALL patterns and contradiction

• The problem user or system undoing an event
  that has already been seen (interpreting events
  as state changes).
• Example Class will start when all the students,
  Alice, Bob, Charles, Dominique, have arrived.

19
Consider this sequence for (ALL A B C D)

• Charles arrives.
• Alice and Bob arrive together.
• Alice starts to sing.
• Charles leaves.
• Dominique arrives.
• Charles arrives.

Order is not relevant Alices singing is not
relevant but Charless leaving undoes his
earlier arrival.
20
ONE-OF Pattern

• Look for any of a set of event forms
• Example Office hours begin as soon as one of the
  professors A,B,C or D arrives.
• (ONE-OF A B C D)

21
Time-based patterns

• Allen relationships
• WITHIN patterns
• WITHOUT patterns

22
Allen Patterns
A contains
B

• contains
• finishes
• starts
• before
• meets
• overlaps
• equal
• overlapped by
• after
• met by
• started by
• finished by
• during

• James Allen described the relationships between
  two intervals.
• Allen patterns look for temporal relationships
  between 2 events or an event and an interval.

A
overlaps B
23
WITHIN and WITHOUT

• WITHIN patterns reflect that the duration of an
  event is no longer than a certain amount of time.
• E.g., an ALL pattern wrapped in a WITHIN pattern.

• WITHOUT patterns reflect that an interval of time
  will pass without the occurrence of an event.
• E.g., Sherlock Holmess significance of the
  barking dog.

24
Pattern Combination

• Go here and a tap within 200 ms.

(within (all (in-order go here) (tap ?x
?y)) 200 ms)
(CNN photo)
25
Safe mode recognizer

• (define-recognizer (safing-complete)
• (pattern
• '(all
• (safing (system pbbwp) (status on))
• (safing (system ro) (status on))
• (safing (system aes) (status on))
• (safing (system pps) (status on))))
• (on-complete (st end)
• (signal-event '(all-safed) st end)))

Some details elided
26
Parsing Algorithms

• The parsing algorithms and recognizer semantics
  are more fully described in the paper.

27
Implementation Details
28
Conclusions

• Standard patterns of events.
• Standard recognizers for these patterns.
• Good for monitoring complex (internal) system
  state.
• Useful for recognizing patterns of complex events
  over multiple modes, over time.

29
Acknowledgments

• Work done under NASA SBIR contract NAS9-00122.
• We would like to especially acknowledge
  collaborators at NASA, including Debra
  Schreckenghost, Pete Bonasso, Carrol Thronesbery
  and others.
• Pulp Images from Pulp of the Daygroups.yahoo.c
  om/group/pulpoftheday

30

• Questions?