Situational Awareness in Emergency Response

1
Situational Awareness in Emergency Response

• Dr. Sharad MehrotraProfessor of Computer
  ScienceDirector, RESCUE Project
• http//www.itr-rescue.org

2
Crisis Response
SYSTEM LEVELS
LAW

• A massive, multi-organization operation
• Many layers of government
• Federal FEMA, FBI, CDC, national guard, ..
• State Governors Office of Emergency Services
  (OES), highway patrol,
• County county EOC, police, fire personnel,
• City city emergency offices, police,
  firefighters,
• Volunteer Organizations
• Red cross, organized citizen teams
• Industry
• Gas, electric utilities, telecommunication
  companies, hospitals, transportation companies,
  media companies .

POLICY
FEDERAL
AUTHORITY
STATE
RESOURCE COORD
LOCAL
OPERATIONS
EMC C2
Incident command C2
FIRST RESPONDERS
VICTIMS
3
Los Angeles County Emergency Management
Organization
LA County Emergency Management Council
Board of Supervisors Chair of the
Board Operational Area Coordinator
Director of Emergency Operations Sheriff

Other Entities
Disaster Management Area Coordinators
Sheriff Contact Stations
Emergency Mgmt Information System
Cities of Los Angeles County (87)
3
4
Operational View of Response

• Crisis Management
• Field level operation
• Command and control
• Usually local government in-charge
• Consequence Management
• Gather information
• Field, Cities, Special districts, County
  departments, Other EOC sections/branches
• Analyze consequences with focus on the future
• Develop plan of action
• Life safety, Property loss, Environment,
  Reconstruction
• Establish who is responsible

5
Operations- Consequence Analysis
Public Safety

• Potential need for
• Security for damaged/evacuated structures
• Route management
• Civil disturbance control
• Casualty/Fatality collection points
• Fire fighting/HAZMAT support
• .

• Shelter requirements
• Impact on poor
• Language, other cultural needs
• Food/water distribution
• Impact on schools
• Impact on non-profit agencies

Care/Shelter
6
Operations Consequence Analysis
Construction

• Need for building inspections
• Removal of hazardous materials
• Demolition/debris removal
• Transportation network impact and restoration
• Water/sewage/flood control system impacts

CONSTRUCTION ENGINEERING

• Impact of utility outages
• Priorities for restoration
• Impact on purchasing system
• Impact on transportation
• Priorities for transportation restoration
• Other support

Logistics
7
Role of Information in Response

• Hypothesis Right Information to the Right Person
  at the Right Time can result in dramatically
  better response

• Response
• Effectiveness
• lives property saved
• damage prevented
• cascades avoided

• Quality of
• Decisions
• first responders
• consequence planners
• public

Quality Timeliness of Information

• Situational
• Awareness
• incidences
• resources
• victims
• needs

8
Challenges in Situational Awareness
Incompleteness uncertainty in
Data Multimodality and Diversity of Data Real
time requirements
Inter-organizational relationships Lack of
incentives Privacy confidentiality concerns,
fear of misuse Dynamically evolving needs

• Diversity of delivery mechanisms
• Variability in warning times urgency
• Scale size of impacted population
• Recipient state characteristics

State of infrastructure Surge
demands Diversity of data sources Concerns of
privacy confidentiality
9
RESCUE Project

• The mission of RESCUE is to enhance the ability
  of emergency response organizations to rapidly
  adapt and reconfigure crisis response by
  empowering first responders with access to
  accurate actionable evolving situational
  awareness

Funded by NSF through its large ITR program
10
RESCUE Partners
11
RESCUE Research
12
Situational Awareness Research in RESCUE
Situational Data Management
Decn. Support Tools
Extraction, synthesis, Interpretation
13
Approach

• Multimodal multi-sensor signal processing
• Robustness to noise noise affecting one
  modality may be independent of the others.
• E.g., multimicrophe speech recognition with
  background noise
• Complementary information in different modalities
  certain events easier to detect in some
  modalities than others. By combining modalities
  we can build systems that detect complex events
• E.g., Tracking people is easier in video whereas
  speaker identification is easier in audio.
• Exploit semantics context for signal
  interpretation
• Knowledge of domain can help interpret data, fill
  missing values, disambiguate.

14
Exploiting Semantics for Situational Awareness

• How does the system obtain represent semantics?
• User specified
• Language for specification of semantics,
  expressibility, completeness
• learnt from data
• expressibility, training set might not be
  available for supervised learning, noise in data
  may skew unsupervised learning
• Principled approach to exploiting semantics to
  interpret data
• Probabilistic models?
• Efficiency
• Most such problems are NP-hard
• Generalizability of the approach
• Can we design a generalized approach that can be
  used to work across diverse types of data and for
  diverse situational awareness tasks.

15
Event Detection from sensors

• 2300 Loop sensors in LA and OC
• Goal Detect events such as baseball game from
  loop sensor count data.
• Semantics
• Historical traffic data both during game night
  and non-game night
• Data is, however, unlabelled.
• Smyth et. al. -- TRBC 06, SIGKDD 06, ACM TKDD,
  AAAI 07, UAI 07

16
Detecting Unusual Events
Unsupervised learning faces a chicken and egg
dilemma (and others)
17
Inference over Time
Time t
Time t1
Note how many hidden variables are in this model
18
Detecting Real Events Baseball Games
Remember the model training is completely
unsupervised, no ground truth is given to the
model
19
Entity Resolution Problem
TODS 2005, IQIS 05, SDM 05, JCDL 07, ICDE 07,
DASFAA 07, TKDE 07
20
Two Most Common Entity-Resolution Challenges

• Fuzzy lookup
• reference disambiguation
• match references to objects
• list of all objects is given

• Fuzzy grouping
• group together object repre-sentations, that
  correspond to the same object

21
Example of the problem CiteSeer top-K

• Suspicious entries
• Lets go to DBLP website
• which stores bibliographic entries of many CS
  authors
• Lets check two people
• A. Gupta
• L. Zhang

CiteSeer the top-k most cited authors
DBLP
DBLP
22
Example of the problem Disambiguating locations
23
Web Disambiguation
Music Composer
Football Player
UCSD Professor
Comedian
Botany Professor _at_ Idaho
24
Context Attraction Principle (CAP)
publication P1
J. Smith

• if
• reference r, made in the context of entity x,
  refers to an entity yj
• but, the description, provided by r, matches
  multiple entities y1,,yj,,yN,
• then
• x and yj are likely to be more strongly connected
  to each other via chains of relationships
• than x and yk (k 1, 2, , N k ? j).

John E. Smith SSN 123
P1
John E. Smith
Jane Smith
Joe A. Smith
Can be translated into a graph connectivity
analysis which can be interpreted using
a probabilisitic interpretation.
25
Experiments Quality (web disambiguation)
By Artiles, et al. in SIGIR05
By Bekkerman McCallum in WWW05
26
GDF vs. Traditional (Robustness)
27
GDF vs. Context (Bhattarya Getoor)
28
Semantics in IE

• Extracting relations from free / semi-structured
  text (slot-filling)
• Exploiting semantics in IE
• declaratively specified
• Specified as (SQL) integrity constraints
• On the relation (s) to be extracted
• Learnt from data
• Mine patterns and associations from the data

29
Declarative Constraints
create table researcher-bios ( name
person title thing employer
organization employer-joined date doctoral-degr
ee degree doctoral-degree-alma
organization doctoral-degree-date
date masters-degree degree masters-degree-alma
organization masters-degree-date
date bachelors-degree degree bachelors-degree-a
lma organization bachelors-degree-date
date previous-employers organization awards
thing CHECK employer ! doctoral-degree-alma CHEC
K doctoral-degree-date masters-degree-date )
30
Pattern mining over data

• Represent data as graph (RDF)
• Mine interesting patterns
• Including graph associations
• Example above
• Mostly people who have a PhD degree from a school
  outside the US also have their bachelors degree
  from a school out side the US.

31
Constraints in Action
TUPLE (POSSIBLE) INSTANCES
John Smith, PhD, UCI, 2000, MS, MIT, 1997, BS,
UCI, 1995 John Smith, PhD, MIT, 1997, MS, MIT,
2000, BS, UCI, 1995 John Smith, PhD, MIT, 2000,
MS, MIT, 1997, BS, UCI, 1995

• CONSTRAINTS
• Order of degree dates
• No toggling of schools

John Smith, PhD, UCI, 2000, MS, MIT, 1997, BS,
UCI, 1995 John Smith, PhD, MIT, 1997, MS, MIT,
2000, BS, UCI, 1995 John Smith, PhD, MIT, 2000,
MS, MIT, 1997, BS, UCI, 1995
32
Experimental Results Improvement
CONSTRAINTS ATTRIBUTE LEVEL CD1. All (CS) PhDs
awarded after 1950 CD2. Current position is from
among a fixed list CD3. PhD awarded only by a PhD
awarding school TUPLE CT1. People do not
toggle between schools CT2. Dates of doctoral,
masters, and bachelors degrees are in orderCT3.
People do not work at the same place they
graduate from CT4. More likely that the grad
school is US and the undergrad school is outside
US (vs other way around)CT5. The grad school
rank is at least as good (or better) than the
undergrad school rank

• researcher-bios domain
• (upto) 300 training documents (Web bios)
• Test set 2000 documents
• Use RAPIER Schema (type) information as
  baseline
• Add several constraints
• Improvement in both precision and recall

33
Challenges

• Language for specifying constraints.
• Principled approach to exploiting constraints/
  patterns for extraction.
• Scalability/efficiency
• Naïve approach of enumerating all possible worlds
  leads to exponential complexity.
• Problem NP hard even with a single FD (e.g., Year
  ? BestMovie)

34
Summary

• Situational Awareness research in RESCUE
• Event detection, extraction, and interpretation
  from multimodal sensor data
• Situational data management (R. Jain, S.
  Mehrotra)
• Tools for decision support (S. Mehrotra)
• Two approaches
• Exploiting multimodal and multisensor input
• Multimodal speech, multi-microphone recog. ? B.
  Rao,
• Speech enhanced video ? M Trivedi
• Bayesian framework for Multi-sensor event
  detection ? P Smyth,
• Exploiting semantics for interpretation
• Text, entity disambiguation ? S Mehrotra
• Sensor data ? P Smyth
• Dynamic recalibration of video based event
  detection system exploiting semantics MMCN 08 ?
  S. Mehrotra, N. Venkatasubramanian
• Automated tagging of images using speech input
  exploiting context and semantics Tech. Report
  08 ? S, Mehrotra

35
Summary

• Situational awareness applications requires
  techniques to translate raw multimodal signals
  into higher level events.
• Extensive research on signal processing but much
  of it studies different modalities in isolation
• Multimodal event detection and exploiting
  semantics to interpret data is a promising
  direction.
• A principled, generalizable, and a comprehensive
  approach represents a major challenge and an
  opportunity.
• Situational awareness tools built on such tools
  could bring transformative changes to the ability
  of first responders and response organizations to
  respond to crisis.

36
Connection to Cyber SA
Most of this talk focussed on here. Techniques
could translate for cyber awareness. Also,
through monitoring physical systems they directly
could impact cyber SA.
interdependencies
Physical systems
Cyber Systems
Adaptation, Security intercepts
Adaptation, refinement
Situational Awareness Of physical Systems
Situational Awareness Of underlying cyber
systems
Awareness of state of physical system helps gain
cyber situational awareness and vice versa. I.e.,
State of physical systems can serve as sensors
for cyber systems and vice versa