Video Event Recognition Algorithm Assessment Evaluation Workshop VERAAE

1
Advanced Research and Development Activity

• Video Event Recognition Algorithm Assessment
  Evaluation Workshop VERAAE

ETISEO NICE, May 10-11
2005 Dr. Sadiye Guler Sadiye Guler -
Northrop Grumman IT/TASC Mubarak Shah, Niels da
Vitoria Lobo - University of Central Florida
Rama Chellappa, Dave Doermann - University of
Maryland
US Government Champions Terrence Adams-NSA,
John Garofolo, Rachel Bowers-NIST
2
Problem

• Comparative study of Video Event Recognition
  (VER) algorithms to assess applicability,
  usefulness and limitations of different
  approaches
• Motivation
• Several promising VER algorithms exist
• The algorithms have varying degrees of success
  with different types of event detection
• No largely accepted criteria or data set (with
  ground truth) exist for VER evaluation (few
  emerging studies..)
• The performance of VER algorithms is highly
  dependent on the results of object detection and
  tracking, rendering fair comparison of just the
  event recognition very difficult

3
Workshop Goals

• Produce realistic operational video event data
  set representing scenarios for surveillance
  domain
• Ground truth the video event data for VER and map
  to suitable Event Ontology developed in previous
  workshops
• Annotate the data set with object detection and
  tracking metadata that serves the needs of all
  participating/expected event recognition
  algorithms
• Develop evaluation criteria and metrics for
  quantitative evaluation of VER algorithms and
  software tools for evaluation
• Assess different VER approaches for the
  applicability to operational scenarios by their
  learning/explanation/ recognition capabilities

4
VERAAE Approach
5
Video Event Recognition and VERAAE
Video data
Signal
Content Extraction
Provided
Raw Information
Event Detection
Evaluation
Semantics, Ontology
Event Recognition
Abnormal and suspicious events Trends,
correlations..
Knowledge, Intelligence
6
VERAAE Domain

• VERAAE domain focus surveillance
• realistic scenarios of interest

Events and activities existing algorithms can
detect
Realistic high level or complex events
end-users want to detect
7
Data Set Planning

• Primary factors that determine the data
  requirements
• Fixed camera views, no PTZ
• Color, BW and IR
• Realistic operational scenarios
• About 10 events with varying complexity, at least
  10 samples per event
• The collection parameters that address the
  functional capabilities of the algorithms
• Annotation will include the object track data
  required by the participating algorithms
  (automatically and manually generated) e.g.
• Silhouettes of tracked objects
• Bounding boxes and centroid of objects (U
  Maryland ViPER tool)
• Object category e.g. vehicle, person, box,
  animal,
• Ground truth for video events will be generated
  using the event ontology work
• Frame numbers (time offsets) for Event Start and
  End, identified simple sub events

8
Event Ontology (Event Taxonomy workshop)

• Simple event
• Domain independent action descriptors
• e.g. abandoning an object
• Compound (complex or multi-threaded) event
• Multiple simple events taking place in time and
  space constraints to achieve complex activities.
• e.g. planting suspicious object, (if considered
  with below simple events
• moving in the wrong direction
• parked car at the curb-side
• no one exiting parked car
• getting in the car
• Domain specific high level event
• Semantic interpretation of events in a
  particular context, over multiple-views and
  multiple data type events
• e.g. sabotaging public facility

9
Recognizing Surveillance Events

• Surveillance Event types from the users point of
  view
• Violation of some rule
• wrong direction (in thru the out door)
• abandoned object ( suitcase left unattended for
  tgtT)
• Suspicious or Interesting activity
• non exit from a parked car
• repeated visits to a store shelf
• Abnormal activity
• approaching several cars in the lot
• several somewhat suspicious events in close
  proximity

Naturally represented by rules and
constraints Users can easily describe them
Highly context dependent, even context from other
camera views Users can not easily describe but
know when they see it
Naturally represented by probabilistic models and
learning Users build a sense of normalcy
10
Recognizing Surveillance Events

• Knowing what can be detected we describe the
  events using not only observable, but also
  detectable actions
• Example Shoplifting

• Camera 2 in the parking lot
• Car in front of emergency exit
• No one exits from car

• Camera 1 in the store
• Repeated visit to an area
• Running in the store

11
Rule Based Event Violation by an activity
constraint car parked in the driveway
12
Rule Based Event Violation by an object class
constraint
13
Suspicious Event testing the exclusion zone
14
Abnormal Event Vehicle casing the building
15
Abnormal Event Large Vehicle at the Gate
16
Workshop Timeline
May 05
December 05
Data, Evaluation criteria generation,
distribution
Evaluation tools development, Evaluation results,
Final report
Planning, invitations communications
Evaluation Criteria Focus Meeting
First Workshop Meeting June 20/21 With CVPR
Scenario Focus Meeting
Workshop Dry-Run Meeting October (3rd week) In
Boston
Workshop Final Meeting
Final report
This is a seedling workshop to investigate
feasibility

17
Workshop Approach

• First Workshop Meeting (2 days, June)
• Purpose
• Workshop goals and vision
• Presentation and determination of algorithms to
  participate in the workshop
• Presentation of example data sequences.
• Outcome
• Outline of the data requirements (object
  tracking, data exchange protocols etc.)
• Draft a rough set of evaluation criteria
• Solicit feedback on scenario complexity and
  realism

18
Workshop Approach

• Evaluation Criteria Focus Meeting (2 days, July)
• Purpose to determine evaluation criteria best
  suited for VER.
• Outcome - Evaluation Criteria will be
  interactively developed in workshop meetings
  leveraging Event Ontology, VEML and ETISEO
  workshop findings
• Evaluation metrics at the component and system
  level will be defined based on
• Recognition rate
• Learning rate
• Recall and Precision rates
• True/False positives, True/False negatives and
  relevance of false detections
• Event decomposition (based on the ontology
  defined sub event recognition rate)

19
Workshop Approach

• Workshop Dry-Run Meeting (2 days, October 05)
• Purpose and Outcomes
• Participants feedback on processing the sample
  data sets.
• Evaluation tools and methodology presentation
• Evaluating the evaluation criteria and
  finalizing all metrics to be used.
• Planning of evaluation format
• Discussion of interpretation of results

20
Workshop Results

• Raw and annotated (with object detection and
  tracking data) video sequences for realistic
  operational scenarios
• Event Recognition ground truth data based on
  surveillance Event Ontology
• Re-usable and extendible Evaluation Criteria
  suitable for VER
• Software tools for event detection evaluation
• The groundwork for a formal VER evaluation process