ViPER Video Performance Evaluation Resource

1
ViPERVideo Performance Evaluation Resource

• University of Maryland

2
Problem and Motivation

• Unified video performance evaluation resource,
  including
• ViPER-GT a Java toolkit for marking up videos
  with truth data.
• ViPER-PE a command line tool for comparing
  truth data to result data.
• A set of scripts for running several sets of
  results with different options and generating
  graphs.

3
Solutions

• Object level matching.
• First, do matching.
• For each ground truth object, get the output
  object that is the closest.
• Alternatively, for each subset of truth objects,
  get the subset of output objects that minimizes
  the total overall distance.
• Measure of precision / recall for all objects.
• Score for each object match.
• O(ex)
• Pixel/object frame level and single-match
  tracking.
• For each frame, generate a series of metrics
  looking at the truth and result pixels and box
  sets.
• Using keys, or the location of object in frame k,
  get success rates for matching individual moving
  boxes.

4
Pixel Graphs
5
Pixel-Object Graphs
6
Tracking Graphs
7
Progress

• Polygons added.
• Slight improvements in memory usage.
• Various responses to user feedback.
• Changed the way certain metrics are calculated.

8
Goals and Milestones

• Defining formats for tracking people, and metrics
  to operate on them.
• Adding new types of graphs to the script output.
• Replacing or upgrading the current graph toolkit.
• Reducing memory usage.

9
Fin

• Dr. David Doermann
• David Mihalcik
• Ilya Makedon
• many others

10
Object Level Matching

• Most obvious solution many-many matching.
• Allows matching on any data type, at a price.

11
Pixel-Frame-Box Metrics

• Look at each frame and ask a specific question
  about its contents.
• Number of pixels correctly matched.
• Number of boxes that have some overlap.
• Or overlap greater than some threshold.
• How many boxes overlap a given box?
  (Fragmentation)
• Look at all frames and ask a question
• Number of frames correctly detected.
• Proper number of objects counted.

12
Individual Box Tracking Metrics

• Mostly useful for the retrieval problem, this
  solution looks at pairs of ground truth boxes and
  a result box.
• Metrics are
• Position
• Size
• Orientation

13
Questions Ignoring Ground Truth

• Assume the evaluation routine is given a set of
  objects to ignore (or rules for determining what
  type of object to ignore). How does this effect
  the output?
• For pixel measures, just dont count pixels on
  ignored regions.
• For object matches, do the complete match when
  finished, ignore result data that matches ignored
  truth.

14
Questions Presenting the Results