ViPER Video Performance Evaluation Resource

1
ViPERVideo Performance Evaluation Resource

• University of Maryland

2
Problem and Motivation

• Unified video performance evaluation resource,
  including
• ViPER-GT a Java toolkit for marking up videos
  with truth data.
• ViPER-PE a command line tool for comparing
  truth data to result data.
• A set of scripts for running several sets of
  results with different options and generating
  graphs.

3
ViPER-GT
4
Data Format

• A series of items, each with a set of attributes.
• 3 Types
• File
• Content
• Object
• Allow the user to specify their own data.
• Let the performance evaluation tool handle the
  mapping between different descriptor
  configurations.
• XML Format
• Allows for easier parsing and more exact
  specification.

5
ViPER-PE

• Object level matching.
• First, do matching.
• For each ground truth object, get the output
  object that is the closest.
• Alternatively, for each subset of truth objects,
  get the subset of output objects that minimizes
  the total overall distance.
• Measure of precision / recall for all objects.
• Score for each object match.
• Pixel/object frame level and single-match
  tracking.
• For each frame, generate a series of metrics
  looking at the truth and result pixels and box
  sets.
• Using keys, or the location of object in frame k,
  get success rates for matching individual moving
  boxes.

6
Object Level Matching

• Most obvious solution many-many matching.
• Allows matching on any data type, at a price.

7
Pixel-Frame-Box Metrics

• Look at each frame and ask a specific question
  about its contents.
• Number of pixels correctly matched.
• Number of boxes that have some overlap.
• Or overlap greater than some threshold.
• How many boxes overlap a given box?
  (Fragmentation)
• Look at all frames and ask a question
• Number of frames correctly detected.
• Proper number of objects counted.

8
Pixel Metrics

• Examine the pixels that were detected, not the
  objects

9
Pixel-Object Object Count

• How many objects were detected over the whole
  clip?
• The simplest metric, this does not take into
  account any attribute data, just the object type.

10
Pixel-Object Detection

• Look at the number of result boxes that overlap
  some truth object.

11
Pixel-Object Localization

• For each object that is detected, only count
  those that have at least some accuracy in terms
  of the number of shared pixels.

12
Pixel-Object Fragmentation

• A function of the number of result objects that
  each truth object overlaps.

13
Frame Metrics

• How many frames that were correctly marked as
  containing data?
• What is the count accuracy for that frame?

Detected
1
2
2
1
Truth
Missed
1
4
4
False
Result
Ignored
Acc.
0
.6
.6
0
0
14
Individual Box Tracking Metrics

• Mostly useful for the retrieval problem, this
  solution looks at pairs of ground truth boxes and
  a result box.
• Metrics are
• Position
• Size
• Orientation

15
Questions Ignoring Ground Truth

• Assume the evaluation routine is given a set of
  objects to ignore (or rules for determining what
  type of object to ignore). How does this effect
  the output?
• For pixel measures, just dont count pixels on
  ignored regions.
• For object matches, do the complete match when
  finished, ignore result data that matches ignored
  truth.

16
Tools

• RunEvaluation
• Runs multiple evaluations and creates graphs to
  compare them.
• gtf2xml, xml2gtf, gtf2gtf, xml2xml
• Allow conversion between data formats
• Allow merging of multiple data files
• Split and clip functions good for producing
  starter data.
• Installation scripts

17
Ground Truth

• Keyframe collection.
• Text
• Face
• Clip collection.
• Text tracking

18
Progress

• GT
• Polygons
• Interpolation
• Zoom scope
• PE
• Penn State metrics
• Improved data filtering
• Scripts
• Improved scripts for handling batch processing
• Manipulation of sets of ground truth data

19
Goals and Milestones

• Defining formats for tracking people, and metrics
  to operate on them.
• Adding new types of graphs to the script output.
• Replacing or upgrading the current graph toolkit.

20
-

• Dr. David Doermann
• David Mihalcik
• Ilya Makedon
• many others

21
Questions Presenting the Results
22
Questions Describing People