Video Trails: Representing and Visualizing Structure in Video Sequences

1
Video Trails Representing and Visualizing
Structure in Video Sequences

• Vikrant Kobla
• David Doermann
• Christos Faloutsos

2
Outline

• Background and Motivation
• Overview
• Video Trails
• Trail Segmentation
• Trail Classification
• Gradual Transition Detection
• Experiments and Results
• Conclusion

3
Background and Motivation

• Video is a valuable information resource
• There are still few efficient ways to provide
  access to the information the video contains
• Early work on indexing video treated video
  sequence as collections of still images, ignored
  the temporal structure
• Efficient analysis and representation of the
  temporal structure of a video is necessary

4
Overview

• Generate a trail of points (Video Trails) in a
  low-dimensional space
• Segment the video trails
• Classify each of those segmented trails into two
  types
• Stationary (low activity) VS Transitional
  (high activity)
• 4. Detect gradual transition

5
Video Trails

• Definition A trail of points in a
  low-dimensional space where each point
  is derived from physical features of a single
  frame in the video clip
• Features DC coefficients of the luminance and
  chrominance components of an MPEG frame
• Dimensionality Reduction (FastMap)
• initial feature vector
• 
  a vector in that dimensional
• target dimension
  space

FastMap
6
Example

• Consider a video clip with a 320x240 frame size
• Each frame has 20x15 MBs( Macroblock)
• Each MB contains 6 DC coefficients ( 4 luminance
  and 2 chrominance)
• Totally, 20x15x61800 coefficients (initial
  vector)
• 1800-by-1 vector
• 
  (X1,X2,X3)
• 3 (target dimension)
  

FastMap
7
Example
8
Example
9
Trail Segmentation

• Segment the video in order to determine regions
  of high activity corresponding to transitions and
  low activity corresponding to individual shots
• The problem of segmenting the video into sets of
  frames is transformed into the problem of
  splitting the video trails into smaller trails
  corresponding to segments of video

10
Splitting Algorithm

1. Start by placing the first point in a new trail
2. Consider each successive point in the sequence in
   order
3. Perform a test for inclusion of this point in
   the current trail
4. if (the test pass)
5. Include the point in the current trail
6. Move to the next point
7. Goto 2
8. else
9. Close the current trail with the previous
   point as the last one
10. Start a new trail with only the current
    point
11. Goto 2

11
Inclusion Test

• Marginal CostTotal cost per point in the trail
• Consider a clip with N frames
• Assume there are m points in the current trail,
  denoted by set , and be the point being
  considered for inclusion
• Define ,d is the
  dimensionality
• So the new marginal cost is
• new marginal cost gt previous one not include
• new marginal cost lt previous one include

12
Example
13
Example (close-up)
14
The sequence of frames that yield the sparse
transition between the two dense clusters
15
Trail Classification

• Classify each of those segmented trails into
• Stationary (low activity) or Transitional
  (high activity)
• Classification Criteria
• Monotonicity W10.4
  
• Sparsity W20.3
• Convex Hull Volume Ratio W30.2
• MBR Shape W40.1

16
Monotonicity

• If a trail is (close to) monotonic, in some
  direction,its likely transitional
  projection of distance along k
• projected distance ratio
• 
  the length of MBR dimension k
• Minimum projected distance ratio
  

17
Monotonicity (Normalization)

• Recall
• W1 is the weight of monotonicity
• Tlow is the lower bound1.1
• Tup is the upper bound2.0

18
Sparsity

• Sparsity total MBR volume per point
• Average Sparsity
• Sparsity Ratio
• Normalize

19
Convex Hull Volume Ratio

• The ratio of volume of the convex hull of points
  in a trail to the volume of MBR
• Normalize

20
MBR Shape

• Cuboidal
• Planar
• Elongated

21
Classification
22
Gradual Transition Detection

• Dissolves, Fades, Wipes
• Difficulty activity arising from camera or large
  object motion also yields trails similar to
  trails resulting from gradual edits
• Filter out any kind of global motion leading to a
  transitional trail, Analysis global motion

23
Results
24
Conclusion

• Provide a compact representation of a video
  sequence structure
• Reduce a sequence MPEG frames to a trail of
  points in a low dimensional space
• Segment trails and classify each segment as
  either stationary or transitional
• Detect gradual edits