Multi-Scale Video Cropping

1
Multi-Scale Video Cropping

• Hazem El-Alfy, David Jacobs and Larry Davis
• Department of Computer Science
• University of Maryland, College Park
• Sep 25th 2007, ACM MM 07

2
Modern Surveillance Systems

• Networks of sur-veillance cameras.
• Control Room
• Fewer monitors than cameras.
• Far fewer operators than monitors.
• Cameras cycle through monitors.

3
Modern Surveillance Systems
Typical Control Rooms airports, subways,
metropolitan areas, seaports, crowd control.
4
Future Control Rooms

• Continuous display wall versus a fixed set of
  discrete monitors.
• Algorithms to control
• where to display videos,
• how much area to assign to them,
• how to display them.

Barco Control Room, Vienna, Austria
5
Video Cropping
Munich Airport Courtesy Siemens, NJ
6
Why Cropping?

• Resize video to save bandwidth or to fit display
  area.
• Cropping before resizing to focus operator
  attention of on important areas.

7
Problem Definition

• Determine trajec-tories of cropping windows
  through the video
• variable size window
• maximize captured saliency
• smooth trajectory
• occasional jumps (cuts) between trajectories.

8
Problem Definition

• Each frame t covered by variable size overlapping
  windows Wi,t
• Saliency measure S(Wi,t)
• argmaxQ St S(Wi,t), over all window sequences Q
• Subject to constraints for smooth window motion
  and size change.

Wi,t
9
Our Approach Overview

• Extract motion energy.
• Model video as a graph.
• Find trajectories as shortest paths in graph.
• Merge trajectories.
• Repeat for other segments of long videos.

10
Extracting Motion Energy

• Motion energy as a saliency measure.
• Frame differences are smoothed using
  morphological operations.

11
Modeling Graph

• Nodes cropping windows in each frame.
• Add dummy source and target nodes.
• Edges allowable window changes (location and
  size) between consecutive frames.

w0
dummy source node
dummy target node
w0
windows of first frame
windows of i th frame
windows of last frame
12
Modeling Graph

• Multi-scale energy function for window W
• E(W) S(W) always favors large windows
• E(W) S(W)/A(W) favors small (dense) windows
• E(W) S(Win)/A(Win) Sbelt/K
• Edge weight wij 1 ENorm(Wj)

13
Modeling Graph

• Energy function computed for all windows in all
  frames.
• Efficiently computed using integral images Viola
  Jones 01
• ii(x,y) Sxltx,ylty i(x,y)
• E(W)ii(x3)-ii(x2)-ii(x4)ii(x1)

14
Shortest Path

• Dials implementation of Dijkstras algorithm
  linear in graph nodes.
• Smoothing low-pass filter cubic Hermite
  interpolation.

15
Merging Trajectories

• More cropping windows needed to capture
  simultaneous activity.
• Wipe captured activity from motion frames and
  repeat earlier process on remaining motion.
• Merge trajectories find shortest path through a
  graph of trajectories.

16
Processing Long Videos

• Problems
• Graph gets too big if video is long.
• Latencies must be short in surveillance systems.
• Solution
• Break long videos into segments with overlap.
• Process each segment then stitch results together.

break here
break here
17
Processing Long Videos

• Issues
• How short can segments be?
• Are there preferable locations to break video?
• Overlap amount needed for smooth transitions?
• We ran many experiments for fixed size crop
• Shortest path converge quickly. Segments can be
  as short as 40 frames.
• Avoid periods of low activity when breaking
  video.
• Overlap intervals of 20 frames are sufficient.

18
Results

• Munich Airport variable size single window.

19
Results
Munich Airport video-in-video display.
20
Results

• Traffic at a stop sign on campus (2 windows).

21
Contributions

• Variable size smooth cropping window.
• Simultaneous multiple cropping windows.
• Relatively short video segments processed vs. the
  entire video (online).
• Empirically shown identical to processing the
  largest video that can be processed as a whole.