Video Analysis beyond Structure from Motion

1
Video Analysis beyond Structure from Motion

• Many applications dont include a moving camera

We now move from a lot of geometry to more
statistics. Fear not, however, we still get to
use linear algebra.
2
What is surveillance

• Background subtraction
• Object/Anomaly Detection
• Object tracking
• Anything else the customer wants.

3
Background subtraction requires

• A representation of the background
• A method for learning the background
• Method of detecting image locations that dont
  fit the background.

4
Background subtraction choices

• Representation background image.
• Learning
• Use last image (change/motion detection).
• average of training data (perhaps incremental
  averaging).
• Detection Use a threshold

5

• Examples of background subtraction.

6

• When is a background image insufficient?
• What else could we do (without getting all
  crazy)?

Can use a model that is like an image, but
keeps the mean and variance at each pixel.
7
Statistical Background Image Modeling 1
PDF of the single pixels intensity over time
Intensity of the pixel

• Sometimes, the PDF of many pixel intensities
  could be reasonably approximated by a simple
  model (like a Gaussian, represented by a mean and
  a variance).

What is a PDF?
8
Statistical Background Image Modeling 2

• Some pixels have more complicated appearances.
  Their PDFs could not be well modeled by the
  simpler methods such as a Single Gaussian.
• Subtracting the mean of this distribution would
  give you a number, but that number is not
  meaningful in terms of fit to background model.

Probability
Intensity of the pixel
PDF of the single pixels intensity over time
9
BIG IDEA number 1.

• If one gaussian doesnt work, perhaps 2 (or 3)
  do.
• Gaussian mixture model. Can approximate any
  PDF with enough Gaussian components. In EE you
  can worry about convergence here we worry about
  computational strategies for updating these
  models

10
(No Transcript)
11
(No Transcript)
12
Representing Probability Distributions

• Easy (in the context of streaming video)
• Solve for best fitting (arbitrarily oriented,
  multi-dimensional) Gaussian.
• Solve for best fitting plane in derivative space.
• Harder to represent distribution as mixture
  model.
• Solving for multiple models with EM (an iterative
  process which requires maintaining all the data).
• Leads to adaptations of EM to streaming data
• Adaptive Mixtures.
• Cascading Models.

13
(No Transcript)
14
Adaptive mixture models

• Have several Gaussians (m,S)
• Given a new measurement m
• How likely is this measurement?
• Scorei(m-mi)Si-1(m-mi)
• If for all i, scorei is large then unlikely to
  fit any model (mark as independent).
• Otherwise, update just the model that has
  smallest score.

15
Covariance implies Ellipsoids of Constant
Probability for Gaussian Distributions
1-D Gaussian
2-D Gaussian
from Duda et al.
16
Fitting Gaussians to Color Distributions
Can parametrize scaling, rotation, translation of
ellipsoid with SVD of covariance matrix
17
Mahalanobis Distance

• Distance of point from Gaussian
  distribution
• Along axes of fitted ellipsoid
• In units of standard deviations (i.e.,
  scaled)

X
adapted from Duda Hart
18
Why multi-modal models are important If the
backgrounds are moving, are there properties that
make their distribution easier to represent?
Mixture model representation of filter response
distribution.
It
Filter responses at each pixel (collected over
many frames)
Image Sequence
Ix
19
Spatio-temporal Background Model

• Classical Motion Detection
• Model background image intensity at each pixel.
  Identify pixels different from background as
  independently moving object.
• Motion Detection with Moving Background
• Instead of intensity, use 3D measurement of Ix,
  Iy, It. Estimate probability distribution of
  these measurements at each pixel.
• Identify measurements that are unlikely to come
  from background as independent motion

Iy
It
Ix
20
Build Background Model
In areas where the background motion is
consistent, there is a more specific background
model. This will allow more independent motions
to be correctly marked.
Kingshighway and Lindell intersection
Specificity of Background Model
trace of the covariance matrix
21
Video Surveillance
22
VAnDaL Video Anomaly Detection and Localization
During normal traffic patterns, build up
background models.
Kingshighway and Lindell intersection, St. Louis,
MO (Chase Park Plaza Hotel)
23
Intuitive Background Model
Build Model of Typical Background Motion. Mark
areas of video that dont fit that model
Cool math fact. The THIRD most important
eigenvector of the (Ix, Iy, It) covariance matrix
IS the best fitting optic flow vector
24
Unusual Traffic Motion
Independent Motion Detection with temporal
integration
25
Pedestrian Traffic Detection
Raw Independent Motion Detection Scores
26
Same Anomaly Detection in Other Contexts
(Detection)
27
Same Anomaly Detection in Other Contexts
(Training)
28
Representation of background models

• Measurement is defined for each pixel in each
  frame
• Multi-dimensional Gaussian defined by
  covariance matrix and mean of measurements
  known to be background, defines error function
• Optic flow based measure motion with a
  particular velocity (u,v) defines a constraint
  between the spatial and temporal derivatives.
  Deviation from this constraint is the error
  measure

29
Measuring Performance
Receiver Operator Characteristic (ROC) Plots,
depict how well a particular score function can
be used to classify measurements as foreground or
background.