Prakash Chockalingam

1
Non-Rigid Multi-Modal Object Tracking Using
Gaussian Mixture Models
Prakash Chockalingam
Committee Members Dr Stan Birchfield (chair) Dr
Robert Schalkoff Dr Brian Dean
Clemson University
2
Tracking Overview
Tracker Tasks
Feature Descriptors
Object Model
Update / Learning Mechanism
Tracking Framework
Object Detection
Color Gradients Texture Shape Motion
Template Contour Active Appearance Probability
Densities
Mean Shift Pixel-wise Classification Optical
Flow Filtering techniques
No Update Adaboost Expectation
Maximization Re-weighting Strategy
Manual Segmentation Feature Points
3
Approach

• Tracking Framework Target and background is
  modeled as a mixture of Gaussians in a joint
  feature-spatial space. A strength map is computed
  indicating the probability of each pixel
  belonging to the foreground.
• Contour Extraction Contour is extracted using a
  discrete implementation of level sets
• Image Segmentation Each Gaussian (fragment) is
  adapted to the image data by segmenting the
  image.
• Update Mechanism The parameters of all the
  Gaussians are updated based on tracked data
• Results

4
Tracking Framework

• Bayesian Formulation

Contour at time t
Previously seen contours
Image data of all frames
Assuming conditional independence among pixels,
Feature vector
5
Object Modeling
f2
Gaussian Mixture Model (GMM)
y
?
Strength Image
f1
gt0 for Foreground lt0 for Background
6
Strength Image
GMM
Linear Classifier
Single Gaussian
7
Strength Image (contd)
Linear Classifier
Single Gaussian

Individual Fragments
Final Strength
Strength Without Spatial Information
8
Topics

• Tracking Framework Target and background is
  modeled as a mixture of Gaussians in a joint
  feature-spatial space. A strength map is computed
  indicating the probability of each pixel
  belonging to the foreground.
• Contour Extraction Contour is extracted using a
  discrete implementation of level sets
• Image Segmentation Each Gaussian (fragment) is
  adapted to the image data by segmenting the
  image.
• Update Mechanism The parameters of all the
  Gaussians are updated based on tracked data
• Results

9
Contour Extraction
(strength image)
(frontier)
gt 0 Inside lt 0 Outside
Energy Functional
Implicit representation of growing region
Likelihood term (Strength image)
Regularization term
10
Contour Extraction (contd)
(Region to be shrunk)
(Region already grown)
(Region to be grown)
(Region that need not be considered)
11
Contour Extraction (contd)
such that
Contraction
x
x
Dilation
x
x
such that
12
Contour Extraction (contd)
Expand
Remove interior points
Contract
Remove exterior points
13
Contour Extraction (contd)
Likelihood
Final Region
14
Topics

• Tracking Framework Target and background is
  modeled as a mixture of Gaussians in a joint
  feature-spatial space. A strength map is computed
  indicating the probability of each pixel
  belonging to the foreground.
• Contour Extraction Contour is extracted using a
  discrete implementation of level sets
• Image Segmentation Each Gaussian (fragment) is
  adapted to the image data by segmenting the
  image.
• Update Mechanism The parameters of all the
  Gaussians are updated based on tracked data
• Results

15
Region Segmentation
Mode-seeking region growing algorithm

• do
• Pick a seed point that is not associated to any
  fragment
• Grow the fragment from the seed point based on
  the similarity of the
• pixel and its neighbors appearance
• Stop growing the fragment if no more similar
  pixels are present in the neighborhood of the
  fragment
• until all pixels are assigned

Seed point
Eigen values of 3x3 RGB covariance matrix
where
16
Region Segmentation (contd)

• Pick the minimum element in S. Create a region
  to hold the pixel and add the neighbors in a
  fixed window.
• Compute Mean µj and Covariance Sj of the region.
• Likelihood
• Grow the region as before with two additional
  steps
• Update µj, and Sj, as a new pixel is added
• Remove the corresponding element in S if a pixel
  is added
• Continue above steps if S is not empty.

Mahalanobis distance
Configurable parameter
Initial region
17
Region Segmentation (contd)
Graph-Based
Mean-Shift
Region Growing
18
Region Segmentation (contd)
Graph-Based
Mean-Shift
Region Growing
19
Topics

• Tracking Framework Target and background is
  modeled as a mixture of Gaussians in a joint
  feature-spatial space. A strength map is computed
  indicating the probability of each pixel
  belonging to the foreground.
• Contour Extraction Contour is extracted using a
  discrete implementation of level sets
• Image Segmentation Each Gaussian (fragment) is
  adapted to the image data by segmenting the
  image.
• Update Mechanism The parameters of all the
  Gaussians are updated based on tracked data
• Results

20
Update Mechanism
f2

• Update parameters of existing fragments
• Detect fragment occlusion
• Find new fragments

f1
Initial Model
Fragment Association
Initial Frame
21
Update Mechanism (contd)
Updating parameters of existing fragments
Weight computed by comparing Mahalanobis distance
Initial Model
(function of past and current values)
22
Update Mechanism (contd)
Occluded fragments If a fragment is associated
with less than 0.2 of the image pixels, then the
fragment is declared as occluded. Finding new
fragments
Helps in handling self-occlusion
23
Spatial Alignment
The spatial parameters are updated using the
motion vectors from Joint Lucas-Kanade approach
Joint Lucas-Kanade
Lucas-Kanade
24
Algorithm summary

• Initial frame
• The user marks the object to be tracked.
• The target object and background scene are
  segmented based on their appearance similarity.
• The target object and background scene are
  modeled using a mixture of Gaussians where each
  Gaussian correspond to a fragment in the joint
  feature-spatial space
• Subsequent frames
• Update the spatial parameters of GMM using the
  motion vectors of Joint Lucas-Kanade
• Each pixel is classified into either foreground
  or background by generating a strength map using
  the Gaussian mixture model (GMM) of the object
  and background.
• The strength map is integrated into a discrete
  level set formulation to obtain accurate contour
  of the object.
• Using the tracked data, the appearance
  parameters of the GMM are updated.

25
Topics

• Tracking Framework Target and background is
  modeled as a mixture of Gaussians in a joint
  feature-spatial space. A strength map is computed
  indicating the probability of each pixel
  belonging to the foreground.
• Contour Extraction Extract contour using a
  discrete implementation of level sets
• Image Segmentation Each Gaussian (fragment) is
  adapted to the image data by segmenting the
  image.
• Update Mechanism The parameters of all the
  Gaussians are updated based on tracked data
• Results

26
Experimental Results
Elmo Sequence
Monkey Sequence
27
Experimental Results (Contd)
Person Sequence
Fish Sequence
28
Experimental Results Self-Occlusion
Without Self-Occlusion Module
With Self-Occlusion Module
29
Conclusion

• A tracking framework based on modeling the
  object as mixture of Gaussians is proposed
• An efficient discrete implementation of level
  sets is employed to extract contour.
• A mode-seeking region growing algorithm is used
  to segment the image.
• A simple re-weighting strategy is proposed to
  update the parameters of Gaussians.
• Future Directions
• Incorporate shape priors.
• Utilize the extracted shapes to learn more
  robust priors.
• An offline or online evaluation mechanism during
  the initialization phase.
• Adding global information into the region
  segmentation process.

30
Questions ?
31
Thank you !