Motion Estimation using Markov Random Fields

1
Motion Estimation using Markov Random Fields

• Hrvoje Bogunovic
• Image Processing Group
• Faculty of Electrical Engineering and Computing
• University of Zagreb
• Summer School on Image Processing, Graz 2004

2
Overview

• Introduction
• Optical flow
• Markov Random Fields
• OFMRF combined
• Energy minimization techniques
• Results

3
Introduction

• Input
• Sequence of images (Video)
• Problem
• Extract information about motion
• Applications
• Detection, Segmentation, Tracking, Coding

4
Spatio-temporal spectrum
f
f
5
Motion aliasing
Large area flicker
f
1/t
f
1/x
Loss of spatial resolution
6
Large motions - temporal aliasing
f
Temporal aliasing
f
Great loss of spatial resolution
7
Temporal anti-aliasing
f
f

• No more overlaping on the f axis.
• filtering (anit-aliasing) is performed after
  sampling, hence the blurring

8
Motion eye tracking
f
f
9
Motion estimation

• Images are 2-D projections of the 3-D world.
• Problem is represented as a labeling one.
• Assign vector to pixel
• Vector field ? field of movement
• Low level vision
• No interpretation

10
Example Ideal
11
Problems

• Problem is inherently ill-posed
• Solution is not unique
• Aperture problem
• Specific to local methods

12
Optical flow

• Main assumption Intensity of the object does not
  change as it moves
• Often violated
• First solved by Horn Schunk
• Gradient approach
• Other approaches include
• Frequency based
• Using corresponding features

13
Image differencing
14
Gradient approach

• Local by nature. Aperture problem is significant.
  
• Image understanding is not required
• Very low level

15
Horn Schunk

• Intensity stays the same in the direction of
  movement. I(x,y,t)

After derivation
16
Horn Schunk

• Spatial gradients Ix,Iy
• e.g. Sobel operator
• Temporal gradient It
• Image subtraction

17
Regularization

• Tikhonov regularization for ill-posed problems
• Add the smoothness term
• Energy function

18
Result
19
Problems of the H-S method

• Assumption There are no discontinuities in the
  image
• Optical flow is over-smoothed.
• Gradient method. Only the edges which are
  perpendicular to motion vector contribute
• Image regions which are uniform do not
  contribute.
• Difficulty with large motions (spatial filtering)

20
Optical flow enhancement

• Optical flow can be piecewise smooth
• Discontinuities can be incorporated
• Solution use the spatial context
• Problem is posed as a solution of the Bayes
  classifier. Solution in optimization sense.
  Search for optimum

21
Bayes classifier

• Main equation

Solution using MAP estimation
22
Markov Random Fields

• Suitable Problems posed as a visual labeling
  problemn with contextual constraints
• Useful to encode a priori knowledge
• required for bayes classifier (smoothness prior)
• equvalence to Gibbs random fields (gibbs
  distribution, exponential like)
• Neighbourhoods
• Cliques
• pairs,triples of neighbourhood points)
• build the energy function

23
MRF

• Define sites rectangular lattice
• Define labels
• define neighbourhood 4,8 point
• Field is MRF
• P(f)gt0
• P(fifS-i)P(fiNi)

24
Coupled MRF

• Field F is an optical flow field
• Field L is a field of discontinuities
• line process
• Position of the two fields.

25
Context

• neighbourhoods and cliques

26
Motion estimation equations
27
Energy for MAP estimation
Parameters are estimated ad hoc
28
Energy minimization

• Global minimum
• Simulated annealing
• Genetic Algorithms
• Local minimum
• Iterated Conditional Modes (ICM) (steepest
  decent)
• Highest Confidence First (HCF)
• specific site visiting

29
Simulated annealing

• (1) Find the initial temperature of the system
  T.
• Assign initial values of the field to random
• For every pixel
• Assign random value to f(i,j)
• Calculate the difference in energy before and
  after If the change is better (diffgt0) keep
  it. Else keep it with the probability
  exp(diff/T)
• (4) Repeat (3) N1 times
• (5) T f(T) where f decreases monotono
• (6) Repeat (3-5) N2 times

30
Results (Square)
OFMRF
Horn-Schunk OF
31
Taxi
32
Results (Taxi)
33
Line process result (Taxi)
34
Cube
35
Results (cube)
36
Line process result (cube)
37
Q A