Avinash Kumar

1
Avinash Kumar 200507010 International Institute
of Information Technology, Hyderabad 2007
Advisor Dr C. V. Jawahar
2
Thesis Motivation
Computer Vision Algorithms can be applied to real
life problems only if they are FAST and ACCURATE
FAST Optimization framework required.
ACCURATE Better algorithms required.
3
Thesis Objective
Propose New algorithms in Optimization framework
for the following Three computer vision problems
4
Thesis Objective
Propose New algorithms in Optimization framework
for the following Three computer vision problems
5
Thesis Objective
Propose New algorithms in Optimization framework
for the following Three computer vision problems
6
Thesis Outline and Contributions
Optimization Framework
Imaging
Depth Estimation
Omni focus Imaging
Background Subtraction
Discrete Optimization
Camera Calibration
Generative Focus Measure
Discrete Optimization
Edge Based
Discrete Optimization
7
Thesis Outline
Optimization Framework
Imaging
Depth Estimation
Omni focus Imaging
Background Subtraction
Discrete Optimization
Camera Calibration
Generative Focus Measure
Discrete Optimization
Edge Based
Discrete Optimization
8
Optimization Framework
9
Labeling Problem
Input Given a set of Sites S and Labels L
10
Labeling Problem
Problem Optimally assign Labels to Sites
2
5
Total number of possible assignments C
1
3
4
Optimal assignment is encoded in an Energy
Function
11
Optimization Framework
12
Markov Random Fields (MRF)
Given a labeling , it belongs to a MARKOV
RANDOM FIELD iff the following two hold.
13
Optimization Framework
14
Ensure Optimal Labeling ?
A Labeling l can be realized if some Observation
d is given
Labeling l belongs to MRF
Given d what is the optimal l ?
15
Bayesian Justification
For optimality labeling in Bayesian Framework
Maximum A Posteriori (MAP) estimate
16
Bayesian Justification
Prior is defined since l belongs to a Markov
Random Field (MRF)
17
Optimization Framework
18
Energy Function E
Taking negative on MAP makes Maximization a
Energy Minimization problem
Assignment Cost
19
Energy Function E
20
Optimization Framework
21
Graph Cuts for Minimization
Source
Graph G
22
Thesis Outline
Optimization Framework
Imaging
Depth Estimation
Omni focus Imaging
Background Subtraction
Discrete Optimization
Camera Calibration
Generative Focus Measure
Discrete Optimization
Edge Based
Discrete Optimization
23
Lens Imaging
Lens
Object
Optical Axis
Image
24
Two Limitations
Depth of Field (DoF)
Narrow Field of View (FoV)
25
Two Limitations
Depth of Field (DoF)
Narrow Field of View (FoV)
26
Depth of Field
Sensor Plane
Lens
A Range of depths get focused at same pixel
location in an image.
27
Sample Image
But, we want images where all objects are in
FOCUS !!
28
Desired Image
Something like this where ALL objects are in
FOCUS !!
29
Two Limitations
Depth of Field (DoF)
Narrow Field of View (FoV)
30
Field of View
Sensor Plane
1 CCD 1 pixel
Lens
FOV is limited by the dimensions of Sensor Plane
31
Sample Image
SMALL Field of View
32
Desired Image
LARGE Field Of View
33
Solving DoF and FoV Simultaneously
34
Solving DoF and FoV Simultaneously
35
Solving DoF and FoV Simultaneously
36
Solving DoF and FoV Simultaneously
37
Solving DoF and FoV Simultaneously
38
A Modified Imaging System
Non-Frontal Imaging CAMera (NICAM) Ahuja93
Conventional Camera
Frontal Sensor
Single depth is captured
39
Non-Frontal Imaging CAMera (NICAM)
Modified Camera
Non Frontal Sensor
Multiple depths are captured simultaneously
40
Non-Frontal Imaging CAMera (NICAM)
Rotate NICAM to get large FOV
41
Non-Frontal Imaging CAMera (NICAM)
Rotate NICAM to get large FOV
42
Thesis Outline
Optimization Framework
Imaging
Depth Estimation
Omni focus Imaging
Background Subtraction
Discrete Optimization
Camera Calibration
Generative Focus Measure
Discrete Optimization
Edge Based
Discrete Optimization
43
Calibration
Tilted Sensor Plane
Pan Center

• Required for pixel level registration of NICAM
  images.
• Pan Centering of NICAM
• Calculating the tilt of the sensor plane

44
Various Coordinate Systems

• Checkerboard is placed in front of the camera
• Camera is placed on a rotating stage
• Coordinate systems assigned to World, Stage and
  Camera

45
Outline
Pan Centering
Tilt of Lens
46
Outline
Pan Centering
Tilt of Lens
47
Pan Centering
Align the optic center with the rotation axis.
Camera is movable on the Stage
Rotation Axis
48
Pan Centering
Align the optic center with the rotation axis.
Camera is movable on the Stage
Rotation Axis
49
Pan Centering Algorithm
STEP 1 Take images of a checkerboard pattern by
rotating NICAM
50
Pan Centering Algorithm
STEP 1 Take images of a checkerboard pattern by
rotating NICAM
51
Pan Centering Algorithm
STEP 1 Take images of a checkerboard pattern by
rotating NICAM
52
Pan Centering Algorithm
STEP 1 Take images of a checkerboard pattern by
rotating NICAM
53
Pan Centering Algorithm
STEP 1 Take images of a checkerboard pattern by
rotating NICAM
54
Pan Centering Algorithm
STEP 2 Use MATLAB Calibration toolbox to obtain
extrinsic parameters of each camera position
Yw
(R t)
Zw
Xw
R Rotation t Translation
Extrinsic Parameters (R and t) Camera coordinate
system (c) in a fixed world coordinate system (w)
attached to the checker board.
55
Pan Centering Algorithm
STEP 3 Since the camera is not pan centered,
camera positions are obtained in a elliptical arc
as shown below
56
Pan Centering Algorithm
STEP 4 Project the points on the XY plane and do
circle fitting to obtain the center of the circle
8 mm
Centering Error 8 mm. The camera is off center
by this amount
57
Pan Centering Algorithm
STEP 5 Move the camera in X and Y directions on
the stage by the centering error calculated above
and repeat STEP 1 to 4 till very small centering
errors
After Convergence
0.2 mm
Centering Error 0.2 mm
58
Outline
Pan Centering
Tilt of Lens
59
Tilt of the Lens
Define the following Transformations between
Coordinate Systems (CS)
Transformation from Board CS to Camera CS
Transformation from Board CS to World CS
Transformation from World CS to Stage CS
Transformation from Stage CS to Camera CS
60
Tilt of the Lens
R,T Unknown Rotation and Translation matrices
3 rotation variables
Rotation about Xs/Xw axis by ? angle increments
Since pan centering is done, Translation matrix
is 0 3 rotation variables
Obtained from MATLAB Calibration Toolbox during
Pan Centering
61
Tilt of the Lens
Thus, we formulate the following equality
After simplification, we get
and
62
Tilt of the Lens
The Error Minimization Function is
where,
are the rotation angles associated with the
rotation matrices.
6 variables, use more than 6 images to formulate
an over determined set of equations. Minimize
using FMINSEARCH (MatLab)
63
Results
Final Rotation angles between the coordinate
systems
64
Results
Registered Images of a checkerboard
65
Conclusions
A new algorithm for Pan Centering is proposed
An optimization technique obtain lens tilt is
given
Accurate registration of NICAM images is done
66
Thesis Outline
Optimization Framework
Imaging
Depth Estimation
Omni focus Imaging
Background Subtraction
Discrete Optimization
Camera Calibration
Generative Focus Measure
Discrete Optimization
Edge Based
Discrete Optimization
67
Omnifocus Imaging
Extend the Depth of Field of the image
Input Given a set of multi focus images.
Output An image where EVERYTHING is in FOCUS.
68
Multi focused Images
Images with different depths in focus
Focused Depth
69
Multi focused Images
Images with different depths in focus
Focused Depth
70
Multi focused Images
Images with different depths in focus
Focused Depth
71
Multi focused Images
Images with different depths in focus
Focused Depth
72
Omni focused Image
Images with all depths in focus
All Depths in Focus
73
Algorithm Outline

• Capture multi focus images using NICAM.
• Register the multi focus images.
• Find the image in which a pixel is most focused.
• Extract the pixel and paste on a new image.
• Repeat the procedure for all pixels in the
  image.
• Obtain an Omni focus image.

74
Algorithm Outline

• Capture multi focus images using NICAM.
• Register the multi focus images.
• Find the image in which a pixel is most focused.
• Extract the pixel and paste on a new image.
• Repeat the procedure for all pixels in the
  image.
• Obtain an Omni focus image.

75
Imaging Optics
Lens
Sensor Plane at multiple locations
Si
Object
D
Optical Axis
u
v
76
Imaging Some Details
77
Algorithm Outline

• Capture multi focus images using NICAM.
• Register the multi focus images.
• Find the image in which a pixel is most focused.
• Extract the pixel and paste on a new image.
• Repeat the procedure for all pixels in the
  image.
• Obtain an Omni focus image.

78
Algorithm Outline

• Capture multi focus images using NICAM.
• Register the multi focus images.
• Find the image in which a pixel is most focused.
• Extract the pixel and paste on a new image.
• Repeat the procedure for all pixels in the
  image.
• Obtain an Omni focus image.

79
Focus Measure

• Focus Measure is a metric to find the BEST
  FOCUSED pixel
• Conventional metric Energy of Gradient
• Higher the Gradient ---------? More Focused is
  the image

Focused
Blurred
Calculate intensity gradient and assign the best
focus image in which it maximizes
80
But there are Drawbacks!
Near an edge, the gradient maximizes for
defocused pixel
Focused
No Gradient
Defocused
High Gradient
81
Thesis Outline
Optimization Framework
Imaging
Depth Estimation
Omni focus Imaging
Background Subtraction
Discrete Optimization
Camera Calibration
Generative Focus Measure
Discrete Optimization
Edge Based
Discrete Optimization
82
A Generative Focus Measure
INPUT Set of multi focus images captured from
NICAM.
3 is blurred with varying amounts in each image
83
A Generative Focus Measure
STEP 1 Select a pixel location p (x, y) in
image frame k.
k
p (x, y)
Assume I(x, y) is a focused pixel
84
A Generative Focus Measure
STEP 2 Calculate the radius R of blur produced
at location (x, y) in image frame k-d. Similarly
in frame kd.
k
kd
k-d
k
p (x, y)
From Imaging Optics,
Std. Dev. for Gaussian Blur
85
A Generative Focus Measure
STEP 3 Obtain the Gaussian Blur Kernel and blur
p(x, y) to obtain new intensities p and p for
k-d and kd frames respectively.
Gaussian Blur Kernel
New intensity for k-d and kd frames
86
A Generative Focus Measure
STEP 4 Compare p and p with the actual
intensities at (x, y) location in k-d and kd
frames.
Let actual intensities be q(x, y) in k-d frame
and r(x, y) in kd frame.
The criteria for comparison is
87
A Generative Focus Measure
STEP 5 If the criteria is satisfied, k becomes a
candidate frame in which p could be focused.
Calculate the focus measure
Due to ambiguity, there could be multiple such
candidates for each pixel. (Proof in thesis)
The focus measure for frame k being in focus at p
(x, y) is
88
Algorithm Outline

• Capture multi focus images using NICAM.
• Register the multi focus images.
• Find the image in which a pixel is most focused.
• Extract the pixel and paste on a new image.
• Repeat the procedure for all pixels in the
  image.
• Obtain an Omni focus image.

89
Thesis Outline
Optimization Framework
Imaging
Depth Estimation
Omni focus Imaging
Background Subtraction
Discrete Optimization
Camera Calibration
Generative Focus Measure
Discrete Optimization
Edge Based
Discrete Optimization
90
Discrete Optimization
To obtain smooth and fast solution, omnifocus
imaging is formulated in a discrete optimization
framework
Labels 1,N, where N is the number of multi
focus images
91
Algorithm Outline

• Capture multi focus images using NICAM.
• Register the multi focus images.
• Find the image in which a pixel is most focused.
• Extract the pixel and paste on a new image.
• Repeat the procedure for all pixels in the
  image.
• Obtain an Omni focus image.

92
Results Synthetic
Two Checkerboards at different depths
Single Checkerboard
93
Results Real Data Set1
94
Results Real Data Set2
95
Conclusions
Omnifocus Imaging is proposed in an Optimization
framework
New Generative Focus Measure is proposed
Fast convergence since Graph cuts takes few
seconds to minimize E
96
Thesis Outline
Optimization Framework
Imaging
Depth Estimation
Omni focus Imaging
Background Subtraction
Discrete Optimization
Camera Calibration
Generative Focus Measure
Discrete Optimization
Edge Based
Discrete Optimization
97
Outline
98
Outline
99
Depth Estimation
Estimate the 3D Depth of objects in a scene
Input Given a set of multi focus images.
Output Depth Map of the scene
100
Input
Set of multi focus images captured from NICAM
101
Outline
102
Thesis Outline
Optimization Framework
Imaging
Depth Estimation
Omni focus Imaging
Background Subtraction
Discrete Optimization
Camera Calibration
Generative Focus Measure
Discrete Optimization
Edge Based
Discrete Optimization
103
Its a Labeling Problem!
Object at different depths produce a different
set of multi focus images
Label set is possible depth values in the 3D world
104
Outline
105
Discrete Optimization Approach
Potts Model
Cost of assigning depth label to pixel p Focus
measure at p across the set of multi focus images.
Apply Graph Cuts to E to obtain depth map
106
Outline
107
Results 1
108
Results 2
109
Results 3
110
Conclusions
Depth Estimation is proposed in an Optimization
framework
Obtain difficult to obtain smooth and sharp depth
boundaries.
Fast convergence since Graph cuts takes few
seconds to minimize E
111
Thesis Outline
Optimization Framework
Imaging
Depth Estimation
Omni focus Imaging
Background Subtraction
Discrete Optimization
Camera Calibration
Generative Focus Measure
Discrete Optimization
Edge Based
Discrete Optimization
112
Outline
113
Outline
114
Background Subtraction
Background Static/Slow moving objects
Foreground Moving objects
115
Gaussian Mixture Model (GMM)
Intensities in an image are modeled as Mixture of
K Gaussians
The weights and parameters of each Gaussian are
learned with time
Able to model multimodal background distributions
116
Outline
117
Intermodal Train Monitoring
118
Outline
119
Thesis Outline
Optimization Framework
Imaging
Depth Estimation
Omni focus Imaging
Background Subtraction
Discrete Optimization
Camera Calibration
Generative Focus Measure
Discrete Optimization
Edge Based
Discrete Optimization
120
Edge Continuous Optimization
STEP 1 Apply Edge Detection on each frame
121
Edge Continuous Optimization
STEP 2 Detect Top Edge
122
Edge Continuous Optimization
STEP 3 Detect Side Edges
123
Edge Continuous Optimization
STEP 4 GMM learning near the edges of gaps
Thus, background is subtracted
124
Outline
125
Thesis Outline
Optimization Framework
Imaging
Depth Estimation
Omni focus Imaging
Background Subtraction
Discrete Optimization
Camera Calibration
Generative Focus Measure
Discrete Optimization
Edge Based
Discrete Optimization
126
Discrete Optimization
Background Subtraction is modeled as Velocity
Estimation
Energy Function is given as
Label set f is unknown velocity of the train in
pixel shifts per frame
v Maximum possible velocity
127
Pictorial Explanation
At any pixel location, take a window of size w_p
and cross correlate across frames with different
velocity shifts v.
128
Data Term 0
Average out NCC over (2n1) frames
p
Input
Background Removed
Data Cost Vs Velocity plot at p
129
Data Term 1
Weighted NCC over distance of a frame from the
reference frame
p
Input
Background Removed
Data Cost Vs Velocity plot at p
130
Data Term 2
From DT1, choose the max from both sides of
reference frame
p
Input
Background Removed
Data Cost Vs Velocity plot at p
131
Outline
132
Results
Input Frames
Velocity Map
Velocity Map after Graph Cuts
Background Subtracted Frames
133
Comparison
Input Frames
Template based
GMM based
Proposed method
Proposed Technique performs better than
conventional techniques
134
Conclusions
Edge based and GMM based background subtraction
techniques is proposed
Background Subtraction for Train Monitoring is
proposed in a Discrete Optimization Framework
Proposed Discrete Optimization technique gives
better results than existing methods.
135
Thank You !!