Vision Sensors for Stereo and Motion

1
Vision Sensors for Stereo and Motion

• Joshua Gluckman
• Polytechnic University

2
Stereo Vision
depth map
3
Stereo With Mirrors
Gluckman and Nayar (CVPR 99)
4
Why Use Mirrors?

• Identical system response
• Better stereo matching
• Faster stereo matching

5
Why Use Mirrors?

• Identical system response
• Better stereo matching
• Faster stereo matching
• Data acquisition
• No synchronization
• Data Storage

6
Stereo Systems Using Mirrors
Mitsumoto 92
Goshtasby and Gruver 93
Inaba 93
Teoh and Zhang 84
Mathieu and Devernay 95
Zhang and Tsui 98
7
Geometry and Calibration
8
Background Relative Orientation
p
p
C
C
R,t 6 parameters
9
Background Epipolar Geometry
p
p
C
e
C
e
10
Background Epipolar Geometry
3
p
p
C
e
C
e
4
Epipolar geometry 7 parameters
11
Background Epipolar Geometry
3
p
p
C
e
C
e
4
Epipolar geometry 7 parameters
12
One Mirror Relative Orientation
mirror
virtual camera
camera
13
One Mirror Relative Orientation
virtual camera
camera
3 parameters
14
One Mirror Relative Orientation
virtual camera
camera
3 parameters
15
One Mirror Epipolar Geometry
2 parameters location of epipole
16
(No Transcript)
17
Two Mirrors Relative Orientation
D
virtual camera
virtual camera
camera
18
Two Mirrors Relative Orientation


-
1
D
D
D
D
D
2
1
2
1
virtual camera
virtual camera
D
2
camera
19
Two Mirrors Relative Orientation
virtual camera
virtual camera
q
5 parameters
camera
20
Two Mirrors Epipolar Geometry
6 parameters
2
p
p
V
e
V
e
21
Two Mirrors Epipolar Geometry
p
p
epipole e
epipole e
22
Two Mirrors Epipolar Geometry
p1
p1
p2
p2
epipole e
epipole e
p3
p3
p4
p4
23
(a)
(b)
(c)
(d)
24
Calibration Parameters
Relative orientation
Epipolar geometry
25
Mirror Stereo Systems
26
(No Transcript)
27
Real Time Stereo System
Calibrate
28
Rectification of Stereo Images
Perspective transformations
29
Why Rectify Stereo Images?

• Fast stereo matching
• O(hw2s) ? O(hw2)
• Removes differences in rotation and scale

30
Not All Rectification Transforms Are the Same
31
Rectification Previous Methods
Ayache and Hansen 88
3D methods need calibration
Faugeras 93
Robert et al. 93
2D methods rectify from epipolar geometry
Hartley 98
Loop and Zhang 99
Roy et al. 97
Non-perspective transformations
Pollefeys et al. 99
32
The Bad Effects of Resampling the Images

• Creation of new pixels causes
• Blurs the texture
• Additional computation
• Loss of pixels
• Loss of information
• Aliasing

Gluckman and Nayar CVPR 01
33
Measuring the Effects of Resampling
determinant of the Jacobian
change in local area
34
Measuring the Effects of Resampling
determinant of the Jacobian
change in local area
35
Measuring the Effects of Resampling
determinant of the Jacobian
change in local area
36
Change In Aspect Ratio Preserves Local Area
pixels created
pixels lost
37
Skew Preserves Local Area
aliasing
38
Minimizing the Effects of Resampling
change in local area

• P and P must be rectifying transformation
• No change in aspect ratio and skew

39
The Class of Rectifying Transformations
Fundamental matrix
Rotation and translation
40
The Class of Rectifying Transformations
41
The Class of Rectifying Transformations
42
The Class of Rectifying Transformations
6 parameters
43
The Class of Rectifying Transformations
no skew
maintain aspect ratio
2 parameters
44
The Class of Rectifying Transformations
scale
perspective distortion
2 parameters
45
Finding the Best Rectifying Transform
change in local area
Find p1 and p8 that minimize e
46
Finding the Best Rectifying Transform
change in local area
Find p1 and p8 that minimize e

• e is quadratic in p1 so the optimal scale can
  be found as a function of p8
• e is a 16th degree rational polynomial in p8

47
Finding the Best Rectifying Transform

• e1 and e2 are symmetric convex polynomials
• e1 has a minimum at p8 0
• e2 has a minimum at p8 f5

The minimum of e is between 0 and f5
48
Finding the Best Rectifying Transform
e1 and e2 depend on the location of epipoles
epipoles at the same distance
49
Finding the Best Rectifying Transform
e1 and e2 depend on the location of epipoles
epipoles at a distance of 3 and 10
50
Rectifying While Minimizing Resampling Effects
Step 1 Rotate and translate the epipolar
geometry
51
Rectifying While Minimizing Resampling Effects
Step 1 Rotate and translate the epipolar
geometry Step 2 Find p1 and p8
that minimize e
52
Rectifying While Minimizing Resampling Effects
Step 1 Rotate and translate the epipolar
geometry Step 2 Find p1 and p8
that minimize e Step 3 Construct P and P
53
Rectifying While Minimizing Resampling Effects
Step 1 Rotate and translate the epipolar
geometry Step 2 Find p1 and p8
that minimize e Step 3 Construct P and
P Step 4 Rectify the images using the
perspective transformations
54
(No Transcript)
55
Rectification
56
Rectification and Stereo Matching
57
Rectified Stereo Using Mirrors
Not rectified
Rectified
Gluckman and Nayar CVPR 00
58
When Is a Stereo System Rectified?

• No relative rotation between stereo cameras
• Direction of translation along the scan lines
  (x-axis)
• Identical intrinsic parameters (focal length)

59
Rectified Stereo Sensors
1
left virtual camera
4
3
5
2
right virtual camera
D
60
Rectified Stereo Sensors
1
left virtual camera
4
3
5
2
right virtual camera
61
What Constraints Must Be Satisfied?
62
How Many Reflections?
Even number of reflections
Odd number of reflections
63
Example Four mirrors Wont Work
64
What Constraints Must Be Satisfied?
65
Single Mirror Rectified Stereo
66
Single Mirror Rectified Stereo
b
camera
virtual camera
67
Three Mirror Rectified Stereo
68
Three Mirror Rectified Stereo
69
A Three Mirror Solution
9 d.o.f. 4 constraints 5 parameter family
of solutions
70
Sensor Size
9 d.o.f. 4 constraints 5 parameter family
of solutions
71
Optimized Solutions
72
Rectified Stereo Sensors
Mirrors
Mirror
73
Rectified Images and Depth Maps
74
Misplacement of the Camera
Mirrors
Mirror
75
Misplacement of the Camera
Mirrors
Mirror
Invariant to misplacement of camera center
76
Misplacement of the Camera
Mirrors
Mirror
Insensitive to tilt of optical axis
77
Misplacement of the Camera
Mirrors
Mirror
Dependent on pan of optical axis
78
Split Shot Stereo Camera
Nikon Coolpix camera
mirror attachment
79
Image Sensors for Motion Computation
80
Camera Motion
motion
rotation, translation, depth
81
(No Transcript)
82
e,e
Anadan and Avidan (ECCV 00)
83
Gluckman and Nayar ICCV 98
Aloimonos et al
84
(No Transcript)
85
(No Transcript)
86
(No Transcript)
87
(No Transcript)
88
(No Transcript)
89
(No Transcript)
90
Future Work
91
Split Shot Stereo Camera
Nikon Coolpix camera
mirror attachment
92
Split Shot Stereo Camera
93
Split Shot Stereo Camera
94
(No Transcript)
95
(No Transcript)
96
(No Transcript)
97
(No Transcript)
98
(No Transcript)
99
(No Transcript)
100
(No Transcript)
101
(No Transcript)
102
The Class of Rectifying Transformations
Rectification projects the images onto a plane
parallel to the camera centers
e
e
p1 changes the distance and p8 changes the tilt
of the rectifying plane
103
Sensing
Pre-processing
Computation