Putting Objects in Perspective

1
Putting Objects in Perspective

• Derek Hoiem
• Alexei A. Efros
• Martial Hebert
• Carnegie Mellon University
• Robotics Institute

2
Understanding an Image
3
Today Local and Independent
4
What the Detector Sees
5
Local Object Detection
True Detection
False Detections
Missed
Missed
True Detections
Local Detector Dalal-Triggs 2005
6
Work in Context

• Image understanding in the 70s

Guzman (SEE) 1968 Hansen Riseman (VISIONS)
1978 Barrow Tenenbaum 1978 Yakimovsky Feldman
1973
Brooks (ACRONYM) 1979 Marr 1982 Ohta Kanade 1973

• Recent work in 2D context

Kumar Hebert 2005 Torralba, Murphy, Freeman
2004 Fink Perona 2003
He, Zemel, Cerreira-Perpiñán 2004 Carbonetto,
Freitas, Banard 2004 Winn Shotton 2006
7
Real Relationships are 3D
Close
Not Close
8
Recent Work in 3D
Han Zu 2003
Oliva Torralba 2001
Torralba, Murphy Freeman 2003
Han Zu 2005
9
Objects and Scenes
Hock, Romanski, Galie, Williams 1978

• Biedermans Relations among Objects in a
  Well-Formed Scene (1981)

• Support
• Size

• Position
• Interposition
• Likelihood of Appearance

10
Contribution of this Paper
Hock, Romanski, Galie, Williams 1978

• Biedermans Relations among Objects in a
  Well-Formed Scene (1981)

• Support
• Size

• Position
• Interposition
• Likelihood of Appearance

11
Object Support
12
Surface Estimation
Image
Support
Vertical
Sky
V-Center
V-Left
V-Right
V-Porous
V-Solid
Hoiem, Efros, Hebert ICCV 2005 Software
available online
13
Object Size in the Image
Image
World
14
Object Size ? Camera Viewpoint
Input Image
Loose Viewpoint Prior
15
Object Size ? Camera Viewpoint
Input Image
Loose Viewpoint Prior
16
Object Size ? Camera Viewpoint
Object Position/Sizes
Viewpoint
17
Object Size ? Camera Viewpoint
Object Position/Sizes
Viewpoint
18
Object Size ? Camera Viewpoint
Object Position/Sizes
Viewpoint
19
Object Size ? Camera Viewpoint
Object Position/Sizes
Viewpoint
20
What does surface and viewpoint say about objects?
Image
P(object)
21
What does surface and viewpoint say about objects?
Image
P(surfaces)
P(viewpoint)
P(object surfaces, viewpoint)
P(object)
22
Scene Parts Are All Interconnected
Objects
3D Surfaces
Camera Viewpoint
23
Input to Our Algorithm
Surface Estimates
Viewpoint Prior
Object Detection
Local Car Detector
Local Ped Detector
Surfaces Hoiem-Efros-Hebert 2005
Local Detector Dalal-Triggs 2005
24
Scene Parts Are All Interconnected
Objects
3D Surfaces
Viewpoint
25
Our Approximate Model
Objects
3D Surfaces
Viewpoint
26
Inference over Tree Easy with BP
Viewpoint
?
Local Object Evidence
Local Object Evidence
Objects
...
o1
on
Local Surface Evidence
Local Surface Evidence
Local Surfaces

s1
sn
27
Viewpoint estimation
Viewpoint Prior
Viewpoint Final
Likelihood
Likelihood
Horizon
Height
Horizon
Height
28
Object detection
Car TP / FP Ped TP / FP
Initial (Local)
Final (Global)
Car Detection
4 TP / 1 FP
4 TP / 2 FP
Ped Detection
4 TP / 0 FP
3 TP / 2 FP
Local Detector Dalal-Triggs 2005
29
Experiments on LabelMe Dataset

• Testing with LabelMe dataset 422 images
• 923 Cars at least 14 pixels tall
• 720 Peds at least 36 pixels tall

30
Each piece of evidence improves performance
Pedestrian Detection
Car Detection
Local Detector from Murphy-Torralba-Freeman 2003
31
Can be used with any detector that outputs
confidences
Car Detection
Pedestrian Detection
Local Detector Dalal-Triggs 2005 (SVM-based)
32
Accurate Horizon Estimation
Dalal- Triggs 2005
Murphy-Torralba-Freeman 2003
Horizon Prior
Median Error
8.5
4.5
3.0
90 Bound
33
Qualitative Results
Car TP / FP Ped TP / FP
Initial 2 TP / 3 FP
Final 7 TP / 4 FP
Local Detector from Murphy-Torralba-Freeman 2003
34
Qualitative Results
Car TP / FP Ped TP / FP
Initial 1 TP / 14 FP
Final 3 TP / 5 FP
Local Detector from Murphy-Torralba-Freeman 2003
35
Qualitative Results
Car TP / FP Ped TP / FP
Initial 1 TP / 23 FP
Final 0 TP / 10 FP
Local Detector from Murphy-Torralba-Freeman 2003
36
Qualitative Results
Car TP / FP Ped TP / FP
Initial 0 TP / 6 FP
Final 4 TP / 3 FP
Local Detector from Murphy-Torralba-Freeman 2003
37
Summary Future Work
Ped
Ped
Car

• Reasoning in 3D
• Object to object
• Scene label
• Object segmentation

38
Conclusion

• Image understanding is a 3D problem
• Must be solved jointly
• This paper is a small step
• Much remains to be done

39
Thank you
40
(No Transcript)
41
A Return to Scene Understanding
Ohta Kanade 1978

• Guzman (SEE), 1968
• Hansen Riseman (VISIONS), 1978
• Barrow Tenenbaum 1978

• Brooks (ACRONYM), 1979
• Marr, 1982
• Ohta Kanade, 1978
• Yakimovsky Feldman, 1973

42
Images