Dynamic 3D Scene Analysis from a Moving Vehicle

1
Dynamic 3D Scene Analysis from a Moving Vehicle

• Bastian Leibe, Nico Cornelis, Kurt Cornelis, Luc
  Van Gool
• Computer Vision Laboratory
• ETH Zurich
• CVPR, Minneapolis, 20.06.2007

VISICSKU Leuven

2
Dynamic 3D Scene Analysis

• Objectives
• Detect objects in environment
• Localize them in 3D
• Predict their future motion

• Challenges
• We are moving
• Objects are moving
• Ground may not be planar
• f

3
Main Ideas

• Integrate and closely couple different modalities
• Structure-from-Motion
• 2D Object detection
• 3D Trajectory estimation
• ? Tasks become easier through integration
• Formulation as hypothesis selection
• Delay making hard decisions
• Collect many individual hypotheses
• Global optimization to explain the observed data
• For each image ? 2D Object detections
• For temporal window ? 3D Trajectories

4
Outline

• Geometric Context Estimation
• Structure-from-Motion
• Ground plane estimation
• Object Detection
• Integration of geometric context
• Multi-category/multi-viewpoint integration
• Temporal Integration Tracking
• Localization of static objects
• Trajectory estimation for dynamic objects
• MDL hypothesis selection

5
Geometric Context Estimation

• Results from SfM
• Camera calibration for each frame

Real-time performance 30 fps (including
windowed bundle adjustment)
Cornelis et al., CVPR06
6
Geometric Context Estimation

• Results from SfM
• Camera calibration for each frame
• Positions of wheel base points

7
Geometric Context Estimation

• Results from SfM
• Camera calibration for each frame
• Positions of wheel base points

• Compute normals locally

8
Geometric Context Estimation

• Results from SfM
• Camera calibration for each frame
• Positions of wheel base points

• Compute normals locally
• Average over spatial window
• ? Ground plane estimate

9
Outline

• Geometric Context Estimation
• Structure-from-Motion
• Ground plane estimation
• Object Detection
• Integration of geometric context
• Multi-category/multi-viewpoint integration
• Temporal Integration Tracking
• Localization of static objects
• Trajectory estimation for dynamic objects
• MDL hypothesis selection

10
Appearance-Based Object Detection

• Battery of 51 single-view ISM detectors
• (Semi-profile detectors also mirrored)
• Each based on 3 local cues
• HarLap, HesLap, DoG interest regions
• Local Shape Context descriptors
• Result detections segmentations

Leibe et al., CVPR05, BMVC06
11
2D/3D Interactions

• Likelihood of 3D hypothesis H
• Given image I and a set of 2D detections h
• 2D recognition score
• Expressed in terms of per-pixel p(figure)
  probabilities

Leibe et al., CVPR05
12
2D/3D Interactions

• Likelihood of 3D hypothesis H
• Given image I and a set of 2D detections h
• 3D prior
• Distance prior (uniform range)
• Size prior (Gaussian)
• ? Significantly reduced search space

Search corridor
Hoiem et al., CVPR06
13
2D/3D Interactions

• Likelihood of 3D hypothesis H
• Given image I and a set of 2D detections h
• 2D/3D transfer
• Two image-plane detections are consistent if they
  correspond to the same 3D object
• ? Cluster 3D detections
• ? Multi-viewpoint integration

14
Image-Plane Hypothesis Selection

• Quadratic Boolean Optimization Problem (from MDL)
• Goal
• Select set of hypotheses H that best explains the
  image
• Constraint
• Each pixel can at most belong to a single
  hypothesis

Leonardis et al,95
15
Image-Plane Hypothesis Selection (2)

• Quadratic Boolean Optimization Problem (from MDL)
• Individual scores (diagonal terms)

Leonardis et al,95
16
Image-Plane Hypothesis Selection (2)

• Quadratic Boolean Optimization Problem (from MDL)
• Individual scores (diagonal terms)
• Interaction costs (off-diagonal terms)

Leonardis et al,95
17
Detections Using Ground Plane Constraints
left camera 1175 framesdata _at_ 25fps
18
Detections Using Ground Plane Constraints
left camera 289 framesdata _at_ 3 fps
19
Quantitative Results

• Detection performance on 2 test sequences
• All cars/pedestrians annotated that were gt50
  visible
• Ground plane constraint significantly improves
  precision
• Performance cars 0.34 fp/image at 47 recall

ped 1.65 fp/image at 43 recall
20
Outline

• Geometric Context Estimation
• Structure-from-Motion
• Ground plane estimation
• Object Detection
• Integration of geometric context
• Multi-category/multi-viewpoint integration
• Temporal Integration Tracking
• Localization of static objects
• Trajectory estimation for dynamic objects
• MDL hypothesis selection

21
Tracking Dynamic Objects

• Spacetime trajectory analysis
• Link up detections to form physically plausible
  ST trajectories
• Select set of ST trajectories that best explain
  the data

22
Spacetime Trajectory Analysis
23
3D Localization for Static Objects

• 3D detections form clusters at static car
  locations
• Mean-shift search to find set of 3D hypotheses H
  (location orientation)
• Estimate orientation from cluster shape and
  detector output

24
Trajectory Construction

• Appearance model
• 8?8?8 RGB Color histogram
• Computed over object segmentationsfrom ISM
  object detector
• ? Mean color histogram for trajectory
• Dynamic model on the ground plane

25
Spacetime Trajectory Estimation

• Trajectory growing
• Collect detections in event cone
• Evaluate under trajectory

26
Spacetime Trajectory Estimation

• Trajectory growing
• Collect detections in event cone
• Evaluate under trajectory
• Adapt trajectory

27
Spacetime Trajectory Estimation

• Trajectory growing
• Collect detections in event cone
• Evaluate under trajectory
• Adapt trajectory
• Iterate

28
Spacetime Trajectory Estimation

• Trajectory growing
• Collect detections in event cone
• Evaluate under trajectory
• Adapt trajectory
• Iterate

29
Spacetime Trajectory Estimation

• Trajectory growing
• Collect detections in event cone
• Evaluate under trajectory
• Adapt trajectory
• Iterate
• Trajectory selection
• Start search from each detection
• Collect all resulting trajectories
• Perform hypothesis selection

H2
H1
30
Trajectory Hypothesis Selection

• Quadratic Boolean Optimization Problem (from MDL)
• Goal
• Select set of trajectories that best explains the
  observations.
• Constraints
• Each detection can at most belong to a single
  trajectory.
• No two trajectories may intersect at any point in
  time.

Leonardis et al,95
31
Trajectory Hypothesis Selection

• Quadratic Boolean Optimization Problem (from MDL)
• Individual scores (diagonal terms)

Leonardis et al,95
32
Trajectory Hypothesis Selection

• Quadratic Boolean Optimization Problem (from MDL)
• Individual scores (diagonal terms)
• Interaction costs (off-diagonal terms)

Leonardis et al,95
33
Incremental Computation

• Trajectories can be grown incrementally!
• Keep old trajectories
• Extend them with new detections
• Start new event cones from new detections down
  the timeline
• Interaction matrix is updated
• Old entries weighted with temporal discount
• Only few updates necessary

t1
H2
H1
34
Online Scene Analysis Results
Results obtained usingonly detections
fromcurrent previous frames
35
Limitations and Future Work

• SfM
• Small look-ahead through windowed bundle
  adjustment
• ? Use additional cues from dense stereo
• Object Detection
• Pedestrian detection can still be improved
• Bicyclists treated as pedestrians
• ? Use dedicated detector motion model for
  better results
• Trajectory Estimation
• No feedback yet to object detection
• ? Integrate feedback to improve results

36
Conclusion

• Integrated system for dynamic 3D scene analysis
• Structure-from-Motion
• Object detection
• 3D Localization for static scene objects
• Trajectory estimation for dynamic objects
• System applied to challenging real-world task
• Object detection 3D localization/tracking with
  moving camera
• Capable of working at low frame rates
• Key message
• Individual components are becoming usable for
  real-world tasks
• Each individual task can benefit from integration
  with others
• ? Many more such cross-links exist and should be
  exploited!

37

• Thank you very much for your attention!

This research was supported by Toyota Motor
Europe and EC projects HERMES and DIRAC.
http//www.vision.ethz.ch/
http//www.esat.kuleuven.be/psi/visics/