Multi-view Stereo via Volumetric Graph-cuts

1
Multi-view Stereo via Volumetric Graph-cuts

• George Vogiatzis, Philip H. S. Torr Roberto
  Cipolla

2
Shape From Images
3
Dense Stereo reconstruction problem

• Input
• Set of images of a scene II1,,IK
• Camera matrices P1,,PK
• Output
• Surface model

4
Shape parametrisation

• Disparity-map parameterisation
• MRF formulation good optimisation techniques
  exist (Graph-cuts, Loopy BP)
• MRF smoothness is viewpoint dependent
• Disparity is unique per pixel only functions
  represented

5
Shape parametrisation

• Volumetric parameterisation e.g. Level-sets,
  Space carving etc.
• Able to cope with non-functions
• Convergence properties not well understood
• Memory intensive
• For Space carving, no simple way to impose
  surface smoothness

6
Solution ?

• Cast volumetric methods in MRF framework
• Benefits
• General surfaces can be represented
• Optimisation is tractable (MRF solvers)
• Occlusions can be approximately modelled
• Smoothness is viewpoint independent

7
Graph cuts
24
5
40
21
23
20
5
12
1
13
50
40
4
13
3
30
8
Graph cuts
24
5
40
21
23
20
5
12
1
13
50
40
4
13
3
30
9
Graph cuts
24
5
40
21
23
20
5
12
1
13
50
40
4
13
3
30
5514318
10
Volumetric Graph cuts for segmentation

• Volume is discretized
• A binary MRF is defined on the voxels
• Regular grid (6 or 26 neighbourhood)
• Voxels are labelled as OBJECT and BACKGROUND
• Labelling cost set by OBJECT / BACKGROUND
  intensity statistics
• Compatibility cost set by edge intensities

11
Volumetric Graph cuts for stereo

• How to define Inside and Outside labels
• How to deal with occlusion

12
Volumetric Graph cuts
13
Face
14
Face - Visual Hull
15
Face - Slice
16
Face - Slice with graphcut
17
Face - Reconstruction
18
Protrusion problem

• Balooning force
• favouring bigger volumes
• L.D. Cohen and I. Cohen. Finite-element methods
  for active contour models and balloons for 2-d
  and 3-d images. PAMI, 15(11)11311147, November
  1993.

19
Protrusion problem

• Balooning force
• favouring bigger volumes
• L.D. Cohen and I. Cohen. Finite-element methods
  for active contour models and balloons for 2-d
  and 3-d images. PAMI, 15(11)11311147, November
  1993.

20
Protrusion problem
21
Protrusion problem
22
Graph
23
Results

• Model House

24
Results

• Model House Visual Hull

25
Results

• Model House

26
Results

• Stone carving

27
Results

• Haniwa

28
Summary

• Novel formulation for multiview stereo
• Volumetric scene representation
• Computationally tractable global optimisation
  using Graph-cuts.
• Visual hull for occlusions and geometric
  constraint

29
Benefits

1. General surfaces and objects can be fully
   represented and computed as a single surface.
2. The representation and smoothness constraint is
   image and viewpoint independent.
3. Multiple views of the scene can be used with
   occlusions approximately modelled.
4. Optimisation is computationally tractable, using
   existing max-flow algorithms.