Multilabel energy minimization via graph cuts

1
Multi-label energy minimization via graph cuts
2
Stereo matching

• Extract correspondences between similar images

3
(No Transcript)
4

• Correspondences via horizontal shifts (called
  disparities)

5

• Correspondences via horizontal shifts (called
  disparities)

6
Stereo matching

• Visualization of disparities (disparity map)
• Disparity inversely proportional to depth
• Stereo matching trivial for humans
• How about computers?

7
Window based approach

• Winner-takes-all approach
• Windows matched independently
• Small or large windows can be used
• With a simple trick, running time can be made
  independent of window size

8
Small vs large windows
small window
large window

• better at boundaries
• noisy in low texture areas

• better in low texture areas
• blurred boundaries

9
Window based approach

• Assumes disparities are independent to each other
  (clearly a bad modeling assumption)
• Optimization is of course trivial in this case
  (separable objective function)
• Fast local solutions
• We need to introduce spatial coherence into our
  energy function
• better modeling,
• but resulting optimization problem gets harder

10
Scan-line approaches

• Match scan lines independently, i.e., introduce
  coherence only along scanlines(what is the
  resulting MRF?)
• Better than window-based approach

• But still not good enough

11
Scan-line approaches

• We can use belief propagation as our optimization
  engine
• Exact optimum can be computed (MRF graph is
  non-loopy)
• In this case, BP reduces to dynamic programming

correspondence
12
Graph-cut approach

• We will use a 2D grid for our MRF
• We will penalize disparity discontinuities both
  in horizontal or vertical direction
• Much better modeling (spatial coherence along
  AND across scanlines)

13
Graph-cut approach

• Resulting MRF energy
• How can we select the weights wpq?
• Why not just apply loopy-BP in this case?

14
MRF optimization via graph-cuts

• Optimizing MRF energies of the following form

• Belief propagation can not guarantee an optimal
  solution (loopy graph)
• We will use graph-cut based methods (exact
  global optimum in polynomial time)
• But how can this be reduced to a graph-cut
  problem?

15
MRF optimization via graph-cuts
16
MRF optimization via graph-cuts
17
MRF optimization via graph-cuts
Lets concentrate on one pair of neighboring
pixels (p,q)
18
MRF optimization via graph-cuts
Lets concentrate on one pair of neighboring
pixels (p,q)
19
MRF optimization via graph-cuts
The combined energy over the entire grid G is
(photo consistency) cost of vertical edges
cost of horizontal edges (spatial consistency)
20
Scan-line stereo vs. Multi-scan-line stereo
s-t Graph Cuts (multi-scan-line optimization)
Dynamic Programming (single scan line
optimization)
21
Scan-line vs. graph-cut stereo
multi scan line stereo (graph cuts)
single scan-line stereo (DP)
22
Scan-line vs. graph-cut stereo
multi scan line stereo (graph cuts)
single scan-line stereo (DP)