Background Removal of Multiview Images by Learning Shape Priors

1
Background Removal of Multiview Imagesby
Learning Shape Priors

• Yu-Pao Tsai, Cheng-Hung Ko, Yi-Ping Hung, and
  Zen-Chung Shih

2
ABSTRACT

• Image-based rendering has been successfully used
  to display 3-D objects for many applications.
  A well-known example is the object movie, which
  is an image-based 3-D object composed of a
  collection of 2-D images taken from many
  different viewpoints of a 3-D object.

3
INTRODUCTION
4
MVI ( Multiview Image)

? ( omicron ) T ( thet ) pan angle F ( phi )
tilt angle
MVI has three basic characteristics
1) When an equi-tilt set of the MVI is captured,
a large proportion of the background scene
is static.
2) Only one interesting object is presented in
every image of the MVI.
3) The foreground and background color
distributions are distinct in most cases.
5
Proposed Flowchart
The proposed approach aims to let every
single image segmentation,rather than only those
in neighboring viewing directions
To take the shape prior into account, the
user is required to selectd a subset of
acceptably segmented images. The 3-D shape is
then generated from these selected images.
6
AUTOMATIC INITIAL SEGMENTATION Graph Cut Image
Segmentation

• Graph cut image segmentation requires the user to
  interactively mark some pixels as being inside
  the foreground objects, and others as a part of
  the background scene.
• All the other pixels are considered to be
  unknown, and then they can be classified into the
  foreground or background by Markov random field
  (MRF) optimization.

7
AUTOMATIC INITIAL SEGMENTATION Trimap Labeling

• Trimap consisting of labels drawn from
• Trimap labeling method ß-labeling and
  ?-labeling
• ß based on the color difference. zero-mean
  normalized cross correlation (ZNCC)
• ?  based on the foreground model.

?  xi .
 ß beta.
µmu.
1.If the color of a pixel varies, the pixel
should be the background and labeled
2.mathematical morphology is applied to filter
out the remained noises such that only one
region exists
3.Each pixel whose color differs widely from the
background model can be labeled .
The ß and µ regions are collected and clustered
by using K-means.
Let denote the mean color of the th cluster
for image
Each pixel p with the label µ in the image
is examined and labeled ?
Where is a strict threshold to ensure that
only the pixels that differ widely from the
background model are labeled ?
8
AUTOMATIC INITIAL SEGMENTATION Trimap Labeling
MVI
ß Labeling
? Labeling Background blackForeground
white Unknow gray
Mathematical morphology in (a)
9
SEGMENTATION WITH SHAPE PRIORS Volumetric
Graph Cuts
For each voxel , let be the
photo-consistency score of , where a lower
value represents a better photo-consistency.
Let be the volume between and the
base surface.
The true surface is determined by
finding the global minimum of the energy function
among all candidate surfaces .
In (6), the first integral tends toward a
photo-consistent surface, while the second,
called the ballooning term, prefers a fatter
reconstructed model.
10
Discrete Medial Axis ConstraintEnergy Function
Analysis
Let be a neighborhood system defined for ,
which containing the set of all pairs of
neighboring voxels.
Let be a family of random
variables defined on the set , in which
each variable takes a label from
D(p) is the penalty according to how well the
voxel fits into the given label, while B(p,q)
indicates whether the surface is likely to pass
through the edge between p and q .
B(p,q) can maintain the smoothness prior
11
Discrete Medial Axis ConstraintImposing the DMA
Constraint
The medial axis is represented by a set of
discrete voxels interior to the 3-D object,
called discrete medial axis (DMA).
To compute the DMA of the base surface, which is
assumed to be an adequate approximation of the
DMA of the true surface.
let be the set of voxels in the DMA. Let dp
be the minimum distance from the voxel to its
nearest voxel in
(12) guarantees that the voxels in are always
labeled as being inside the surface.
12
Discrete Medial Axis ConstraintSegmentation
Refinement
C1.C2.C4 built hull. C3.C5 projection
13
EXPERIMENTSInitial Segmentation Results
Error
MVI
Trimap Labeling (ß Labeling ? Labeling)
Error
14
EXPERIMENTSLearning Shape Prior
Without DMA andballooning increased
a.Visual hull base surface b. DMA of ( a ) c.d.
reconstructed model

1. Visual hull
2. DMA of ( a )
3. MVI
4. Our method

15
EXPERIMENTSRectification of Segmentation Errors
Projection of the reconstructed
16
EXPERIMENTSRectification of Segmentation Errors
MVI
Trimap Labeling ß Labeling ? Labeling
Automatic initial segmentation
Projection of the reconstructed
Shape priors
17
EXPERIMENTSRectification of Segmentation Errors
Shape prior1 use 10 images Shape prior2 use 20
images
18

• Thank you for your
• listening !
• 2008.12.09