Edgeflow Image Segmentation and other animals

1
Edgeflow Image Segmentation and other animals

• Jackie Assa

2
Outline

• Segmentation Overview
• Edgeflow basics - Manjunath/Ma
• filters
• Textures
• Learning - Malik
• Benchmarking

3
What is segmentation?

• Dividing image into its constituent regions or
  objects

Dilemmas
Object vs. Region
Region vs. Edge
Foreground vs. Background
4
Segmentation
5
Terminology

• Segmentation, grouping, perceptual organization
  gathering features that belong together
• Fitting associating a model with observed
  features
• Top-down segmentation pixels belong together
  because they come from the same object
• Bottom-up segmentation pixels belong together
  because they look similar

6
The goals of segmentation
human segmentation
image
Berkeley segmentation databasehttp//www.eecs.be
rkeley.edu/Research/Projects/CS/vision/grouping/se
gbench/
7
The goals of segmentation
superpixels

• Separate image into coherent objects
• Top-down or bottom-up process?
• Supervised or unsupervised?
• Group together similar-looking pixels for
  efficiency of further processing

X. Ren and J. Malik. Learning a classification
model for segmentation. ICCV 2003.
8
The goals of segmentation

• Separate image into coherent objects
• Top-down or bottom-up process?
• Supervised or unsupervised?
• Group together similar-looking pixels for
  efficiency of further processing
• Related to image compression
• Measure of success is often application-dependent

9
Approaches for Segmentations

• Ingredients
• Intensity values Similarity and discontinuity
• Color information
• Texture information
• Displacement images from motion analysis
• 3D depth images

• Methods
• Thresholding
• Region growing
• Region splitting
• Region merging
• Split and Merge
• Dont really know learning

10
EdgeFlow A Technique for Boundary Detectionand
Image Segmentation()Wei-Ying Ma and B. S.
Manjunath
() Additional improvements made by Manjunath
students over the years, especially Baris
Sumengen
11
Problem Statement
12
Overview (1)

• compute local edge energy for
• Color intensity
• Textures
• Combine to a unified field
• Propagate field to boundaries
• Boundary completion
• Region merging

13
Edge detection

• Convert a 2D image into a set of curves
• Extracts salient features of the scene
• More compact than pixels

14
Origin of Edges
surface normal discontinuity
depth discontinuity
surface color discontinuity
illumination discontinuity

• Edges are caused by a variety of factors

15
Edge detection

• How can you tell that a pixel is on an edge?

16
Profiles of image intensity edges
17
Edge detection

• Detection of short linear edge segments (edgels)
• Aggregation of edgels into extended edges
• (maybe parametric description)

18
Edgel detection

• Difference operators
• Parametric-model matchers

19
Edge is Where Change Occurs

• Change is measured by derivative in 1D
• Biggest change, derivative has maximum magnitude
• Or 2nd derivative is zero.

20
Image gradient

• The gradient of an image
• The gradient points in the direction of most
  rapid change in intensity

21
The discrete gradient

• How can we differentiate a digital image fx,y?
• Option 1 reconstruct a continuous image, then
  take gradient
• Option 2 take discrete derivative (finite
  difference)

22
The Sobel operator

• Better approximations of the derivatives exist
• The Sobel operators below are very commonly used

• The standard defn. of the Sobel operator omits
  the 1/8 term
• doesnt make a difference for edge detection
• the 1/8 term is needed to get the right gradient
  value, however

23
Gradient operators
(a) Roberts cross operator (b) 3x3 Prewitt
operator (c) Sobel operator (d) 4x4 Prewitt
operator
24
Effects of noise

• Consider a single row or column of the image
• Plotting intensity as a function of position
  gives a signal

Where is the edge?
25
Solution smooth first
Where is the edge?
26
Derivative theorem of convolution

• This saves us one operation

27
Laplacian of Gaussian

• Consider

Laplacian of Gaussian operator
Where is the edge?
Zero-crossings of bottom graph
28
2D edge detection filters
Laplacian of Gaussian
Gaussian
derivative of Gaussian

• is the Laplacian operator

29
It all begins with a Gaussian
Gaussian Derivative
GD
DOOG
difference of offset Gaussian
30
Edgeflow Vector Field

• Key operator Difference of Offset Gaussians

Offset 4?
31
Why

• Edges in natural images directional
• Ah, yes Also the brain

32
Examples of best Doog for each pixel
33
Edgeflow Vector Field
Edge
34
Example of resulting field
35
Theory
is a pixel in an image.
is an edge energy at location s along the
orientation theta.
is the probability of finding an image boundary
in the direction theta from s.
is the probability of finding an image boundary
in the direction thetapi from s.
36
Intensity (and color) Edge Flow
37
Texture Edge flow
Wait a second Lets see Gabor Filters
38
Gabor filters
Dennis Gabor 5 June 1900, Budapest,
Hungary

• Zsa Zsa Gabor
• 6 February 1917, Budapest, Hungary

39
Gabor Filters
40
Gabor Filter Bank Example
Scale
Orientation

• Example Gabor filter bank with 3 scale values and
  4 orientation values

41
(No Transcript)
42
Gabor filter responses
43
Standard way to handle texture

• Apply Gabor filter bank and generate a texture
  vector for each pixel.
• Perform Kmeans/GMM/pca on the texture vector
  space, each group centroid is called texton
• For each pixel(texton) neighborhood,
• calculate texton histogram
• The resulting histogram values are the texton
  vector space.

44
Chi square distance between texton histograms
Chi-square
i
0.1
j
k
0.8
45
Were backTexture Edge flow

• Detecting a texture is being done by Gabor
  Filters which define Textons.
• The rest is (almost) the same as Intensity field

46
Texture Edge flow
In the original paper
In more recent papers
The same as intensity but on the textons space
47
So What do we have

• Vector fields of
• Each color (RGB) intensity
• Texture
• Gabor phase

48
Edge Flow integration

• Combining different types
• Combining different directions

49
(No Transcript)
50
Boundary propagation
51
Regions completion and merging
52
(No Transcript)
53
(No Transcript)
54
Learning to Detect Natural Image Boundaries Using
Local Brightness, Color, and Texture Cues and
others David R. Martin, Charless C. Fowlkes,
Jitendra Malik Advances in Neural Information
Processing Systems, volume 14, 2002-2004
55
Original Image
56

• Psychophysics of localization
• Multi-Attribute Boundaries Rivest/Cavanagh 1996
• luminance, color, motion, texture
• Information pooled prior to localization
• Texture Boundaries Landy/Kojima 2001
• frequency, orientation, contrast
• Our approach Supervised learning to optimally
  combine boundary cues.

57
(No Transcript)
58
Individual Features

• 1976 CIE Lab colorspace
• Brightness Gradient BG(x,y,r,?)
• Difference of L distributions
• Color Gradient CG(x,y,r,?)
• Difference of ab distributions
• Texture Gradient TG(x,y,r,?)
• Difference of distributions of V1-like filter
  responses

59
Dataflow
Pb
Image
Optimized Cues
Cue Combination
Brightness
Model
Color
Texture
60
Texture feature

• Texture Gradient TG(x,y,r,?)
• ?2 difference of texton histograms
• Textons are vector-quantized filter outputs

61
Chi square distance between texton histograms
Chi-square
i
0.1
j
k
0.8
(Malik)
62
Cue Combination Models

• Classification Trees
• Top-down splits to maximize entropy, error
  bounded
• Density Estimation
• Adaptive bins using k-means
• Logistic Regression, 3 variants
• Linear and quadratic terms
• Confidence-rated generalization of AdaBoost
  (SchapireSinger)
• Hierarchical Mixtures of Experts (JordanJacobs)
• Up to 8 experts, initialized top-down, fit with
  EM
• Support Vector Machines (libsvm, ChangLin)
• Gaussian kernel, ?-parameterization
• Range over bias, complexity, parametric/non-parame
  tric

63
Classifier Comparison
64
Various Cue Combinations
65
Two Decades of Local Boundary Detection
66
Detector Comparison
Canny
2MM
Us
Human
Image
67
Dataflow
EstimatedAffinity
Image
Region Cues
E
Segment
Edge Cues

• Eij affinity between pixels i and j
• Representation for graph-theoretic segmentation
  algorithms
• Minimum Spanning Trees - Zahn 1971, Urquhart 1982
• Spectral Clustering - Scott/Longuet-Higgins 1990,
  Sarkar/Boyer 1996
• Graph Cuts - Wu/Leahy 1993, Shi/Malik 1997,
  Felzenszwalb/Huttenlocher 1998,
  Gdalyahu/Weinshall/Werman 1999
• Matrix Factorization - Perona/Freeman 1998
• Graph Cycles - Jermyn/Ishikawa 2001

68
Dataflow
EstimatedAffinity (E)
Image
Region Cues
Segment
Edge Cues
69
Pb Images
Canny
2MM
Us
Human
Image
70
Pb Images II
Canny
2MM
Us
Human
Image
71
Pb Images III
Canny
2MM
Us
Human
Image
72
Benchmarking
73
(No Transcript)
74
(No Transcript)
75
How to validate
76
(No Transcript)
77
(No Transcript)
78
Performance Measure

• Match edges to ground truth
• Precision (P) What percentage of edges match
  ground truth.
• Recall (R) What percentage of ground truth is
  correctly identified.
• F-measure gives the performance

79
Quantitative Evaluation

• Compare
• Edgeflow-based Anisotropic Diffusion (EF)
• Perona-Malik Flow (PM)
• Self-Snakes (SS)
• using Berkeley segmentation data set.

80
Performance Results
Margin EF to PM 0.05
81
Comparison Examples
Perona-Malik
Edgeflow
Self-Snakes
82
Comparison Examples
Self-Snakes
Perona-Malik
Edgeflow