Feature Extraction

1
Feature Extraction
CSC 59866CD Fall 2004

• Lecture 9
• Image Segmentation

Zhigang Zhu, NAC 8/203A http//www-cs.engr.ccny.cu
ny.edu/zhu/ Capstone2004/Capstone_Sequence2004.ht
ml
2
Finding Circles

• If we dont know r, accumulator array is
  3-dimensional
• If edge directions are known, computational
  complexity if reduced
• Suppose there is a known error limit on the edge
  direction (say /- 10o) - how does this affect
  the search?
• Hough can be extended in many ways.see, for
  example
• Ballard, D. H. Generalizing the Hough Transform
  to Detect Arbitrary Shapes, Pattern Recognition
  13111-122, 1981.
• Illingworth, J. and J. Kittler, Survey of the
  Hough Transform, Computer Vision, Graphics, and
  Image Processing, 44(1)87-116, 1988

3
Region Segmentation

• Partitioning of an image into different regions
  (connected components), each having uniform
  properties in some (set of) image feature(s)
• gray value
• color value(s)
• textural qualities
• local gradient
• motion
• shape info
• ..... etc.

4
Goal of Segmentation

• Segment a scene into image elements which may
  correspond to meaningful scene elements

5
Primary Goal of Segmentation

• Segmenting an image into image elements which
  may correspond to meaningful scene elements
• What sort of image elements may correspond to
  meaningful scene elements?
• Answer depends on type and complexity of images
• Less constrained scenes must be segmented more
  conservatively.
• Segmentation is not a well defined problem.

6
Region Segmentation Example
7
Color Image Segmentation
?

• Given a grayscale image, how do we generate a
  region segmentation?
• In general, regions can be formed from the
  original image data or from 'derived' images
• - color images from R, G, B
• - textural images
• - displacement images from motion analyses
• - 3D depth images

8
Problems with Segmentation
In general, high level contextual knowledge is
required for successful segmentation
9
Formal Definition of Regions

• A region segmentation of an image, I, is a
  partition of the set of pixels of I into a set
  of K regions Rj, 1jK, such that

K
Ç
1. I
Rj
Every pixel belongs to a region
i1
No pixel belongs to more than one region
2. Ri Ç Rj Æ for i ? j
3. p connected to p for all p, p in Rj
Spatial coherence
Feature coherence
4. For some predicate P
P(Ri) is TRUE for I 1,2,,K P(Ri Rj) is
FALSE for Ri, Rj adjacent and i?j
Ç
10
Representing Regions

• Region Occupancy Map
• A set of region labels in registration with image
  I specifying the region association for each pixel

Image
Occupancy Map or Label Plane
11
Contour Representation
Image
12
Chain Code

• The chain code representation of a boundary is
  found by 'walking' counterclockwise around the
  boundary and recording the direction to turn to
  stay on the border

Direction Code
13
Chain Code
CC (i,j) 5 5 6 6 6 0 0 0 0 0 0 0 1 1 2 2 2 4 4
5 4 3 4 4
14
Region Segmentation

• Basic Approaches
• Generalized thresholding
• Region growing
• Region merging
• Region splitting
• Split and Merge
• Extensions to split and merge
• K-means clustering
• Watershed algorithms

Partitioning methods
Grouping methods
15
Partitioning Methods

• Partitioning
• Given a large data set.
• Goal carve it up according to some notion of the
  association between items inside the set.
• We would like to decompose it into pieces that
  are good according to our model.
• For example, we might
• decompose an image into regions which have
  coherent color and/or texture inside them
• take a video sequence and decompose it into shots
  segments of video showing about the same stuff
  from about the same view point
• decompose a video sequence into motion blobs,
  consisting of regions thatwhave coherent color,
  texture and motion.

16
Grouping Methods

• Grouping
• Given a set of distinct data items
• Goal we wish to collect together sets of data
  items that look similar according to our model.
  
• Effects like occlusion mean that image components
  that belong to the same object are often
  separated.
• Examples of grouping include
• collecting together tokens that, taken together,
  form an interesting object
• collecting together tokens that seem to be moving
  together.
• Collecting together regions that have similar
  color and/or texture

17
Region Segmentation

• Two Basic Techniques
• Region Merging
• START with many trivial regions (each pixel?)
• MERGE regions into larger regions based on some
  similarity criteria
• CONTINUE merging until no further merging is
  possible
• Region Splitting
• START with a single large region (entire image?)
• SPLIT into several smaller regions based on a
  'splitting' criterion
• CONTINUE until no further splitting is possible
  (regions are 'uniform')

18
Region Segmentation

• Basic Approaches
• Generalized thresholding
• Region growing
• Region merging
• Region splitting
• Split and Merge
• Extensions to split and merge
• K-means clustering
• Watershed algorithms

19
Generalized Thresholding

• RLP(i,j) k if Tk-1 lt I(i,h) lt Tk k 1,...,m
• Tk are thresholds.
• m is the number of distinct thresholds.
• RLP (Region Label Plane) may contain
  significantly more than m 'regions', hence
  connected components must be found and the
  regions relabeled with distinct label numbers.
• The thresholds Tk may depend on
• the entire image I(i,j), GLOBAL
  THRESHOLDS
• N(i,j) (local neighborhood), LOCAL THRESHOLDS or
• I(i,j) and N(i,j).
  DYNAMIC THRESHOLDS
• To apply thresholding, must determine
• m - the number of thresholds, and,
• Tk - the threshold values.

20
Threshold Selection

• Manual try one and see if it looks good
• Histogram analysis
• Strategies
• Search for minimum between P1 and P2
• (search for minima between several peaks -
  multi-thresholding)
• Fit second order equation
• Differentiate to find minimum
• Smooth image and/or histogram first
• Histogram only points not on edges in histogram
• Or unweight contribution of pixels having high
  gradient magnitude

21
Threshold Selection

• Gaussian fitting
• Intensity distribution for objects assumed to be
  normally distributed
• Minimize false positives/false negatives
• Match properties of binary and gray level image
• e.g. moments
• Choose threshold to maximize/minimize some
  function of the total edge gradient
• Maximize property of image or histogram
• e.g. entropy
• and probably lots of others

22
Optimal Thresholding

• Approximate the histogram using a weighted sum of
  two or more probability densities with normal
  distribution
• Threshold set at the closest gray level
  corresponding to the minimum probability between
  the maxima of two or more normal distributions.
• Results in minimum error segmentation
• Need to estimate parameters of the density
  functions
• Implies optimization

Probability distributions of backgrounds and
object
Corresponding histograms and optimal thresholds
23
Local or Adaptive Thresholding

• Intensity values can vary as a function of
  lighting (for example).e.g.
• Very difficult to threshold using conventional
  methods.

24
Local or Adaptive Thresholding

• Each pixel in image needs a unique threshold
• Two basic approaches
• Chow and Kaneka Adaptive Thresholding (1972)
• Compute thresholds in a local window at sampled
  locations using histogram technique
• Interpolate local thresholds across image
• Local Thresholding
• Examine statistically pixel values in local
  neighborhood around pixel to be thresholding
• Use local statistic as threshold
• Possibilities include mean, median, or mean of
  max and min value

Mean of 7x7 neighborhood
25
Improved Results

• Results can be improved if the threshold employed
  is not the mean, but (mean-C), where C is a
  constant.
• Using this statistic, all pixels which exist in a
  uniform neighborhood (e.g. along the margins) are
  set to background.

26
Local Histogram Thresholding
27
Region Segmentation

• Basic Approaches
• Generalized thresholding
• Region growing
• Region merging
• Region splitting
• Split and Merge
• Extensions to split and merge
• K-means clustering
• Watershed algorithms

28
Region Growing

• Goal Segment the image my repeatedly starting
  from a particular pixel, called a "seed" point,
  growing it into a region by iteratively adding
  neighboring points while some similarity
  criterion is met.
• Is a set of algorithms to group pixels with
  similar attributes together
• GENERAL IDEA
• A pixel is added to a partially grown region if
  two conditions are satisfied
• The pixel must be adjacent to the region, and,
• The pixel must be "similar enough" to pixels
  already in the region.
• The process continues until no further points can
  be added.
• Some other seed point, not already in any region,
  is chosen and the process is repeated until the
  entire image is segmented.

29
Region Growing

• Use a region label plane RLP
• same size as the original image
• contains a 1 if the corresponding image point is
  in the region, 0 otherwise
• initially contains a 1 corresponding to the seed
  point

Image
Region Label Plane
30
Similarity Criteria

• When is a pixel "similar enough" to be added to
  the region?
• different choices
• statistical population tests
• simple feature differences
• General Idea
• Apply iteratively to all pixels until no more
  pixels are added.

IF RLP(i,j) 0 AND RLP(i-n, j-m) 1 for some
n,m -1 to 1 AND I(i-n, j-m) - I(i,j) lt T (a
predefined threshold) THEN RMP(i,j) 1
31
Example Simple Region Growing
Seed Point
T16
T32
T64
T128
32
Problems

• Whats the similarity criterion?
• How do I select the threshold?
• Why does the algorithm leak?
• Because of leaks, features in a region can vary
  arbitrarily.
• Fixed thresholds do not take into account
  characteristics of global spatial distribution.

pixel sequence p1,..., pk pj,
pj1neighbors pj - pj1 lt T but p1 - pk
gt T
pk
p1
33
Region Segmentation

• Basic Approaches
• Generalized thresholding
• Region growing
• Region merging
• Region splitting
• Split and Merge
• Extensions to split and merge
• K-means clustering
• Watershed algorithms

34
Region Merging

• Define a distance function between regions.
• general form
• dij D(Ri, Rj) gt 0
• Typically D is a distance measure in feature
  space and is a function only of the feature
  vectors associated with regions Ri and Rj
• dij D(fi, fj) gt 0
• Merge regions with minimum distance
• Need to define some kind of termination criteria

35
Example Distance Measures

• Distance between color means
• Distance between region centers
• Weighted combination of the two
• Where Nm is the number of pixels in Rm

Ni
Nj
2
2
mi - mnew
mj - mnew

cij
Nnew
Nnew
dij a cij b rij
36
Region Merging

• Define a distance measure dij D(f(Ri), f(Rj))
  gt0
• Algorithm
• While termination condition false
• Determine the minimum distance
  regions
• i, j arg min dij
• Merge the minimum distance regions
• Ri Ri U Rj
• Remove merged region from region list
  L L-Rj
• Compute termination condition

Ri, Rj Î L
i,j
37
Merging Hierarchy

• Algorithm generates a binary tree
• Merging can be terminated when the minimum
  distance exceeds a threshold
• d i,j gt T stop
  merging
• Different thresholds produce different
  segmentations

38
Fishers Criterion

• Mean and variance are good features to use for
  merging
• Especially if data is distributed normally
• e.g. is modeled by the Gaussian distribution
• Peak occurs at m (the mean) and has a value a
• a is (1 / 2ps2) and ensures that the area
  under the curve 1
• For modeling histograms
• Compute area under the histogram and divide by
  (2p n)
• Parameter n is called the variance
• s sqrt (n) is the standard deviation
• n measures the flatness of the distribution

1/2
1/2
39
Histogram Model
40
Fishers Criterion

• Discrimination between regions of different means
  and standard deviations can be done using
  Fischers criterion
• l is a threshold
• If two regions have good separation in the means
  and low variance, then we can separate them.

41
Uniformity

• Thus, the merging threshold for the mean
  intensity for two adjacent regions, l, should
  vary depending on the expected uniformity of the
  merged region
• Less uniform regions will require a lower
  threshold to prevent under merging Uniformity a
  function of both intensity mean and variance of
  the region
• Combine them (heuristic) as
• Uniformity 1 - n / m
• In range 0-1 for case where the samples are all
  positive
• The threshold value l decreases with the decrease
  in uniformity as
• l (1 - n / m ) l0
• User need supply only one threshold l0

2
2
42
Region Segmentation

• Basic Approaches
• Generalized thresholding
• Region growing
• Region merging
• Region splitting
• Split and Merge
• Extensions to split and merge
• K-means clustering
• Watershed algorithms

43
Region Splitting

• Region Splitting
• (1) Start with a single large region (initially,
  entire image).
• (2) Recursively split it into smaller regions.
• (3) Continue splitting until each region is
  uniform (no further splits are possible).
• A simple approach Global Thresholding
• Define a global threshold T
• Apply to every pixel in the image I
• RLP(i,j) 0 if I(i,j) lt T
• RLP(i,j) 1 if I(i,j) T

44
Region splitting Multiple Features

• Old algorithm Ohlander-Price
• Basically a sequential histogram-based multiple
  threshold algorithm.
• Features used RGB, HSI, and YIQ (9 images)
• General Idea
• Start with the entire image as the initial
  region.
• Get the next region to be segmented
• if none - the segmentation is complete.
• Compute the set of one-dimensional histograms.
• Select the "best" peak and find valleys on either
  side
• if none, the region is "done" - put on finished
  list.
• Apply the threshold to the region and determine
  connected components.
• Add these regions to the list of regions to be
  further segmented and go to step 1.

45
Ohlander-Price Data
46
Ohlander-Price Result
47
Region Splitting
48
Peak Selection Criteria

• 0 Intensity peak in 0-60 or 200-255 ranges
• Best is closest to end
• 1 Both minima lt 10 highest value
• max/min ratio gt 4
• Another peak exists with
  max/min ratio gt 2
• 2 Both minima lt 25 of the peak value
• max/min ratio gt 2
• Another peak with max/min
  ratio gt 2
• 3 Max/min ratio gt 2
• Another peak with max/min
  ratio gt 2
• If maxima are within 10, then
  both are acceptable
• (a bimodal
  distribution)
• 4 (Saturation only)
• Minima in 0-200 (lowest 20)lt
  25 of the peak value
• max/min ratio gt 2
• Specified minima must
  separate peak with max/min ratio gt 1.2
• 5 Minima lt 10 of highest value
• 10 of all points must be
  outside the peak
• 6 Minima lt 70 of highest value

49
Region Segmentation

• Basic Approaches
• Generalized thresholding
• Region growing
• Region merging
• Region splitting
• Split and Merge
• Extensions to split and merge
• K-means clustering
• Watershed algorithms

50
Hybrid Technique

• Split and Merge
• combination splits followed by merges (or
  vice-versa)
• split and merge decisions can be either
• local a pixel and its immediate neighbors
• a region and its immediate
  neighbors
• global on the basis of a large number of
• pixels scattered throughout the
  image

51
Split and Merge

• General idea
• Begin with an arbitrary region decomposition in a
  quadtree plane
• Initial decomposition entire image?
• Split each region which violates a uniformity
  predicate into its 4 quadtree children
• Merge (recursively) all regions which jointly
  satisfy a uniformity criterion
• Supporting data structure region adjacency graph

52
Split and Merge Example
53
Region Segmentation

• Basic Approaches
• Generalized thresholding
• Region growing
• Region merging
• Region splitting
• Split and Merge
• Extensions to split and merge
• K-means clustering
• Watershed algorithms

54
Segmentation by k-means clustering

• Assume we know the data has k clusters (k known)
• Each cluster is assumed to have a center ci
• The jth element to be cluster is described by a
  feature vector xj

• For scattered points, x would be coordinate(s)
• For an intensity image, x might be the intensity
  at a pixel
• Define an objective function that measures how
  good the clustering result is.
• Develop an algorithm to maximize the objective
  function.

55
Objective Function

• Assume that element are close to the center of
  their cluster
• One possible objective function is
• Note that if allocation of points to clusters is
  known, we can compute the best center easily
• Far too many associations of points to clusters
  to search this space for a minimum
• Instead, define an algorithm that alternates
• Assume centers are known, allocate points
• Assume allocation is known, compute centers

56
k-means algorithm

• Choose k data points to act as cluster centers
• Random selection
• First k data points
• Until the cluster centers are unchanged
• Allocate each data point to cluster whose center
  is nearest
• Now ensure that every cluster has at least one
  data point possible techniques for doing this
  include
• supplying empty clusters with a point chosen at
  random from points far from their cluster center.
• Replace the cluster centers with the mean of the
  elements in their clusters.
• end
• Apply connected components algorithm to generate
  regions

57
k-means Clustering
58
k-means Clustering
59
k-means Clustering
60
k-means Clustering
61
k-means Clustering
62
k-means Clustering
63
k-means Clustering
64
k-means Clustering
65
Example

• Assume k5
• Each pixel is represented by the mean value of
  the cluster to which the pixel belongs
• A connected components algorithm must be applied
  to make these true region segementations

K5
Original Image
k-means on intensity
k-means on color
66
Results, k11

• Sample clusters with k-means clustering based on
  color

67
Other Distance Measures

• Suppose we want to have compact regions
• New feature space 5D
• (2 spatial coordinates, 3 color components)
• Points close in this space are close both in
  color and in actual proximity
• Problem with simple Euclidean distancewhat if
  coordinates range from 0-1000 but colors only
  range from 0-255?
• Depending on how things are scaled, gives
  different weight to different kinds of data
• Weighted Euclidean distance adjust weights to
  emphasize different dimensions

S
68
Hybrid Edge-Region Approaches

• Same idea as region mergingstart with
  oversegmented image.
• Use edge detection information as well.
• Merge not based on region statistics but on weak
  boundaries.
• Usually use fraction of edge pixels along shared
  boundary that are below some threshold.
• Often used as post-processing for edge-based
  segmentation.

69
Region Segmentation

• Basic Approaches
• Generalized thresholding
• Region growing
• Region merging
• Region splitting
• Split and Merge
• Extensions to split and merge
• K-means clustering
• Watershed algorithms

70
Watershed Algorithms

• A gray scale image can be viewed as a topographic
  relief map where the intensity function of the
  image represents the altitude.
• A watershed region or catchment basin is defined
  as the region over which all points flow
  downhill to a common point.
• Points at which water would be equally likely to
  fall to more than one such minimum watersheds or
  watershed lines
• Watersheds of gradient magnitude make useful
  region-based segmentation primitives.
• Boundaries of watersheds are one way to define
  ridges
• this is basically the
• same idea as finding
• a ridge of gradient
• magnitude.

71
Watersheds
72
Example on GrayScale Image
73
Immersion Algorithm

• Start with all pixels with the lowest possible
  value (grad. mag.)
• these form the basis for initial watersheds
• For each intensity level k
• For each group of pixels of intensity k
• If adjacent to exactly one existing region, add
  these pixels to that region
• Else if adjacent to more than one existing
  regions, mark as boundary
• Else start a new region
• Can use a histogram-like structure to keep lists
  of all pixels with each intensity level k.

74
Watershed Problems

• Oversegmentation
• watershed from markers
• Computationally intensive computation intensive
• graph algorithm and appropriate data structures
  graph
• Graylevel might not be the optimal choice as the
  local similarity measure similarity measure
• other local features
• Statistical
• edge enhanced image
• distance transformed imagee transformed image)

75
Over-segmentation Problem

• Oversegmentation due to noise and other local
  irregularities of the gradient
• Solution markers (of object locations)

76
Tobogganing

• Again work on the gradient magnitude image.
• Link each pixel to the smallest of its neighbors.
• If no smaller neighbors, become a are region
  seed.
• All pixels that flow downhill (smallest
  neighbor) to the same point form a single region.

77
Multispectral Segmentation

• Given N multi-spectral images
• 1. Independently segment each multi-spectral
  feature image using the localized histogram
  algorithm.
• 2. Intersect the N segmentation to create a new,
  combined segmentation.
• 3. Merge using a region merging algorithm.
• See J. Beveridge, J. Griffith, R. Kohler, A.
  Hanson, and E. Riseman, Segmenting Images Using
  Localized Histograms and Region Merging, IJCV
  2(3), January 1989, pp. 311-347

78
Color Image
Intensity
Red
Green
Blue
79
Segmentations
Intensity
Red
Green
Blue
80
Three Color Union
Note This result was obtained by taking the
union of three color segmentations obtained with
a more sensitive set of parameters than those
shown on the previous page and consequently has
more boundaries.
81
Final Segmentation After Merging
Segmentation obtained from over-segmented image
on previous slide after applying a rule-based
merging strategy using regions similarity, size,
and connectivity.
82
House Results
Three Color Union
Intensity Only
83
Region Features

• Many features can be use to characterize a region
  and its properties
• Supports many tasks, including object
  recognition.
• Example features
• area, height, and width
• perimeter, bounding box, area of bounding box
• centroid
• orientation
• compactness
• moments
• .....etc.

84
Region Features

• Basics
• Perimeter
• Area
• Orientation
• Rotation/translation/scale invariant
• Compactness perimeter2/area
• Rectangularity AreaRect/AreaObject (next slide)
• Euler number regions - holes
• Convexity AreaConvexHull/AreaObject

85
Rectangularity

• W and L are a function of q
• Rectangularity WxL/A
• Choose q to get WxL minimum
• Called the minimum bounding rectangle
• Minimized for rectangular objects

86
Perimeter
boundary
Region plane
or
87
Moments

• Centroid center of mass
• Higher order moments
• Note A m00 r m10 / m00 c m01 / m00

88
Central Moments

• Moments around the center of mass
• Note that A u00 u01 0 u10 0
• Higher order (2nd order and higher) moments are
  used heavily in oject recognition

89
Orientation

• Let q be the region orientation with respect to
  the r axis
• Can be shown that

90
Summary

• Covered only basic approaches
• simple region growing
• split, merge, and split and merge algorithms
• generalized thresholds
• .
• Many approaches to region segmentation
• statistical techniques parameter estimation,
  mode estimation, clustering, decision theoretic
  methods...
• surface fits to the intensity surface constant,
  plane, or bivariate polynomial,...
• relaxation traditional, stochastic (simulated
  annealing)
• markov random fields methods
• context sensitive and knowledge-driven methods
• combinations with edge detection techniques
• optimization and learning methods
• multi-resolution
• Literature is enormous

91
Next Topic

• Next Camera Model