2D Texture Synthesis

1
2D Texture Synthesis

• Instructor Yizhou Yu

2
Texture synthesis

• Goal increase texture resolution yet keep local
  texture variation

3
Synthesis by global statistics

• Idea Obtain statistics of input image, match
  with output image
• Histograms normalized graph of intensity
  frequencies

4
Synthesis by global statistics

• Consider extreme case
• Histogram matching can generate two images

5
Using higher-order global stats

• Histogram matching does not take into account
  spatial information.
• Need additional info
• For every pixel, examine local neighborhood
• Gives local but not global features (e.g.
  veins)
• Use higher order stats
• Correlation
• average of the product of the intensities
• tells how interdependent pixel values are

6
General procedure

• 1.) Define error metric
• What do you want to match?
• 2.) Match statistics
• Histogram matching
• Get the density value of a pixel in the output
  image. Map this density to a pixel intensity in
  the input image ? Overwrite the pixel in the
  output image with the mapped pixel intensity in
  the input image

7
Pyramid-based Texture Analysis/Synthesis

• Paper by David J. Heeger and James R. Bergen from
  SIGGRAPH 1995
• http//www.cns.nyu.edu/david/ftp/reprints/heeger-
  siggraph95.pdf

8
Pyramid-based Synthesis

• Downsample image several times and keep track of
  only differences in a feature image
• To reconstruct, upsample small images and add
  differences
• Pixels in feature images are close to zero
• Provides multiple scales with each feature image
  representing different feature sizes

9
Laplacian Pyramid

• 1-D filter (1/16) (1 4 6 4 1)
• To generalize to 2D, use tensor product
• Normalize by dividing by 256

10
Upsampling

• To upsample, copy pixels to every other pixel in
  larger image
• Fill the rest with zeroes, run low pass filter to
  fill in values

11
Oriented Filters

• Gaussian filters are symmetric, thus edges and
  contours are not detected
• Use multiple oriented filters to catch
  non-symmetric features

Left top series Oriented filters Right image
Texture Left bottom series Filtered textures
12
Pyramids with Oriented Filters

• Each oriented filter creates its own feature
  image
• Thus, for each downsampled image, keep one image
  for each oriented filter
• Each oriented filter captures one orientation of
  lines

13
Matching Image Features

• Input parameters
• noise initial noisy texture
• texture texture to be matched
• Output texture is stored in noise

14
Matching Image Features

• Helper functions
• MatchHistogram(noise, texture)
• Matches histogram, using method described last
  time
• MakePyramid(texture)
• Create pyramid images (base and feature images)
• CollapsePyramid(pyramid)
• Constructs high resolution image from base and
  feature images

15
Matching Image Features

• MatchTexture(noise, texture)
• MatchHistogram(noise, texture) //
  first-order matching
• analysis_pyr MakePyramid(texture) // create
  pyramid from input texture
• for several iterations
• synthesis_pyr MakePyramid(noise) // create
  pyramid from noise
• for each feature image, fi of analysis_pyr
• for each feature image, fj of synthesis_pyr of
  same orientation
• MatchHistogram(fi, fj)
• end for
• end for
• noise CollapsePyramid(synthesis_pyr)
• MatchHistogram(noise, texture)
• end for

16
Matching Image Features

• MatchTexture matches histograms of feature
  images, not pixel values, thus providing a much
  better matching
• Feature images already consider local
  neighborhoods, resulting in better approximations
• Good for randomized textures
• Textures with large scale features or thin and
  long features not matched well
• e.g. stripes in wood, lines in coral

17
Texture Synthesis Using Local Neighborhoods

• Main goal is to keep local spatial coherence, but
  not global stats.
• Randomized method
• Pick pixel and copy it along with its
  neighborhood to random parts in synthesized image
• If two neighborhoods overlap, just blend
• This can result in features getting cut off if
  larger than local neighborhood

18
Neighborhood-Based Texture Synthesis

• Patch-Based
• Patch-based sampling achieves real-time speed.
  Liang et. al. 2001
• Image quilting high-quality results and simple
  implementation Efros Freeman 2001
• Graph cut provides a powerful and refinable
  scheme. Kwantra et. al. 2003
• Pixel-Based
• Efros Leung 99, Wei Levoy 2000,
  Ashikhmin 2001, Hertzmann et. al. 2001,
  Zhang et. al. 2003

19
Pixel-wise Synthesis

• Grow pixel by pixel
• Start from an existing patch of the texture (as
  opposed to noise texture like Pyramid-based
  Synthesis)
• Look for regions in input texture most similar to
  current region in new texture
• Copy pixels next to best-match region to expand
  new texture

20
Following Raster Order

• Blue region already set by algorithm
• Green region compare this region to input image
• Yellow region closest match
• Red region replace these pixels with magenta
  region to maintain local integrity

21
Hierarchical Synthesis

• Build a multi-resolution pyramid for the example
  texture
• Generate a synthesized pyramid for the output
  texture
• At each level, follow the raster order

22
Randomizing Synthesis

• Instead of picking pixels from closest matching
  region, use threshold to pick a few candidate
  matches
• Randomly pick one of the candidates
• Refer to Texture Synthesis by Non-parametric
  Sampling
• http//www.cs.berkeley.edu/efros/research/synthes
  is.html

23
Results
24
Patch-Based Synthesis

• Search in a sample texture for neighborhoods most
  similar to a context region
• Merge a patch with the partially synthesized
  output texture

25
The Seam Problem

• Feature discontinuities may appear in patch-based
  synthesis.

26
The Second Reason

• Didnt find the smoothest transition between the
  incoming patch and context region.
• Solutions use dynamic programming Efros and
  Freeman 2001 or graph cut Kwatra et al. 2003
  to find an optimal cut.

Original
Warped
27
The First Reason

• Rigid template matching (SSD) often employed in
  the first step does NOT consider geometric
  similarity

Original
Warped
28
Feature Map Guided Texture Synthesis

• Basic Steps Wu and Yu 2004
• Maintain an input and output feature map
• Match and align curvilinear features
• Integrate feature maps into template-based
  texture synthesis

29
Comparisons
Sample
Feature Map
Graphcut
Quilting
Texton Mask
30
Acceleration Schemes

• Fourier Transform
• Acceleration techniques for nearest-neighbor
  search
• Tree-structured vector quantization
• Kd-trees
• Minimizing the candidate set
• Coherent synthesis, Ashikhmin 2001
• Precomputing candidate set
• K-coherent search, Tong et al. 2002
• Jump Map, Zelinka and Garland 2002
• GPU with parallel synthesis
• Lefebvre and Hoppe 2005

Original
Warped