Title: Texture Synthesis
1TextureSynthesis
- Aude Oliva
- CVC Seminar
- Fall 2008
2Papers
- Heeger Bergen 1995
- Portilla Simoncelli, 2001
3What is a texture?
- Definition 1 A spatially homogeneous pattern
that typically contains repeated structures,
often with some random variations. - Definition 2 a set of elements at periodic or
random locations within an image - Definition 3 a visual pattern which has a
stationary distribution of features the
texture elements (e.g. textels) is repeated
everywhere in the image.
4What is a texture?
- Two categories of textures
- 1) determinist or regular textures determined
by a set of primitives and a placement rule (e.g.
a tile floor). Those are determined by repeated
elements or groups of elements. - 2) stochastic textures does not have easily
identifiable primitives (e.g. granite, sand). - Natural textures are characterized by both
properties
5Why is Texture important ?
- All surfaces have texture
- In real world, texture imposes the structure and
give form to separate regions. - Texture is implicated in the process of
segregating the visual scene into discrete
regions (e.g. separating the sidewalk from the
grass, a low level analysis) and make basic
decision about the nature of the regions (high
level analysis) - Affordance when walking across uneven outdoor
terrain identify surface that affords good
footing (the tilt of some surface indicates that
some surfaces should not be stepped on)
Camouflage property An animal (e.g. snake) with
a texture similar to those of surrounding object
to provide camouflage. Camouflage is the process
of effectively imitating texture (cf. texture
synthesis).
6Why is Texture important ? Texture Gradient
- Texture gradient (Gibson, 1950) In addition to
providing information about depth, texture
gradient also inform observers about the
orientation of a surface in depth and about its
curvature. - Complex surface shape can be realistically
depicted by textural variations
- Natural texture gradients many natural surfaces
are defined by texture elements of about the same
size and shape so that surface depth and
orientation can be recovered from systematic
changes in their projected sizes and shapes
(Gibson, 1950) - Artificial texture gradients. Artificial surfaces
of arbitrarily complex shapes can be rendered by
using identical texture elements in computer
graphics displays (Marr, 1982)
7Texture gradient
8Texture synthesis
- Camouflage property An animal (e.g. snake) with
a texture similar to those of surrounding object
to provide camouflage. Camouflage is the process
of effectively imitating texture (cf. texture
synthesis). - How can we imitate a texture?
9Representing textures
- What filters?
- experience suggests spots and oriented bars at a
variety of different scales - What statistics?
- within reason, the more the merrier.
- At least, mean and standard deviation
- better, various conditional histograms.
- Textures are made up of quite stylized
subelements, repeated in meaningful ways - Representation
- find the sub-elements (textons, textels, etc),
and represent their statistics - But what are the sub-elements, and how do we find
them? - Find sub-elements by applying filters, looking at
the magnitude of the response
10Questions about Textures
- (1) How can we represent the information
contained in a texture ? - (2) How can we synthesis, imitate, replicate, a
texture so that it looks like the original ?
(Notion of camouflage) - (3) Which texture information can be used by
human observers (and a model) to classify
together similar textures ?
11Texture Synthesis Approach
- Philosophy To capture samples of the real world
as images and use them to synthesize novel views
rather than recreating the entire physical world
from scratch. - Image-based texture synthesis algorithms such
an algorithm should be able to take a sample of
texture and generate an unlimited amount of image
data which, while not exactly like the original,
will be perceived by humans to be the same
texture. - The origin D. J. Heeger and J. R. Bergen (1995).
Pyramid Based Texture Analysis/Synthesis.,
Computer Graphics Proceedings. p. 229-238. - The approach is based on a model of human
texture perception, and has potential to be a
practically useful tool for graphics applications
12Which textures are we going to talk about?
Stationary Stochastic
13When are two textures similar?
All these images are different instances of the
same texture We can differentiate between them,
but they seem generated by the same process
14The trivial texture synthesis algorithm
?
?
15Texture synthesis and representation
?
Set of equivalent textures
?
?
Space of all images
Set of equivalent textures generated by exactly
the same physical process
16Texture synthesis and representation
?
Set of equivalent textures
?
Set of perceptually equivalent textures
?
Space of all images
Set of equivalent textures generated by exactly
the same physical process
Set of perceptually equivalent textures well,
they just look the same to me
17The main idea it works by kind of projecting a
random image into the set of equivalent textures
Set of equivalent textures
Set of perceptually equivalent textures
Space of all images
18Overview of the algorithm
Two main tools 1- steerable pyramid 2- matching
histograms
19Steerable Pyramid
Freeman Adelson, 91 Freeman Simoncelli, 95
201-The steerable pyramid
Decomposition
subband
Simoncelli, Freeman, Adelson
211-The steerable pyramid
Decomposition
Reconstruction
Low-pass residual
221-The steerable pyramid
Representation in pixel space
Alternative representation
Low-pass residual
231-The steerable pyramid
But why do I want to represent images like this?
241-The steerable pyramid
Argument used by Hegger Bergen Statistical
measures in the subband representation seem to
provide a distance between textures that
correlates with human perception better than
pixel-based representations.
251-The steerable pyramid
- In general seems a good idea to have a
representation that - Preserves all image information (we can go back
to the image) - Provides more independent channels of information
than pixel values (we can mess with each band
independently) - But all this is just indirectly related to the
texture synthesis task. But let assume is good
enough
261-The steerable pyramid
271-The steerable pyramid
Steerable pyramid
Input texture
28Overview of the algorithm
Two main tools 1- steerable pyramid 2- matching
histograms
292-Matching histograms
There are many ways of changing the histograms of
two signals.
302-Matching histograms
75 of pixels have an intensity value smaller
than 0.5
9 of pixels have an intensity value within the
range0.37, 0.41
5 of pixels have an intensity value within the
range0.37, 0.41
312-Matching histograms
Z(x,y)
We look for a transformation of the image Y Y
f (Y) Such that Hist(Y) Hist(f(Z))
?
Y(x,y)
322-Matching histograms
The function f is just a look up table it says,
change all the pixels of value Y into a value
f(Y).
Y f (Y)
Y(x,y)
Y 0.8
Y 0.5
New intensity
Original intensity
332-Matching histograms
Y f (Y)
34Another example Matching histograms
10 of pixels are black and 90 are white
?
?
5 of pixels have an intensity value within the
range0.37, 0.41
35Another example Matching histograms
The function f is just a look up table it says,
change all the pixels of value Y into a value
f(Y).
Y f (Y)
Y(x,y)
Y 0.8
Y 1
Original intensity
New intensity
36Another example Matching histograms
Y f (Y)
In this example, f is a step function.
37Matching histograms of a subband
38Matching histograms of a subband
Y f (Y)
39Texture analysis
(histogram)
Wavelet decomposition (steerable pyr)
Input texture
(histogram)
(Steerable pyr Freeman Adelson, 91)
The texture is represented as a collection of
marginal histograms.
40Texture synthesis
Heeger and Bergen, 1995
Wavelet decomposition
(histogram)
Input texture
(histogram)
iterate
41Why does it work? (sort of)
42Why does it work? (sort of)
The black and white blocks appear by
thresholding (f) a blobby image
Iteration 0
43Why does it work? (sort of)
The black and white blocks appear by
thresholding (f) a blobby image
44Why does it work? (sort of)
After 6 iterations
Histograms match ok
red target histogram, blue current iteration
45Color textures
R
G
Three textures
B
46Color textures
R
G
B
47Color textures
R
This does not work
G
B
48Color textures
Problem we create new colors not present in the
original image. Why? Color channels are not
independent.
B
G
R
49PCA and decorrelation
G
B
R
In the original image, R and G are correlated,
but, after synthesis,
G
G
R
R
50PCA and decorrelation
The texture synthesis algorithm assumes that the
channels are independent. What we want to do is
some rotation
G
U2
Rotation
U1
R
See that in this rotated space, if I specify one
coordinate the other remains unconstrained.
51PCA and decorrelation
G
correlation(R,G)
1.0000 0.9303 0.6034 0.9303 0.9438
0.6620 0.6034 0.6620 0.5569
C
R
PCA finds the principal directions of variation
of the data. It gives a decomposition of the
covariance matrix as
0.6347 0.6072 0.4779 0.6306 -0.0496
-0.7745 0.4466 -0.7930 0.4144
C D D
D
By transforming the original data (RGB) using D
we get
R G B
U1 U2 U3
D
3 x Npixels
3 x Npixels
3 x 3
The new components (U1,U2,U3) are decorrelated.
52Color textures
R
G
Rotation Matrix (3x3)
D
B
These three textures Look less similar (lower
dependency)
These three textures look similar (high
dependency)
53Color textures
R
Inverse Rotation Matrix
G
D
B
54Color textures
55Color channels
56Color channels
57Color channels
58Examples from the paper
Heeger and Bergen, 1995
59Examples from the paper
60Examples not from the paper
Input texture
Synthetic texture
But, does it really work even when it seems to
work?
61But, does it really work???How to measure how
well the representation constraints the set of
equivalent textures?
All the textures in this set have the same
parameters.
?
?
?
?
?
62How to identify the set of equivalent textures?
?
This does not reveal how poor the representation
actually is.
63We need a space that is more perceptual
In a perceptual space all these noise images are
very close. But in pixel space, they are very
far away.
How big is this set in a pixels space?
64We need a space that is more perceptual
In a perceptual space all these noise images are
very close. But in pixel space, they are very
far away.
How big is this set in a perceptual space?
How big is this set in a pixels space?
65How to identify the set of equivalent textures?
These trajectories are more perceptually salient
This set is huge
66How to identify the set of equivalent textures?
67Portilla Simoncelli
Heeger Bergen
Portilla Simoncelli
68Portilla Simoncelli Texture Analysis
- Representation and synthesis of visual textures
- Texture Representation
- Crudely stated, our goal is to establish a
minimal set of statistical measurements such that
two textures are identical in appearance if and
only if they agree on these measurements. We
first decompose an example texture image using a
multi-scale oriented linear basis (specifically,
a steerable pyramid). - Texture Synthesis
- Starting with an image of Gaussian white noise,
we alternate between 1) constructing the
steerable pyramid and forcing the sample
statistics of each subband to match those of a
reference texture image, 2) reconstructing an
image from the pyramid, and forcing the sample
statistics of the resulting pixels to match those
of the reference image.
http//www.cns.nyu.edu/lcv/texture/
69(No Transcript)
70Texture analysis and synthesis
Original
Marginal Histograms (Heeger-Bergen)
Higher order statistics
Portilla Simoncelli, 00
71Four statistics
72(1) Marginal Statistics
73(No Transcript)
74(2) Coefficient correlation
It captures periodic or globally oriented
structure (within a neibhorhood size, e.g. 9
pixels). The local correlation of each subband.
It characterizes the salient spatial frequencies
and the regularity of the texture, as represented
by periodic or globally oriented structure
75(3) Magnitude correlation
Capture structure (edges, bars, corners) and
second-order textures. cross-correlation of
each subband magnitudes with those of other
orientations at the same scale, and
cross-correlation of each subband magnitude with
all orientations at a coarser scale.
In the square texture large magnitude are at the
same location. Large magnitude of the two
diagonal orientations are anti-correlated in the
herringbone texture.
76(3) Magnitude correlation
77(3) Magnitude correlation
78(4) Cross-scale phase statistics
Cross-scale phase statistics Distinguishes edges
from lines. Help represented gradients/lighting
effects. A local representation of the phase
(position), in order to represent edges and
lines. Important to represent 3dimensional aspect
and shadows, and more generally gradients due to
lighting effects.
79Types of Texture classes
- Artificial-Non periodic
- (jrotpluses.jpg)
- Artificial-Periodic
- (checkerboard.jpg)
- Photographic/structured
- (food0008.jpg, windows.jpg)
- Photographic/pseudo-periodic
- (D76.jpg)
- Photographic/random
- (metal0004.jog)
- Inhomogeneous (non-textures)
- (face.jpg)
http//www.cns.nyu.edu/lcv/texture/
80(No Transcript)