Hybrid Texture Synthesis

1
Hybrid Texture Synthesis

• Andrew NealenMarc AlexaDiscrete Geometric
  Modeling Group (DGM)Technische Universität
  Darmstadt
• Eurographics Symposium on Rendering 2003

2
Outline
3
Introduction

• 2D Texture Synthesis

nxm Input Texture
NxM Output Texture

• The goal Synthesize an output texture which is
  perceptually similar to the input texture. Also
  ensure that the result contains sufficient
  variation.

4
Introduction

• Existing (and Impressive) Technology
• Pixel-Based
• Non-parametric Sampling Efros and Leung 1999
• Tree-structured Vector Quantization Wei and
  Levoy 2000
• Image Analogies Hertzmann et al. 2001
• Patch-Based
• Patch-Based Sampling Guo et al. 2001
• Image Quilting Efros and Freeman 2001

5
Introduction

• Possible Drawbacks ?
• Loss of global structure
• Loss of scale
• Boundary mismatch
• Blurring

Wei/Levoy Algorithm Can occur when using
structured Textures with rich histograms. This
is due to the L2 norm.
6
Introduction

• Possible Drawbacks ?
• Loss of global structure
• Loss of scale
• Boundary mismatch
• Blurring

Image Quilting Algorithm boundary mismatch
artifacts are noticeable
7
Introduction

• Possible Drawbacks ?
• Loss of global structure
• Loss of scale
• Boundary mismatch
• Blurring

Patch-Based Sampling Algorithm Can occur when
texture features are not well-aligned across
patch boundaries
8
Hybrid Texture Synthesis

• Combine best aspects from other approaches while
  avoiding (or improving over) known pitfalls
• Patch-based algorithms
• Are good at preserving global structure
• Can introduce artifacts along patch boundaries
• Pixel-based algorithms
• Preserve local coherence (MRF model local and
  stationary)
• Possibly fail to preserve global structure
• This is especially problematic for textures with
  rich histograms and many high frequency features,
  due to the smoothing nature of the distance
  metric (L2 norm)

9
Hybrid Texture Synthesis (HTS)

• Hybrid, two-fold approach to texture synthesis
• Adaptive patch sampling Soler et al. 2002
• Start with a uniform quadrilateral grid of
  patches (the output)
• Recursively split a patch if the best fit taken
  from the input texture is not good enough (a
  user-defined tradeoff ?max)
• Overlap re-synthesis (novel overlap repair
  strategy)
• Mark invalid pixels in the overlap region
• Validity map is defined by the error surface
  E(x)(xsrc xdst)2 and a user-defined tolerance
  parameter ?max
• Define an ordering for the invalid pixels and
  re-synthesize them using a per-pixel strategy

10
HybridSynthesize Algorithm

• 1 HYBRIDSYNTHESIZE(T,P,ov,?max,dmax,R) R
• 2 for all patches pi ?P do
• 3 Pi,?i? FINDBESTPATCH(T,Ri-1, pi,ov)
• 4 if (?i lt ?max or ISSINGLEPIXEL(pi)) then
• 5 Si ? ERRORSURFACE(Pi,Ri-1)
• 6 Pi ? MARKINVALIDPIXELS(Pi,Si,dmax)
• 7 Mi ? BUILDTRAVERSALMAP(Pi)
• 8 Ri,composite ? COMPOSE(Pi,Ri-1)
• 9 Ri ? OVERLAPRESYNTHESIS(T,Ri,composite,Mi
  )
• 10 else
• 11 Ps ? SPLITPATCH(pi)
• 12 ovs ?max(3, ov/2)
• 13 Ri ? HYBRIDSYNTHESIZE(T,Ps,ovs,?max,dmax,
  Ri-1)
• 14 end if
• 15 end for
• 16 return R

11
FindBestPatch(Algo.)

• 1 FINDBESTPATCH(T,Ri-1, pi,ov) Pi,?i
• 2 Ii, Ji ? BUILDIMAGEMASK(Ri-1, pi,ov)
• 3 Ei ? ERRORIMAGE(T, Ii, Ji)
• 4 Ei,trim ? TRIMINVALIDREGIONS(Ei, Ji, pi)
• 5 Pi,?i? BESTPATCH(Ei,trim,T)
• 6 return Pi,?i

12
In Search of Good PatchesSynthesizing a single
patch
Synthesize Black Patch i
13
In Search of Good PatchesExtracting the image
mask
Synthesize Black Patch i

• Grow patch by overlap (toroidally)

14
In Search of Good PatchesExtracting the image
mask
Synthesize Black Patch i
Image Mask (Ii)
Binary Support (Ji)

• Extract image mask (Ii) and binary support (Ji),
  and circularly shift to upper left corner

15
In Search of Good PatchesComputing the error
image
Image Mask (Ii)
Input Texture (T)
Binary Support (Ji)

• Compute error Ei(x0) between Ii and T for each
  circular shift x0(?x,?y) of the input texture T

16
In Search of Good PatchesComputing the error
image
CR,G,B (both with RGB color values in
0,1) WR,G,B 0.299,0.587,0.114 X0
circular shift of T
Error Image (Ei)
Input Texture (T)
(1)
17
In Search of Good PatchesSelecting a Patch
Error Image (E)
Input Texture (T)
Selected Patch (P)
(2)

• Given that Ji(x) Ii(x) Ii(x),and the cross
  correlation between two images (functions) f ?
  g is defined as ( f ? g )(x0) Sx f (x)g(xx0)
• The correlation f ? g between two functions
  can be computed in O(N logN)
• in fourier space as f ? g F-1(F( f )F(g))
• (N number of pixels in the input texture)
• In implementation its can be pre-compute the
  fourier transform for T and need only recompute
  the fourier transforms of Ii and Ji for each new
  patch pi.

18
In Search of Good PatchesSelecting a Patch
Error Image (E)
Input Texture (T)
Selected Patch (P)

• Note error image can be computed efficiently in
  the Fourier domain. Complexity O(n log n) per
  patch.

19
Adaptive Patch SamplingPrinciple
Synthesize White Patch
?i gt ?max ? Split Patch
?i lt ?max

• Essentially Hierarchical Pattern Mapping applied
  to the plane Soler et al. 2002

20
Adaptive Patch SamplingTrade-off
?max 1.0
?max 0.01
?max 0.04

• Trade-off between preserving global structure and
  avoiding detail artifacts

21
Adaptive Patch SamplingTrade-off
?max 1.0
?max 0.01
?max 0.04

• Trade-off between preserving global structure and
  avoiding detail artifacts

22
Adaptive Patch SamplingTrade-off
?max 1.0
?max 0.01
?max 0.04

• Trade-off between preserving global structure and
  avoiding detail artifacts

23
Overlap Re-synthesisOverlap error and pixel
invalidation
Patch and Image Mask
Selected Patch (P)
24
Overlap Re-synthesisOverlap error and pixel
invalidation
Image Mask (I)
Selected Patch (P)
25
Overlap Re-synthesisOverlap error and pixel
invalidation
Image Mask (I)
Selected Patch (P)
Error Surface (S)
26
Overlap Re-synthesisOverlap error and pixel
invalidation
Patch and Image Mask
P with Invalid Pixels (blue)
Error Surface (S)

• If(Ji(x) !0 and Si(x)gt dmax )
• as invalid and in need of per-pixel
  re-synthesis.
• else if( Ji(x) ! 0 and Si(x) lt dmax )
• are preserved and a stepwise alpha mask
  (feathering) is applied during Compositing.
• (Trade-off user-defined ?max is the pixel error
  tolerance. Setting to 1 results in pure
  feathering.)

27
Overlap Re-synthesisCompositing
P OVER Ri-1
Intermediate Result
Patch with Invalid Pixels
28
Overlap Re-synthesisCompositing
Composited Result
Patch with Invalid Pixels

• Simple scanline re-synthesis possibly
  insufficient causal neighborhood within valid
  pixel region
• Solution alternative ordering (Pixel Traversal
  Map)

29
Pixel Traversal-Map

• 1 BUILDTRAVERSALMAP(Pi) Mi
• 2 level 1
• 3 Dlevel ?binary image of size Pi initialized to
  0
• 4 Dlevel ?set all valid pixels ? Pi to 1
• 5 Mi ?Dlevel
• 6 while (?pixel ? Dlevel ? pixel 0) do
• 7 level level1
• 8 Dlevel ? DILATE(Dlevel-1,Disk)
• 9 Mi ?Mi (Dlevel ?Dlevel-1) level
• 10 end while
• 11 return Mi

30
Overlap Re-synthesisPixel Traversal Map
(Ordering)
Composited Result
Patch with Invalid Pixels
Pixel Traversal Map (M)

• Pixel Traversal Map repeated morphological
  dilation of the binary support for valid pixels
  with a euclidian disk of radius 1 (city-block
  distance transform).

31
Overlap Re-synthesisPixel Traversal Map
(Ordering)
Composited Result
Patch with Invalid Pixels
Pixel Traversal Map (M)

• Pixel Traversal Map step 1

32
Overlap Re-synthesisPixel Traversal Map
(Ordering)
Composited Result
Patch with Invalid Pixels
Pixel Traversal Map (M)

• Pixel Traversal Map step 2 ...

33
Overlap Re-synthesisPixel Traversal Map
(Ordering)
Composited Result
Patch with Invalid Pixels
Pixel Traversal Map (M)

• Pixel Traversal Map ... step n

34
Overlap Re-synthesisPixel Traversal Map
(Ordering)
Composited Result
Patch with Invalid Pixels
Pixel Traversal Map (M)

• Art Restorer Analogy stepwise restoration of the
  hole from the boundary of the existing image.

35
Overlap Re-synthesisPer-Pixel Re-synthesis
Pixel Traversal Map (M)
Intermediate Result

• Synthesize red pixel from valid (or already
  re-synthesized) pixels

36
Overlap Re-synthesisPer-Pixel Re-synthesis
Image Mask (Ij)
Binary Support (Jj)
Intermediate Result

• Extract image mask (Ij) and binary support (Jj)
  for best-pixel search in the input texture.
  Complexity O(n log n) per pixel.

37
Overlap Re-synthesisPer-Pixel Re-synthesis
Intermediate Result
Pixel Traversal Map (M)
38
Overlap Re-synthesisPer-Pixel Re-synthesis
Intermediate Result Shifted
Pixel Traversal Map (M)
39
Overlap Re-synthesisPer-Pixel Re-synthesis
Intermediate Result with Traversal Map
Pixel Traversal Map (M)
40
Overlap Re-synthesisPer-Pixel Re-synthesis
Overlap Re-synthesis Step 1
Pixel Traversal Map (M)
41
Overlap Re-synthesisPer-Pixel Re-synthesis
Overlap Re-synthesis Step 2
Pixel Traversal Map (M)
42
Overlap Re-synthesisPer-Pixel Re-synthesis
Overlap Re-synthesis Step 3
Pixel Traversal Map (M)
43
Overlap Re-synthesisPer-Pixel Re-synthesis
Overlap Re-synthesis Result
Pixel Traversal Map (M)
44
ResultsOverlap Repair Comparisons
No Repair
HTS
PBS
IQ
45
ResultsOverlap Repair Comparisons
No Repair
HTS
PBS
IQ
46
ResultsOverlap Repair Comparisons
No Repair
HTS
PBS
IQ
47
ResultsOverlap Repair Comparisons
No Repair
HTS
PBS
IQ
48
ResultsOverlap Repair Comparisons
No Repair
HTS
PBS
IQ
49
ResultsOverlap Repair Comparisons
No Repair
HTS
PBS
IQ
50
ResultsOverlap Repair Comparisons
No Repair
HTS
PBS
IQ
51
ResultsOverlap Repair Comparisons
No Repair
HTS
PBS
IQ
52
ResultsOverlap Repair Comparisons
No Repair
HTS
PBS
IQ
53
ResultsOverlap Repair Comparisons
No Repair
HTS
PBS
IQ
54
ResultsOverlap Repair Comparisons
No Repair
HTS
PBS
IQ
55
ResultsOverlap Repair Comparisons
No Repair
HTS
PBS
IQ
56
ResultsOverlap Repair Comparisons
No Repair
HTS
PBS
IQ
57
ResultsSynthesis Comparisons
Input
Efros/Leung
Wei/Levoy
IQ
PBS
HTS
58
ResultsSynthesis Comparisons
Input
Efros/Leung
Wei/Levoy
IQ
PBS
HTS
59
ResultsSynthesis Comparisons
Input
Efros/Leung
Wei/Levoy
IQ
PBS
HTS
60
ResultsSynthesis Comparisons
Input
Efros/Leung
Wei/Levoy
IQ
PBS
HTS
61
ResultsSynthesis Comparisons
Input
Efros/Leung
Wei/Levoy
IQ
PBS
HTS
62
ResultsSynthesis Comparisons
Input
Efros/Leung
Wei/Levoy
IQ
PBS
HTS
63
ResultsSynthesis Comparisons
Input
PBS
HTS
Input
PBS
HTS
64
ResultsSynthesis Comparisons
Input
PBS
HTS
Input
PBS
HTS
65
ResultsSynthesis Comparisons
Input
PBS
HTS
Input
PBS
HTS
66
Results
presence of high frequency structure cause the
error metric (L2 Norm)
67
Conclusions and Future Work

• Improve Computational Complexity
• Pixel neighborhoods of patch and pixel stage are
  not known a priori, so precomputation is not
  straightforward
• We can solve this in the pixel stage by employing
  k-coherence search Tong et al. 2002 Ashikhmin
  2001
• Improve Error Metric
• Still using the L2 norm due to its simplicity
• Develop a metric which takes feature mismatch
  into account