Graphcut Textures:Image and Video Synthesis Using Graph Cuts

1
Graphcut TexturesImage and Video Synthesis Using
Graph Cuts

• Vivek Kwatra ,Arno Schodl ,Irfan Essa ,
• Greg Turk ,Aaron Bobick

2
Outline

• Texture Synthesis
• What is texture?
• How to synthesis?
• Graphcut Texture
• Main idea, Contribution
• Patch placement and matching techniques
• Patch fitting
• Refinements and extensions
• Video Textures

3
Texture

• Generate a large image from smaller image or
  longer video from smaller one
• How to do it?
• Copy patches (or pixels ) from input to output
• Problems
• Artifacts ( e.g., boundaries of patches)

4
Solution

• Copy patches from input to output with overlap

5
Definitions

• Where to position the input texture called Offset
• Which part of the input texture to transfer
  called Seam

6
Cutting the graph
cut
1
4
7
8
8
cut
existing pixels A
new patch B
new patch
8
8
corresponding graph
old pixels
2
5
8
8
8
3
6
9
overlap
Use Max Flow/ Min Cut algorithm to find the best
cut where the cost between two adjacent nodes s
and t M(s,t,A,B) A(s) B(s) A(t)
B(t) A(s) and B(s) are pixel colors in
position s in the old and new patch respectively
7
Cutting the graph
8
old pixels
cut
existing pixels
8
1
4
7
8
corresponding graph
cut
2
5
8
new patch
8
3
6
9
8
new patch
overlap
8
8
old cut
existing pixels
1
5
9
13
cut
overlap
2
6
10
14
old pixels
new patch
3
7
11
15
corresponding graph
new cut
4
8
12
16
8
8
Seam nodes
Problem What should we do if we lay down a patch
in a spot that already has existing seams? How
should we take preexisting seams into account
when making a cut?
old cut
new cut
old cut
new cut
1
4
7
8
8
existing pixels A
new patch B
new patch
8
8
corresponding graph
old pixels
2
5
8
8
8
overlap
3
6
9
9
Seam nodes
Solution add a seam node between two pixels and
connect the seam node with an arc to the new
patch. The weight of the arc will be the old
cost between the pixels.
old cut
new cut
old cut
new cut
1
4
7
8
8
S1
existing pixels A
new patch B
S2
new patch
8
8
corresponding graph
old pixels
2
5
8
S3
8
8
overlap
3
6
9
S4
Then add arcs from pixels to seam node with new
costs
M(1,2, A, B)
M(1,2, B, A)
B
A
1
4
7
S1
8
8
10
(No Transcript)
11
(No Transcript)
12
The Synthesis Process

• Step1Patch placement and Matching ( Choose
  candidate patches or offset)
• Random Placement
• Entire Patch Matching
• Sub-patch Matching
• Step2Patch Fitting ( Choose optimal portion or
  seam)
• Only those pixels are copied that are chosen by
  graph-cut algorithm
• Cost of graph-cut is a measure of similarity

13
Stemp1. Patch Placement

• Random Placement
• Entire input image is translated to random
  location in the output image
• Good results for random textures
• Patch Matching ( Entire or Sub )
• Used when we already have some patches in the
  output image (refinement).
• Every seam has a cost (min graph cut cost)
• Uses error region

14
Sub-Patch Matching
15
Entire Patch Matching
16
Error Region

• Error seam cost
• Sum of costs along minimum cut patch
• Choose a pixel with largest error
• Select a region around that pixel, called error
  region
• Patch Matching ( entire or sub ) will select
  those patches that completely cover our error
  region

17
Patch Placement

• Entire Patch Matching
• Search for translated input versions and choose
  that gives best match
• Matching criteria
• Normalized SSD
• C(t) The smaller, the better (means similar)

18
Patch Placement

• Compute cost for all possible offsets. Cost is
  inversely proportional to similarity
• Choose the cost that has the highest probability
  of resulting in a similar region
• A low value of k leads to picking of only those
  patch locations that have a very good match with
  the output whereas a high value of k leads to
  more random patch selection.
• Good results for structured and semi-structured
  texture

19
Patch Placement

• Sub-Patch Matching
• Pick a small sub-patch from output
• Search for output-patch in input texture or look
  for translations of input sub-patch
• Matching criteria
• Use the same probability function
• Best results for unstructured regions or video
  textures ( fire, waves, smoke, etc.)

20
Patch Fitting

• Make graph for overlap region
• Every pixel is a node
• Edge weights
• Associate weight with each edge
• Find minimum graph cut

21
Refinements

• Modified Matching cost function

22
Refinements

• Search across all translations is costly (use
  FFT)
• speed up
• 150x100x30 video sequence
• Search for a new patch reduces from around 10
  minutes (using naïve search) to 5 seconds (using
  FFT-based search)

23
Extensions

• Translation to Transformation
• Rotation, scaling, affine or projection
• Interactive merging and blending
• Many source images
• User specifies position constraints pixels
• Algorithm finds best seam
• SIGGRAPH banner

24
Interactive merging and blending
25
Interactive merging and blending
26
Extensions

• Goal is to loop the video forever
• Video texture
• One way is to find the pair of similar looking
  frames and use them to repeat the video
• Video synthesis using graphcut
• Find time of transition on pixel-wise basis

27
Video Texture GraphCut

• Finding Seams for Video Transitions
• Find good transition between pair of images
• Take a window around transition (60 frames)
• Construct the graph by connecting a pixel to its
  neighbors in space and time
• Min cut will give you time of transition on per
  pixel basis
• Use translation in time and space both

28
Video Synthesis
29
Video Synthesis

• Temporal Constraints for Video
• To loop the video, add k frames in start and end
  of video (10 frames), constraint these frames to
  stay the same, graph is generated and best seam
  is found, then k frames are removed.

30
Video Synthesis

• We fix the first k and last k frames of the
  output sequence to be the same k frames of the
  input.
• The pixels in these frames are now constrained to
  stay the same.
• This is ensured by adding links of infinite cost
  between these pixels and the patches they are
  constrained to copy from, during graph
  construction
• Use graphcut to find the best seam given that
  these pixels dont change.
• Once the output has been generated, we remove the
  first k frames from it.
• This ensures a video loop since the kth frame of
  the output is the same as its last frame before
  this removal operation

31
(No Transcript)
32
(No Transcript)