ImageVideo Editing Models A Review

1
Image/Video Editing Models -- A Review

• ZengXiang

2
The Taxonomy
Low

• Geometry/Graph based models
• PDE based(Level set, Poisson editing, etc),
  Snakes and most image processing techniques (edge
  detection, wavelets, )
• Statistic based models
• MRF, MoG, SVM, HMM,
• 3D info based models
• SFS, view morphing, optical flow, shade from
  motion,

Grab Cut
High
3
The Methodology

• User interaction quite essential!
• Energy minimization
• Combination leads to novelty!
• Global vs Local
• Interpolation
• Problem domain confinement.
• Weights exist in most models

4
Example

• Interactive Digital Photomontage
• Good combination of the Poisson Editing and
  Graph-cut techniques. Avoiding their drawbacks
  cleverly! And compensate for each other.

5
Interactive Digital Photomontage(The framework)

• The photomontage framework
• A source window
• Selecting from many source images
• A composite window
• Where the user can see and interact with the
  current result
• A labeling
• An array that specifies the source image for each
  pixel in the composite.

6
Interactive Digital Photomontage(The Framework)

• Objectives (Image Seam)
• The image objective at each pixel specifies a
  property that the user would like to see at each
  pixel in the designated area of the composite (or
  the entire composite, if it is specified
  globally).
• Image objective
• Designated color
• Minimum or maximum luminance
• Minimum or maximum contrast
• Minimum or maximum likelihood
• Eraser, Minimum or maximum difference
• Designated image
• Seam objective
• Colors
• Colors gradients
• Colors edges.

7
Interactive Digital Photomontage(The Framework)

• Brushes (for local editing)
• Multiple/single-image brush
• Refinement
• First, the user can enlarge the portion of the
  selected source image by painting additional
  strokes in the results window.
• Second, the user can adjust an inertia
  parameter the higher the inertia setting, the
  smaller the region of the source image that is
  automatically selected.

8
Interactive Digital Photomontage(Graph Cut)

• Graph-cut optimization
• The tth iteration of the inner loop of the
  algorithm takes a specific label a and a current
  labeling Lt as input and computes an optimal
  labeling Lt1 such that Lt1(p) Lt (p) or
  Lt1(p) a.
• The outer loop iterates over each possible
  label. The algorithm terminates when a pass over
  all labels has occurred that fails to reduce the
  cost function.

9
Interactive Digital Photomontage(Graph Cut)

• Cost Function
• Data penalty Cd , the distance to the image
  objective.
• Interaction penalty Ci, the distance to the seam
  objective.

10
Interactive Digital Photomontage(Graph Cut)

• The inertia control
• By calculating an approximate Euclidean
  distancemap Danielsson 1980 D(p) that describes
  the distance from the painted area at each point
  p.

11
Interactive Digital Photomontage(Gradient Fusion)

• Simply applying poisson image editing model to
  the region calculated by graph-cut.

12
Interactive Digital Photomontage(Applications)

• Selective composites.
• Image-based relighting.
• Extended depth-of-field.
• Image mosaics.
• Background reconstruction.
• Visualizing motion.
• Time-lapse mosaics.

13
Demo
14
A General Form For Image Editing

• Most problems that consider geometrical model can
  be expressed as
• and in bayesian form

15
Examples

• Graph cut
• Textureshop
• Poisson Cloning
• Feature Matching
• Flow synthesis
• Poisson Matting
• Colorization

16
And a more general framework

• From A unified variational image editing model
• Two questions
• What can be input image?
• How to make visually plausible?

17
What can be input?

• From other images or itself

18
What can be input?

• Color, luminance

19
What can be input?

• Alpha(Label)

20
How to be visually plausible?

• Minimizing Energies!
• Both Internal force
• and External force
• (Graph cut is only a
• discrete(0/1) two
• external force
• instance)

21
What can we do?
Low

• Inprovement of the existing algorithm
• More applications (more inputs)
• Consider more reasonable energy measurement
• Consider more image features(edges, contours)
• Combination of the existing models (should be
  seamless)
• New models (Unrealistic!)

High
22
What can we do?

• Colorization using graph cut my model
• Advantages faster, more reasonable.
• Problems how to deal with boundaries?

23
What can we do?

• Extension of graph cut algorithm
• The original one is only 0/1 logic, and discrete.
• Can it be n radix? Or be continuous?
• Combine the graph cut and my model together, make
  the graph cut an extreme case of the new model,
  it will be more applicable!

24
What can we do?

• NPR (non-realistic photo rendering)
• Define templates
• Energy minimization
• Embed style Learning
• No limit for the result!

25
What can we do?

• Consider more image features
• gradient, contours, edges. What else?
• Textures, higher level info normals, depth,
  strokes, anything that can be combined and
  minimized!

26
Combination of the existing models

• v

Geometrical Model
Statistical Model
3D model
Retexture, video alignment
Grab Cut
PDE based