Image Compositing and Matting

1
Image Compositing and Matting
2
Introduction

• Matting and compositing are important operations
  in the production of special effects. These
  techniques enable directors to embed actors in a
  world that exists only in imagination, or to
  revive creatures that have been extinct for
  millions of years.
• During matting, foreground elements are extracted
  from a film or video sequence.
• During compositing, the extracted foreground
  elements are placed over novel background images.

3
Traditional approaches to matting

• Traditional approaches to matting include
  blue-screen matting and rotoscoping.
• The former requires filming in front of an
  expensive blue screen under carefully controlled
  lighting
• The latter demands talent and intensive user
  interaction.

4
Rotoscoping

• Rotoscoping --- the process of tracking contours
  in a video sequence

5
Basic problem of blue screen matting

• Given an image of a foreground object shot in
  front of a backing color (blue screen or green
  screen or other colors)
• Obtain a matte of the foreground object so that
  the foreground object can be blended into a new
  background image using the matte to produce a new
  composite image.

6
Notations

• Let C R, G,B denote a color with 0 R, G, B
  1.
• Let a denotes a transparency value with 0 a
  1.
• Foreground image color Cf Rf ,Gf ,Bf , af
  1
• Backing (screen) color Ck 0, 0,Bk, ak 1
  (assuming blue screen)
• Original foreground object color Co Ro,Go,Bo
• Background image color Cb Rb,Gb,Bb, ab 1
• Composite image color Cc Rc,Gc,Bc

7
Problem Statement

• Given Cf and Cb at corresponding pixels, and Ck a
  known backing color, and assuming Cf aoCo (1
  - ao)Ck
• Determine ao and Co, which then gives the
  composite color Cc aoCo (1 - ao)Cb at the
  corresponding point, for all points that Cf and
  Cb share in common.

8
Solution

• Since Cf aoCo (1 - ao)Ck , we have
• 4 unknowns, 3 equations

Rf aoRo Gf aoGo Bf aoBo (1 - ao)Bk .
9
Case 1 No Blue

• There is no blue in Co, i.e., Bo0, and Bk?0.
• Then
• This case is very restrictive.
• It rules out many colors, including grays because
  grays have blue.

10
Case 2 Gray and Skin Color

• Assume RoaBo (or GoaBo) and Bk?0
• Then
• where

11
Example 1 Gray

• Then
• This case applies to science fiction movie in
  which the spaceships are mostly gray.

12
Example 2 Skin Color

• This case applies to human faces, hands, legs,
  etc.

13
Solution for General Cases

• To solve the matting problem in general
• Need to take the same image with two different
  backing colors.
• This gives four equations for solving the four
  unknowns
• Case 1 Use Two Different Shades of Blue.
• Case 2 Use Two Different Backing Colors.

14
Case 1

• Use two different shades of blue Bk1 and Bk2 as
  backing colors.
• Then
• so

15
Case 2

• Use two different backing colors Ck1 and Ck2.
• Or
• Over-constrained, 1 unknowns, three equations

16
Case 2 Solution 1

• Adding up 3 equations

17
Case 2 Solution 2

• Apply least squares method
• Define
• Then
• Least-squares

18
Least-Squares

• Set

19
Case 1 Example
20
Case 2 Example
21
Other Developments in Matting

• Matting Without Blue Screen
• A method proposed by Ruzon and Tomasi
• User specify object region and boundary region.
• Alpha value of object region is set to 1.
• Alpha value of boundary region is computed by
  estimating the contributions of neighboring
  objects colors.

22
Example
23
Bayesian Approach
24
Shadow Matting

• Pull a matte of shadow.
• Acquire photometric and geometric properties of
  the target scene by sweeping oriented linear
  shadows across it.
• Then, composite the shadow onto the scene.

25
Example
26
Environmental Matting
Left alpha matte. Middle environment matte.
Right photo.
27
References

1. Y.-Y. Chuang, B. Curless, D. H. Salesin, and R.
   Szeliski. A bayesian approach to digital matting.
   In Proc. IEEE CVPR, pages II264II271, 2001.
2. Y.-Y. Chuang, D. B. Goldman, B. Curless, D. H.
   Salesin, and R. Szeliski. Shadow matting and
   compositing. ACM Transactions on Graphics,
   22(3)494500, July 2003.
3. M. A. Ruzon and C. Tomasi. Alpha estimation in
   natural images. In Proc. IEEE CVPR, pages 1825,
   2000.
4. A. R. Smith and J. F. Blinn. Blue screen matting.
   In Proc. ACM SIGGRAPH, pages 259268, 1996.
5. D. E. Zongker, D. M. Werner, B. Curless, and D.
   H. Salesin. Environment matting and compositing.
   In Proc. SIGGRAPH, pages 205214, 1999.

28
Digital Compositing

• Digital compositing means digitally manipulated
  integration of at least two source images to
  produce a new image.
• The new image must appear realistic.
• It must be completely and seamlessly integrated,
  as if it were actually photographed by a single
  camera.

29
Example 1
30
Example 2
31
More examples

• http//www.beezlebugbit.com/digital/efx/efx_top.ht
  m

32
Main Topics

• Alpha blending blending foreground and
  background
• Keying separating foreground and background
• Luma, chroma, difference keying
• Rig removal removing unwanted elements

33
Alpha Blending

• C a F (1 a) B
• If a 1, then C F, foreground is shown, i.e.,
  foreground is opaque.
• If a 0, then C B, background is shown, i.e.,
  foreground is transparent.
• 0 lt a lt 1 semi-transparent, e.g., shadow, smoke,
  etc.
• If a ranges from 0 to 255, then the formula
  becomes C a F (1 a) B / 255

34
Example No Background
35
Example With Background
36
(No Transcript)
37
Note

• For shadow, a must take fractional value (0 lt a lt
  1). Otherwise, shadow looks unreal.

38
Boundary area

• a at boundary area should also be fractional.
  Otherwise, have dark fringes unrealistic.

39
Summary

• A good matte has fractional a in shadow, and
  along object boundaries and shadow boundaries.

40
Keying

• Separating foreground from background, creating a
  matte of foreground.
• Also called pulling a matte (of foreground), or
  keying out (i.e., making transparent) background.
• Recall
• A good matte has fractional a in shadow, and
  along object boundaries and shadow boundaries.

41
Basic methods

• Luma keying based on luminance (i.e., intensity)
• Chroma keying based on color (i.e., blue screen,
  green screen)
• Difference keying requires a clean plate, i.e.,
  a background image without the foreground
  element.

42
Basic Idea

• Compute difference between foreground and
  background (based on luma, chroma, or color)
• Very small diff ? a 0.
• Very large diff ? a 1.
• Intermediate diff ? intermediate a

43
Luma Keying

• Key out the background based on luminance.
• Useful when background has a uniform luminance
  that is very different from foreground luminance.

44
Result
45
Chroma Keying

• Key out the background based on color.
• Useful when background has a uniform color that
  is very different from foreground color.
• Example Image shot with blue screen.

46
Characteristics of blue screen image
47
Difference Keying

• More general than luma and chroma keying.
• Key out background based on pixel-wise color
  difference between foreground and background
  footage.

48
Final Composition
49
Rig Removals

• Rigs are equipment that support the actors or the
  props.
• Sometimes, rigs cannot be removed by keying
  alone.
• So, have to apply masking technique to remove
  rigs.
• Need clean plate of background footage.
• If camera moves, then need motion-controlled
  camera
• Computer controls camera to move the same way
  twice
• Without foreground objects get clean plate.
• With foreground objects.

50
Basic Idea

• Apply a mask to mask out the rig.
• Then, replace pixels in masked area by
  corresponding pixels in clean plate background.
• If rig moves in footage, then have to animate the
  mask accordingly.