Special Effects Based on Motion

1
Special Effects Based on Motion

• Paul Giaccone

2
Uses of motion in special effects

• Completed work extracting (segmenting) objects
  from background based on differing motion of
  objects and background
• for background replacement (compositing)
• for relocation of foreground objects
• for construction of background mosaics
• Future ideas depth extraction based on relative
  motion
• for reconstruction of 3D scenes
• for realistic insertion of new objects in scene
  in front of, behind and between existing objects

3
Object segmentation by motion and colour
Original sequence
1 Make colour model of each object in first
frame
2 Determine motion of each object in every frame
3 Label pixels based on their motion and colour
Masks for object 1 (background)
Masks for object 2 (actress)
4
Object segmentation by motion and colour

• Step 1 Generate colour models of objects in the
  first image of the sequence

Colour model for object 1 (background)
Colour model for object 2 (foreground)
5
Object segmentation by motion and colour

• Step 2 Determine motion of each object in the
  image

• Find background motion using a pan, tilt and
  zoom model to represent changes in rigid
  background due to camera movements

• Find foreground motion using an affine model to
  represent movements of each section of foreground
  object

6
Object segmentation by motion and colour

• Step 3 Assign each pixel to the object whose
  motion and colour it matches best

Original sequence
Sequence of segmentation masks

• Note fast-moving action
• Cyan and magenta areas are background that has
  been covered or uncovered by the motion of a
  foreground object

7
Post-processing

• To produce final masks
• Set covered pixels to foreground and uncovered to
  background
• Remove small clusters of misclassified pixels
• Make dense by filling large holes (manually
  selected)

8
Application object relocation

• Problem object in wrong position in image
• Possible solution
• Segment misplaced object
• Move object to new position

Original image
New image

• Need to reconstruct missing background how can
  this be done?

9
Background reconstruction

• Missing background may be recovered from a mosaic
  created by joining the background pixels of
  several images into a single image

Original sequence
Mosaic obtained from background pixels of all
images in sequence (not to same scale)
10
Mosaicking algorithm

• Segment and remove foreground objects from
  current image and neighbouring images
• Resulting background images contain background
  pixels only
• Compute motion from each background image to
  current image
• Displace (warp) pixels of background images by
  computed motion
• Pixels of all background images are now aligned
  with current image
• Combine warped images into background mosaic
• Blend overlapping parts of images to eliminate
  joins

Current image
11
Background recovered from mosaic
Relocated foreground
Original image
New image background taken from mosaic
12
Applications of mosaicking

• Viewport centralisation move off-centre objects
  to middle of every image of a sequence
• Image steadying remove shakiness from hand-held
  camera footage
• Action replays show events from various
  viewpoints
• 3D data from 2D images after recovery of
  occluded regions, image layers may be
  repositioned or viewed from different angles

13
Summary

• Segmentation objects may be segmented from image
  sequences on the basis of their motion and colour
• Series of masks produced for sequence
• Works even when action is fast-moving
• Mosaicking background pixels from several images
  may be aligned and combined to form panoramic
  views
• Applications in creating a variety of special
  effects

14
Acknowledgements

• The author wishes to thank the Computer Film
  Company for supplying unprocessed digital footage
  from the film Dolores Claiborne.