View Morphing by Steven M. SeitzCharles R. Dyer

1
View MorphingbySteven M. Seitz Charles R. Dyer
Comp 767 Advanced Topics in Graphics
Image-Based Rendering

• Irwin Chiu Hau
• Computer Science
• McGill University
• Winter 2004

2
Overview
Comp 767 Advanced Topics in Graphics
Image-Based Rendering
Mona Lisa view morphs Source http//www.cs.washin
gton.edu/homes/seitz/vmorph/vmorph.htm

• What is view morphing?
• How to do view morphing?
• Results
• Conclusion

3
View Morphing
Comp 767 Advanced Topics in Graphics
Image-Based Rendering
Virtual Cameras Source View Morphing Steven M.
Seitz, Charles R. Dyer

• What is view morphing?
• What is it?
• Why do we care about it?
• Where do we see them?
• Image Morphing vs View interpolation vs View
  Morphing

4
View Morphing Key Idea
Comp 767 Advanced Topics in Graphics
Image-Based Rendering

• Beier-Neely morph is NOT shape-preserving!
• distortions
• un-natural

A Shape-Distorting Morph Source Steven M. Seitz,
Charles R. Dyer
5
View Morphing Key Idea
Comp 767 Advanced Topics in Graphics
Image-Based Rendering

• View morphing uses 3D shape preserving morph!
• no distortions
• natural
• A morph is 3D shape preserving if the results of
  two
• different views represent new views of the same
  object

View morphing fixes these intermediate steps!
A Shape-Distorting Morph Source Steven M. Seitz,
Charles R. Dyer
6
Why do we care?
Comp 767 Advanced Topics in Graphics
Image-Based Rendering

• View morphing is efficient
• Produces new views without
• 3D modelling
• Taking additional photos
• View morphing creates impressive effects
• Camera motion
• Image morphing

trueSpace Source www.caligari.com
7
Image Morphing vs View interpolation vs View
Morphing
Comp 767 Advanced Topics in Graphics
Image-Based Rendering

• View Morphing is an extension to
• Image Morphing (Beier and Neely, 1992)
• Produces physically plausible new views of a
  scene
• View Morphing (Seitz and Dyer, 1996) is an
  improvement
• over View Interpolation (Chen and Williams,
  1993)
• Addresses non-rigid transformations problems
• Does not require depth values
• Creates realistic image transitions

8
Overview
Comp 767 Advanced Topics in Graphics
Image-Based Rendering
Mona Lisa view morphs Source http//www.cs.washin
gton.edu/homes/seitz/vmorph/vmorph.htm

• What is view morphing?
• How to do view morphing?
• Results
• Conclusion

9
How to do View Morphing?
Comp 767 Advanced Topics in Graphics
Image-Based Rendering

• View morphing in three steps
• Prewarp two images
• Compute a morph between the prewarped images
• Postwarp each in between images produced by the
  morph

View Morphing Procedure Source Steven M. Seitz,
Charles R. Dyer
10
How to do View Morphing?
Comp 767 Advanced Topics in Graphics
Image-Based Rendering

• View morphing in three steps
• Prewarp two images
• Compute a morph between the prewarped images
• Postwarp each in between images produced by the
  morph

• View morphing in 1 steps Assume parallel views!
• Compute a morph between the parallel images

11
Parallel Views
Comp 767 Advanced Topics in Graphics
Image-Based Rendering

• Basic Theory

image point p0 (x0,y0) scene point P
(X,Y,Z) p0 ?0 P where ?0 is a projection matrix
image point p1 (x1,y1) scene point P
(X,Y,Z) p1 ?1 P where ?1 is a projection matrix
Morphing Parallel Views Source Steven M. Seitz,
Charles R. Dyer
12
Parallel Views
Comp 767 Advanced Topics in Graphics
Image-Based Rendering

• Basic Theory

image point p (x,y) scene point P (X,Y,Z) p
? P where ? is a homogenous projection matrix ?
H - HC H position and orientation of
image plane C euclidean position of the camera
Morphing Parallel Views Source Steven M. Seitz,
Charles R. Dyer
13
Parallel Views
Comp 767 Advanced Topics in Graphics
Image-Based Rendering

• Mathematics for View Interpolation

?s Linear interpolation of ?0 and ?1 ?s (1
s) ?0 s ?1 s 0,1 Cs (sCx, sCy, 0) fs
(1 f) f0 s f1
f focal lengh C center of a camera
14
View Interpolation Recap
Comp 767 Advanced Topics in Graphics
Image-Based Rendering

• View Interpolation vs View Morphing

Morphing Parallel Views Source Steven M. Seitz,
Charles R. Dyer
View Interpolation Source 3D Games by Alan Watt
and Fabio Policarpo
15
How to do View Morphing?
Comp 767 Advanced Topics in Graphics
Image-Based Rendering

• View morphing in three steps
• Prewarp two images
• Compute a morph between the prewarped images
• Postwarp each in between images produced by the
  morph

• View morphing in 1 steps Assume parallel views!
• Compute a morph between the parallel images

16
Non-Parallel Views
Comp 767 Advanced Topics in Graphics
Image-Based Rendering

• The General Case

• This is where the 3-Step Algorithm
• comes into play
• Prewarping I0 to Î0 and I1 to Î1
• Morphing Î0 and Î1 into Îs
• Postwarping Îs into Is

Morphing in Three Steps Source Steven M. Seitz,
Charles R. Dyer
17
Non-Parallel Views
Comp 767 Advanced Topics in Graphics
Image-Based Rendering

• Mathematics for Image Reprojection

HH-1p p H and H are 3x3 matrices that represent
the position and the orientation of their image
planes The resulting 3x3 matrix, HH-1 is a
projective transformation that reprojects the
image plane I into Î
Morphing in Three Steps Source Steven M. Seitz,
Charles R. Dyer
18
Limitations
Comp 767 Advanced Topics in Graphics
Image-Based Rendering

• Singular Views cannot be reprojected to form
  parallel views
• Singular configurations are settings where one of
  the camera resides in the field of view of
  another camera
• Still works, just kind of conceptually hacky

Singular view Source Steven M. Seitz, Charles R.
Dyer
Parallel view Source Steven M. Seitz, Charles R.
Dyer
19
Traditional Problems
Comp 767 Advanced Topics in Graphics
Image-Based Rendering

• Change in visibility creates
• Folds
• Occurs when a visible area becomes occluded
• Holes
• Occurs when an occluded area becomes visible

Area
Point
Fold Source Irwin Chiu Hau
Penumbra, umbra and hole regions Source Chen and
Williams
20
Producing The Morph
Comp 767 Advanced Topics in Graphics
Image-Based Rendering

• We have talked about theory behind algorithm
• How to project/unproject images to parallel
  planes
• How to warp between parallel planes
• Theoretical problems
• Now, lets do an actual View Dependent Morph!

21
Producing The Morph
Comp 767 Advanced Topics in Graphics
Image-Based Rendering

• We need
• Two images I0 and I1
• Two perspective projection matrices ?0 and ?1
• Correspondence between pixels
• Note that a sequence of projection matrices ?s is
  required to
• control the entire animation, but ?s can be
  computed automatically
• if we know Hs .

22
Controlling The Morph
Comp 767 Advanced Topics in Graphics
Image-Based Rendering

• Hs can be obtained indirectly by establishing
  constraints
• Recall Hs is position and orientation of image
  plane
• One way is to specify four control points
• Note Control points implicitly determine the
  postwarping
• transformation

Four control points form the red bounding box to
determine the postwarping stage Yellow lines are
set of features to determine the prewarping stage
View Morphing Procedure Source Steven M. Seitz,
Charles R. Dyer
23
View Morphing sans Prewarping
Comp 767 Advanced Topics in Graphics
Image-Based Rendering

• Prewarping is not necessary for
• Objects that arent closely related
• Prewarping is less effective
• Computation is unstable
• Images that are approx. orthographic (eg.
  telephoto)
• However, postwarping should not be left out to
• Reduce image plane distorsions

24
Overview
Comp 767 Advanced Topics in Graphics
Image-Based Rendering
Mona Lisa view morphs Source http//www.cs.washin
gton.edu/homes/seitz/vmorph/vmorph.htm

• What is view morphing?
• How to do view morphing?
• Results
• Conclusion

25
Results
Comp 767 Advanced Topics in Graphics
Image-Based Rendering
Facial view morphs Source Steven M. Seitz,
Charles R. Dyer
26
Results
Comp 767 Advanced Topics in Graphics
Image-Based Rendering
Facial view morphs Source http//www.cs.washingto
n.edu/homes/seitz/vmorph/vmorph.htm
27
Results
Comp 767 Advanced Topics in Graphics
Image-Based Rendering
Mona Lisa view morphs Source http//www.cs.washin
gton.edu/homes/seitz/vmorph/vmorph.htm
28
Results
Comp 767 Advanced Topics in Graphics
Image-Based Rendering

• Image Morphing vs View Morphing

Image Morphing vs View Morphing Source Steven M.
Seitz, Charles R. Dyer
29
Conclusions
Comp 767 Advanced Topics in Graphics
Image-Based Rendering

• Things to remember
• View Morphing
• Powerful extension to image morphing
• Produces new views of a scene
• 3D shape preserving

30
References
Comp 767 Advanced Topics in Graphics
Image-Based Rendering

• View Morphing - Seitz and Dyer, 1996
• View Interpolation - Chen and Williams, 1993
• Image Morphing - Beier and Neely, 1992
• 3D Games Realtime rendering and Sofware
  Technology
• - Alan Watt and Fabio Policarpo

31
Questions?
Comp 767 Advanced Topics in Graphics
Image-Based Rendering