ImageBased Rendering

1
Image-Based Rendering
2
What Is an Image

• A 2D array of pixels
• (a continuous function on )
• each pixel (x,y) has
• a RGB (and ?) value
• more ?

3
What Is Rendering (1)

• Generation of a 2D image from a 3D scene
• The rendering pipeline
• Modeling
• Arranging geometric primitives in space
• Assembling objects from sets or hierarchies of
  primitives
• Assigning appearance parameters to the objects
  (color, shininess, texture, transparency)
• Describing how the objects move over time
  (animation)

4
What Is Rendering (2)

• Visibility
• Hidden-line
• Hidden-surface
• Hidden-volume
• Shading
• Display (frame buffer, z-buffer, CRT)

5
Computer Graphics
6
Problems of Geometric model Based Rendering

• Modeling is hard
• lack of realism
• Rendering is slow
• cost of rendering is dependent on the scene
  complexity

7
Computer Vision
8
Computer Graphics Computer Vision
9
(No Transcript)
10
What Is Image-Based Rendering?(1)
11
What Is Image-Based Rendering(2)

• Creating new views of a 3D environment based on
  existing images
• Advantages
• Speed,independent of scene complexity
• Source of images real or synthetic
• Disadvantages
• Memory usage
• Finite resolution
• possibly high precomputation costs

12
Previous Work Before Image-based Rendering

• Texture-mapping
• Environment-mapping
• Movie Map

13
Categories of Image-based Rendering

• Image mosaics
• Interpolation of images
• View morphing
• Interpolation from dense samples (Light field
  rendering)
• CG Rendering Acceleration

14
Image Mosaics
Different Images
combination
Higher resolution or lager image
15
Mosaic Image Representation
Planar Image
Cube
Sphere
Cylinder
16
Work in Nakajima Lab --Box Mosaic
Using images with forward motion

• Model each image into subspaces
• Decompose images into sub-images
• Compose sub-images correspondingly
• Form a box-like pseudo-3D space

17
3-D modeling from one image
For each image Specify vanishing point
Model 3D scene as a box-like with 3D polygons
2D texture Extract less than five
sub-images 3D polygons

• Top
• Bottom
• Left
• Right
• Rear

18
3D modeling from image sequence
- Based on image sequence with forward motion
- Decompose each image into less than five
sub-images
- Compose sub-images correspondingly into large
2D images
- Form a box-like pseudo-3D space
19
Image composition
Pseudo-3D space model A group of 3D
blocks
Four sides of pseudo-3D space Compose
correspondingly from four sides of
each blocks
Estimate coefficients between two images
20
Experiment and results(1)
Source image sequence
21
Experiment and results(2)
New views of virtual environment
22
Interpolation From Samples
Interpolation Morphing
Novel image
gt2 images

• View morphing
• Interpolation from dense sample

23
Image Morphing

• Rearrange pixels in an image

24
View Morphing (1)

• Morphing between parallel views
• parallel views
• camera position
• camera parameter projective matrices
• linear interpolation of pixels of two images.

25
View Morphing (2)
26
View Morphing (3)

• Morphing between Non-parallel views
• Prewarp
• Morph
• Postwarp

27
View Morphing (4)
28
Interpolation From Dense Sample

• The Plenoptic Function

29
Light Field Rendering (1)

• 4D Light Field
• a snap shot in time
• monochromatic wavelength
• convex hull of a bounded object

30
Light Field Rendering (2)

• Creating a Light Field

31
Light Field Rendering (3)

• Array of Images

32
Light Field Rendering (4)

• Fast Rendering

33
Light Field Rendering (5)

• 4D Interpolation

34
Light Field Rendering (6)

• Limitation
• Sampling density must be high
• Large, densely occluded environments
• Fixed illumination, static scenes

35
Geometrically-Valid Pixel Reprojection

• Transfer method
• Use relatively small number of images
• Use geometric constraints
• Epipolar constraints
• Trilinear tensors

36
Epipolar Geometry (1)
Left image plane
Right Image Plane
Left Optical Camera
Base Line
Right Optical Camera

• Epipole
• Epipolar plane
• Epipolar line

37
Epipolor Geometry (2)

• Fundamental Matrix
• Point is in the novel view image

Point in image 0 point
in image 2 F fundamental
matrix of rank 2
38
CG-Rendering Acceleration

• Depth Image
• RGB(?) value
• a Z (depth) value
• a Normal
• Depth image generation
• Ray tracing
• Z buffer

39
Depth Image Rendering
40
Special Case

• Infinite Depth
• Translation Invariant
• Environment Map
• Co-planar points
• Texture mapping
• Nearby images
• 2D affine transform

41
Work in Nakajima Lab (2)

• Purpose driving simulation
• Method video image-based rendering

Forward moving image sequence
Reproject
Virtual view
Depth Images
Simple Geometry
blend
Modeling
Rendering
42
Algorithm
low resolution
Large field of view
High resolution
43
Experiment Result
44
Image-Based RepresentationsComparison
Representation
Movement
Geometry
Lighting
Geometry Materials
Continuous
Global
Dynamic
Geometry Images
Continuous
Global
Fixed
Image Depth
Continuous
Local
Fixed
Light Field
Continuous
None
Fixed
Movie Map
Discrete
None
Fixed
Panorama
None
None
Fixed
45
Open Problem of Image-based Rendering

• Real imagery computer vision
• Camera pose hard to get
• Depth even harder to get
• Data Size maybe huge
• Changes Difficult
• light
• geometry