Presentaci

1
COMS 6160 Real-Time High Quality Rendering Nov
3rd, 2004
Relighting Framework
Sebastian Enrique Columbia University senrique_at_cs.
columbia.edu
2
COMS 6160
Relighting Framework
Nov 3rd, 2004
About Relighting
What is Relighting? Given a set of different
illuminated images from a scene, relighting is
the process of producing new images of the same
scene with new lighting conditions, composing in
some way the original data.
Why it is useful? Photorealistic real-time
rendering of complex scenes with complex
illumination is an open problem. One IBR approach
is to capture or render a set of original images
from a scene, and then relight it in real time to
produce the same scene but with novel
illumination!
Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
2
3
COMS 6160
Nov 3rd, 2004
Relighting Framework
About Relighting (cont.)
What is one of the most challenging parts of
it? High quantity of images is needed to produce
good results, the challenge is to find/use a good
compression technique in order to relight fast
and using as less memory as possible.
Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
3
4
COMS 6160
Nov 3rd, 2004
Relighting Framework
Some Previous Work

• Doorsey et al. used a progressive radiosity
  method where they pre-rendered synthetic scenes
  to simulate lighting conditions superimposing
  single light source images- in opera stages in
  1991.
• Many (Hallinan 94, Epstein 95, etc.) have
  pre-acquired real images changing the lighting
  direction.
• Debevec compressed the pre-acquired images in
  JPG and processed in the compressed domain in
  2000.
• Sloan uses low-frequency spherical harmonics on
  geometry in 2002.
• In 2003 Sloan uses clustered or VQPCA on
  spherical harmonic coefficients.
• Ng compressed data using wavelets in 2003.
• Sloan (Local Deformable PRT) and Ramamoorthi
  (Triple Product Wavelets) are the most recent
  related techniques, 2004.

Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
4
5
COMS 6160
Nov 3rd, 2004
Relighting Framework
LSD Relighting Approach

• The Lighting Sensitive Display (LSD) was
  developed by Shree Nayar, Peter Belhumeur, and
  Terry Boult.
• Basically, it is a monitor with an attached
  camera that captures the lighting conditions of
  the environment. The monitor shows an scene,
  which changes (is relighted) as the illumination
  in the environment changes.
• They adopted an image-based approach, using a
  large set of images.

Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
5
6
COMS 6160
Nov 3rd, 2004
Relighting Framework
LSD Relighting Approach (cont.)

• They have developed a novel algorithm (using two
  stages of Principal Component Analysis or PCA)
  that compresses that large set of images and
  allows the relighting in real-time with complex
  lighting conditions.
• They have reached compression ratios of 4761
  for colored images.
• The algorithm simultaneously exploits
  correlations over the lighting domain as well as
  coherences over the spatial domain of the image.

Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
6
7
COMS 6160
Nov 3rd, 2004
Relighting Framework
LSD Relighting Approach (cont.)

• We based our Relighting Framework on the LSD
  approach.
• In the following slides we will explain
• How the input images for the LSD algorithm
  should be taken.
• First compression stage of the algorithm.
• Second compression stage of the algorithm.
• Real-Time rendering.
• Then, we will get on
• Extending the LSD approach with the use of
  cubemaps.
• Problems found.
• Future directions.
• Discussion and Videos

Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
7
8
COMS 6160
Nov 3rd, 2004
Relighting Framework
Setting Up Images

• To take the images, a grid of light source
  positions is generated in the front face of an
  imaginary cube (a plane parallel to the image
  plane).
• For each position on the grid, place the distant
  light with maximum intensity and generate (render
  or capture) an image.

Scene to Render / Capture
Distant Light Source For Single Image
Fixed Viewpoint
Grid of Positions for Distant Light Sources, its
in the front face of an imaginary cube.
Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
8
9
COMS 6160
Nov 3rd, 2004
Relighting Framework
First Stage SVD

• The first SVD (Singular Value Decomposition)
  exploits the fact that locally (within small
  regions of the images) the variation due to
  changes in illumination can be approximated by a
  small number of bases.
• The image is divided into m square blocks, each
  containing p pixels, and bases are computed for
  each.
• Each image Ii is an image of the scene
  illuminated by a single distant point light
  source.

Image divided in m square blocks
Each block has p pixels
Image Ii
IMAGES
Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
9
10
COMS 6160
Nov 3rd, 2004
Relighting Framework
First Stage SVD (cont.)

• Iij denote the jth block of the ith image.
• For each block in the scene a low-dimensional
  approximation is computed as follows
• Ij is a p x n matrix representing a collection
  of image blocks. The I column of Ij is formed by
  p pixels of the jth block.
• Using SVD, a rank b approximation to Ij is found
  as
• Ej is a p x b column-orthogonal matrix, called
  block bases.
• Sj is a b x b diagonal matrix.
• Cj is an n x b column-orthogonal matrix.
• The singular values from Sj are absorbed into
  CjT, getting a b x n matrix, Lj, called block
  lighting coefficients.

Each Ij represents the pixels for the same block
on all images
Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
10
11
COMS 6160
Nov 3rd, 2004
Relighting Framework
First Stage SVD (cont.)

• SVD

Ij
Sj
Ej

x
x
p x n
p x n
p x p
CjT
diagonal matrix (singular values)
n x n
Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
11
12
COMS 6160
Nov 3rd, 2004
Relighting Framework
First Stage SVD (cont.)

• Then, stacking all of the m Lj block matrices,
  we get the lighting coefficient matrix L.
• The image bases m Ej are also stacked in the
  matrix E.
• The collection of submatrices within E and L
  contain all the information needed to approximate
  the collection of images corresponding to the n
  lighting directions.

Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
12
13
COMS 6160
Nov 3rd, 2004
Relighting Framework
First Stage SVD (cont.)
Image 640x480 n64x644096 m40x301200
p16x16256 b10
Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
13
14
COMS 6160
Nov 3rd, 2004
Relighting Framework
Second Stage SVD

• The fact that there is much coherence in image
  blocks is exploited using a second SVD.
• It is applied to the light coefficient matrix L
• U is an (m x b) x q column-orthogonal matrix,
  called lighting coefficient bases.
• V is a q x n matrix, called compressed
  coefficient matrix.
• Rank q denotes the number of linear bases kept
  to approximate L.

Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
14
15
COMS 6160
Nov 3rd, 2004
Relighting Framework
Second Stage SVD (cont.)
Image 640x480 n64x644096 m40x301200
p16x16256 b10 q200
Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
15
16
COMS 6160
Nov 3rd, 2004
Relighting Framework
Examples
Bad Rank Election
(showing only red color component)
Relighted Image SVD First Stage Rank 3
Original Image
Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
16
17
COMS 6160
Nov 3rd, 2004
Relighting Framework
Examples (cont.)
Higher Ranks
Only small differences in brightness are
noticeable, but there are no visible
block discontinuities.
Relighted Image SVD First Stage Rank 10 SVD
Second Stage Rank 20
Original Image (in fact, two combined original
images)
Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
17
18
COMS 6160
Nov 3rd, 2004
Relighting Framework
Real-Time Rendering

• Real-Time illumination is represented by the
  illumination field vector s.

• Each element si corresponds to a point light,
  and its value represents the intensity that that
  point light is contributing with to the scene.
• To render the relighted scene in real-time,
  preprocessed matrices E, U, and V should be used

1. Compute a compressed coefficient vector as the
   product (q elements)
2. Compute a lighting coefficient vector I as
   (mb elements)
3. To render the j block, a subvector Ij from I must
   be multiplied with the corresponding stacked Ej
   matrix. This must be done each frame for the m
   blocks.

(b elements) (p
elements)
Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
18
19
COMS 6160
Nov 3rd, 2004
Relighting Framework
Real-Time Rendering (cont.)
Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
19
20
COMS 6160
Nov 3rd, 2004
Relighting Framework
Demo Point Lights / David
Image 480x640 n6x16x161536 m32x321024
p16x16256 b10 q20
Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
20
21
COMS 6160
Nov 3rd, 2004
Relighting Framework
Extending LSD Cube Lighting

• In place of using only the front face of the
  cube to capture images with different lighting
  condition, use all of the 6 faces.
• The parameters that change are the quantity of
  input images and the length of the s light field
  vector (both now 6 n).
• In this way, the scene can be relighted with
  illumination coming for all around.

Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
21
22
COMS 6160
Nov 3rd, 2004
Relighting Framework
High-Dynamic Range Cubic Environment Map

• Having the relighting scheme for the full cube,
  we can add cubic environment maps to relight the
  scene.
• To have more accuracy, High-Dynamic Range (HDR)
  cubic environment maps are used (floating point
  values in place of 0..255).
• The CubicMap could be rotated, and each element
  in the s light field vector is contributed by the
  value of the corresponding texel, weighted by the
  solid angle.

• HDR(i) is the corresponding texel value N is
  the corresponding cube face normal R is the
  vector from the origin to the center of the
  element i in the cube grid m is the face cube
  grid resolution (squared is how many images per
  face exist) each cube face has a size of 2 x 2.

Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
22
23
COMS 6160
Nov 3rd, 2004
Relighting Framework
Demo CubeMap Point Lights / Nicole
Image 512x512 n6x16x161536 m32x321024
p16x16256 b10 q20
Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
23
24
COMS 6160
Nov 3rd, 2004
Relighting Framework
Framework Summary
Generate Scene-Matrices For Images Set Of Each
Scene to Relight
Relight in Real-Time the Given Scene-Matrices
with HDR CubeMap and Point Lights
Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
24
25
COMS 6160
Nov 3rd, 2004
Relighting Framework
Problems Found

• Preprocess takes TOO MUCH TIME.
• Preprocess uses TOO MUCH MEMORY.
• Few things could be done in current graphics
  hardware (HDR cubemap processing specially).

Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
25
26
COMS 6160
Nov 3rd, 2004
Relighting Framework
Future Directions

• Optimize preprocessing stages (distributed
  computations?)
• Use error metrics to automatically select
  adequate ranks.
• Extend user interface to allow the relighting
  using lower ranks than that given in the input
  matrices.
• Allow viewpoint changes.

Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
26
27
COMS 6160
Nov 3rd, 2004
Relighting Framework
End of Talk

• Ready to hear
• Comments
• Suggestions
• Discussions
• Questions
• More VIDEOS to show while chatting

Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
27