Panoramas

1
Panoramas
2
Creating Full View Panoramic Image Mosaics and
Environment Maps

• Richard Szeliski and Heung-Yeung Shum
• Microsoft Research

3
Outline

• Main Contribution
• Introduction
• Details
• Results

4
Contributions

• Novel approach to creating full view panoramic
  mosaics from image sequence
• Not necessarily pure horizontal camera panning
• Do not require any controlled motions or
  constraints
• Represent image mosaics using a set of transforms
• Fast and robust
• Method to recover camera focal length
• Extract environment map from image mosaic

5
Introduction

• Image-based rending
• Realistic rendering without geometry models
• IBR without depth info
• Only support user panning, rotation and zoom
• QuickTime VR, Surround Video
• Cylindrical image, spherical maps..

6
Introduction

• To capture panoramic images
• Using panoramic camera to get cylindrical image
• Using a lens with a large field of view (fisheye
  lens)
• Mirrored pyramids and parabolic mirrors
• Hardware-intensive methods
• Take a series regular picture or video and stitch
  them together
• Require carefully-controlled camera motion
• Produce only cylindrical image

7
Novel Algorithms

• They use 3-parameter rotational motion model
  fewer unknowns, more robust
• Instead of general 8-parameter planar perspective
  motion model
• Estimate focal length from a set of 8-parameter
  perspective registration
• Gap closing

8
Cylindrical Panoramas

• Cylindrical panorama is easy to construct
• Coordinate transformation

cylindrical image
9
Cylindrical Panorama

• With ideal pinhole camera and known f,
• Distortion horizontal lines becomes curved

10
Spherical Panorama
11
Motion Model

• Warp image to cylindrical panorama
• Ideal horizontal panning sequence
• Rotation-gtTranslation in angle
• In practice
• Vertical translation to compensate for vertical
  jitter and optical twist

On WARPED Image!
12
Motion Recovery

• Estimate incremental translation
• by minimizing the intensity error
• Taylor series expansion

13
Motion Recovery

• Minimization -gt Least square solution

14
Motion Recovery

• Large initial displacement
• Coarse to fine optimization scheme

J. R. Bergen, P. Anandan, K. J. Hanna, and R.
Hingorani. Hierarchical model-based motion
estimation
15
Motion Recovery

• Large initial displacement
• Coarse to fine optimization scheme
• Discontinuities in intensity or color between
  images being composed
• Feathering algorithm weighted by distance map

16
Limitations of Cylindrical or Spherical Panorama

• Only handle pure panning motion
• Ill-sampling at north pole and south pole cause
  big registration errors
• Require knowing the focal length f
• Estimation of focal length of lens by registering
  images is not very accurate

17
Perspective Panoramas

• Planar perspective transform between images using
  8 parameters

For example, if only translation, m2, m5 are the
unknowns
18
Perspective Panoramas

• Iteratively update transform matrix using
• Resampling image I_1 with x(ID)Mx to

19
Perspective Panoramas

• Minimize

20
Perspective Panoramas

• Least square minimization

21
Perspective Panoramas

• Works well if initial estimates of correct
  transformation are close enough
• Slow convergence
• Get stuck in local minima

22
Rotational Panoramas

• Cameras centered at the origin

Simplicity of rotation set c_xc_y0, pixel
start from image center
23
Rotational Panoramas

• Camera rotating around its center of projection
• Focal length is known and the same for all images
  V_k V_l V
• Angular velocity

24
Rotational Panoramas

• Incremental rotation matrix (Rodriguezs formula)

25
Rotational Panoramas

• is the deformation matrix as
• Jacobian of

26
Rotational Panoramas

• 3 parameters Incremental rotation vector
• Update R_k in
• Much easier and more intuitive to interactively
  adjust

27
Estimate Focal Length
1/
1/
28
Estimate Focal Length

• If fixed focal length, take average of f_0 and
  f_1
• If multiple focal length for every images, use
  median value for final estimate
• We can also update the focal length as part of
  the image registration process using least
  squares approach

29
Closing gap in a panorama

• Matching the first image and the last one
• Compute the gap angle
• Distribute the gap angle evenly across the whole
  sequence
• Modify rotations by
• Update focal length
• Only works for 1D panorama where the camera is
  continuously turning in the same direction

30
Conclusion

• Does not place constraints on how the images to
  be taken with hand held cameras
• Accurate and robust
• Estimate only 3 rotation parameters instead of 8
  parameters in general perspective transforms
• Increases accuracy, flexibility and ease of use
• Focal length estimation

31
Results
32
Photographing Long Scenes with Multi-Viewpoint
Panoramas

• http//grail.cs.washington.edu/projects/multipano/

33
Abstract

• Multi-viewpoint panoramas of long, roughly planar
  scenes
• Façades of buildings along a city street
• Panoramas are composed of relatively large
  regions of ordinary perspective
• User interactions
• to identify the dominant plane
• To draw strokes indicating various high-level
  goals
• Markov Random Field optimization

34
Introduction

• Long scene
• Hard to take photographs at one point
• Wider field of view large distortion
• Single perspective photographs are not very
  effective at conveying long scenes
• Street side in a city
• Bank of a river
• Aisle of a grocery store

35
Introduction

• Take photographs
• Walk along the other side and take handheld
  photographs at intervals of one large step
  (roughly one meter)
• Output
• a single panorama that visualizes the entire
  extent of the scene captured in the input
  photographs and resembles what a human would see
  when walking along the street

36
Contributions

• A practical approach to creating high quality,
  high-resolution, multi-viewpoint panoramas with a
  simple and casual capture method.
• A number of novel techniques, including
• An objective function that describes desirable
  properties of a multi-viewpoint panorama, and
• A novel technique for propagating user-drawn
  strokes that annotate 3D objects in the scene

37
Related Work

• Single-viewpoint panoramas
• Rotating a camera around its optical center
• Strip Panoramas
• Translating camera
• Orthographic projection along the horizontal axis
• Perspective along vertical
• Varying strip width by depth estimation of
  appearance optimization
• High-speed video camera and special setups

38
Strip Panoramas

• Exhibit distortion for scene with varying depths,
  especially if these depth variations occur across
  the vertical axis of the image
• Created from video sequences, and still images
  created from video rarely have the same quality
  as those captured by a still camera
• Low resolution, compression artifacts
• Capturing a suitable video can be cumbersome

39
Approach

• Inspired by the work of artist Michael Koller
• Multi-viewpoint panoramas of San Francisco
  streets
• Large regions of ordinary perspective photographs
• Artfully seamed together to hide the transitions
• Attractive and informative

40
multi-viewpoint panoramas

• Each object in the scene is rendered from a
  viewpoint roughly in front of it to avoid
  perspective distortion.
• The panoramas are composed of large regions of
  linear perspective seen from a viewpoint where a
  person would naturally stand (for example, a city
  block is viewed from across the street, rather
  than from some faraway viewpoint).
• Local perspective effects are evident objects
  closer to the image plane are larger than objects
  further away, and multiple vanishing points can
  be seen.
• The seams between these perspective regions do
  not draw attention that is, the image appears
  natural and continuous.

41
Properties

• A dominant plane in the scene
• Not attempt to
• Create multi-viewpoint panoramas that turn around
  street corners
• show all four sides of a building

42
System Overview

• Pre-processing
• Takes the source images
• Removes radial distortion
• Recovers the camera projection matrices
• Compensates for exposure variation

• Panorama Surface
• Defines the picture surface
• Source photographs are then projected
• onto this surface

• Composition
• Selects a viewpoint for each pixel in the output
  panorama
• Interactively refine by drawing strokes

43
Capture Images

• Use a digital SLR camera with auto-focus and
  manually control the exposure to avoid large
  exposure shifts
• Use a fisheye lens to insure a wide field of view
  for some data

44
Pre-Processing

• Recover projection matrices of each camera so
  that we can later project the source images onto
  a picture surface
• Use structure-from-motion system Hartley and
  Zisserman 2004 built by Snavely et al. 2006
• Using sift for keypoint detection and matching
• Bundle adjustment

http//phototour.cs.washington.edu/bundler/
45
Exposure Compensation

• Adjust the exposure of the various photographs so
  that they match better in overlapping regions
• Recover the radiometric response function of each
  photograph Mitsunaga and Nayar 1999
• Simpler approach

46
Picture Surface Selection

• The picture surface should be roughly aligned
  with the dominant plane of the scene
• Extrude in Y direction

47
Picture Surface Selection

• Define the coordinate system of the recovered 3D
  scene
• Automatic fit a plane to the camera viewpoints
  using principal component analysis
• The dimension of greatest variation (the first
  principal component) is the new x-axis, and
• The dimension of least variation the new y-axis
• Interactive user selects a few of these
  projected points that lie along the desired axes
• Draw the curve in the xz plane that defines the
  picture surface

48
Picture Surface Selection

• Easy to identify for street scenes
• River bank hard to specify by drawing strokes
• The user selects clusters of scene points that
  should lie along the picture surface
• The system fits a third-degree polynomial z(x) to
  the z-coordinates of these 3D scene points as a
  function of their x-coordinates

49
Sample Picture Surface

• Project each S(I,j) on picture surface to source
  photograph

50
Average Image
the average image of all the projected sources
the average image after unwarping to straighten
the ground plane and cropping
51
Interactive Refinement

• Small drifts that can accumulate during
  structure-from-motion estimation and lead to
  ground planes that slowly curve
• The user clicks a few points along the average
  image to indicate y values of the image that
  should be warped straight
• Resample and crop

52
Viewpoint Selection

• How to choose color for each pixel on panorama
  from one of the source image I_i(p)

Determine L(p) L is the image no.
53
Viewpoint Selection

• Optimization using Markov Random Field
• Cost function

54
Viewpoint Selection

• Minimize the overall cost function for each pixel
  and each pair of neighboring pixel
• Solve using min-cut optimization
• compute the panorama at a lower resolution so
  that the MRF optimization can be computed in
  reasonable time
• create higher-resolution versions using the
  hierarchical approach described by Agarwala et
  al. 2005.
• We thus composite the final Panorama in the
  gradient domain to smooth errors across these
  seams

55
Viewpoint Selection

• Not vertical strips

56
Interactive Refinement

• The user should be able to express desired
  changes to the panorama without tedious manual
  editing of the exact seam locations.
• Three types of strokes
• View selection use certain viewpoint
• Seam suppression no seam should pass an object
• Inpainting eliminate undesirable features

57
Interactive Refinement
58
Results

• 1 hour to capture images (100) and 20 mins for
  interactions
• Not for every scene
• Suburban scenes with a range of different depths
• More results can be found at http//grail.cs.washi
  ngton.edu/projects/multipano/