Generating Seamless Stereo Mosaics from Aerial Video

1
Generating Seamless Stereo Mosaics from Aerial
Video

• Zhigang Zhu
• Allen R. Hanson, Harpal S. Bassali
• Howard Schultz, Edward M. Riseman
• Computer Vision Lab
• Computer Science Department
• University of Massachusetts at Amherst
• zhu_at_cs.umass.edu
• http//www.cs.umass.edu/zhu

2
Introduction

• Objectives
• Develop methods to automatically generate
  geo-referenced stereo mosaics from video
  sequences
• Definitions
• Free Mosaic composite of video sequence by
  registering overlapping frames
• subject to drift relative to the terrain
• constrains pure rotation or planar scenes
• Geo-Mosaic use 3D instrumentation to constrain
  mosaic to world coordinates
• Stereo Mosaics a pair of mosaics from a single
  camera
• seamless under motion parallax
• preserve 3D information
• can be viewed in 3D directly

3
Important Issues in 3D Video Mosaicing

• Representation
• compact representation for a large-scale 3D scene
• orthogonal (DEM), perspective (mosaic),
  parallel-perspective
• Computation
• how expensive are the computations of the
  algorithms?
• Goal affordable, efficient and robust mosaicing
• Accuracy
• accurate for 3D viewing and 3D reconstruction ?

4
Geometry, Representation and Properties

• Sensor motion is pure translation

5
Re-Organizing the images.
6
Recovering Depth from Mosaics

• parallel-perspective stereo mosaics
• Depth accuracy independent of depth

7
Stereo mosaics of Amazon rain forest

• 166-frame telephoto video sequence -gt 7056944
  mosaics

8
Stereo viewing

• Red Right view Blue/Green Left view

9
Computation how expensive in in real world
application?

• Pro-processing
• Arbitrary motion other than a 1D translation / 3D
  translation
• Camera orientation estimation and image
  rectification
• Mosaicing
• Image sequence is not dense enough for seamless
  mosaics
• how to generate parallel-perspective projection ?
• 3D recovery
• how expensive is the match in stereo mosaics?
• Baseline and epipolar geometry of stereo mosaics
• Post-processing
• parallel-perspective 3D mosaics to an orthogonal
  DEM

10
Step 1. Pro-ProcessingMotion estimation
rectification

• Camera pose estimation using navigation
  instrumentation and bundle adjustment
• only sparse tie points widely distributed in the
  two images are needed
• Image rectification
• transformation on two narrow slices in each frame

11
Step 2. 3D Mosaicingseamless mosaicing with
motion parallax
In a multi-perspective projection mosaic, each
sub-image is full perspective, but sub-images
from different frames will have different
viewpoints. This may cause seams in the mosaic
due to motion parallax.

• Geometric Seams -
• Clearly Visible to Human Eyes, especially along
  depth boundaries
• Introduce Error in Height Estimation

12
PRISM parallel ray interpolation for stereo
mosaicing
Interpolated view
Mosaic coordinates
- Take a slice of certain width from each frame -
Perform local registration between the
overlapping slices - Generate parallel
interpolated views between two known views -
Re-project the point back to the mosaic
13
Comparison 2D mosaic 3D mosaic
14
Step 3. 3D reconstructionepipolar geometry of
stereo mosaics

• Epipolar curve
• 1D search
• Near horizontal line
• Coverged pair
• small search region

• epipolar curve

15
Epipolar Geometry in Real Stereo mosaics
Left Mosaic
16
Epipolar Geometry in Real Stereo mosaics
Right Mosaic
17
Epipolar Geometry in Real Stereo mosaics
Depth Map Brighter is higher elevation
18
Step 4. Post-Processing from 3D mosaic to DEM
Just a transformation !! (X,Y,Z) world
coordinates (xl,yl) left mosaic
coordinates (xr,yr) right mosaic coordinates H
a reference height (depth) F focal length of the
camera dy distance between left and right
slits by adaptive baseline Dy yr - yl
19
3D Rendering Result
20
Motion Refinement for Geo-Mosaic- when geo-data
is not accurate
Ground Truth
Geo Meaursement - absolute error
EKF Esimation
Flying path
Unconstrained -Image Match -Accumulating error
Time Update Predict from image registration
Measurement Update correct by geo-data
Motion parameter A Warp matrix t
translation

• Extended Kalman Filter (EKF) approach

21
Geo-Reference Mosaicsframe-by frame mosaicing
22
Geo-Reference Mosaicsglobal warping from free
mosaic
23
Methods Summarydistribute the computations in
four steps

• Pro-processing
• Motion estimation sparse tie points distirbuted
  in entire frames
• Rectification and Mosaicing (PRISM)
• Process two narrow slices
• 3D recovery (Terrest)
• stereo match only in two mosaics
• Post-processing
• just a coordinate transformation

24
Accuracy of 3D from Stereo Mosaics

• Adaptive baselines and fixed disparity - uniform
  depth resolution
• Ray interpolation between two successive views is
  similar to image rectification
• 3D recovery accuracy is comparable to that of a
  perspective stereo with an optimal baseline

25
Next Steps

• Camera calibration and bundle adjustments
• Geometric and Photometric Seamlessness
• Using Structure Information and video photo
  matching
• Error analysis in geo-referenced mosaic and 3D
  reconstruction using ground truth