Digital Imaging and Remote Sensing Laboratory

1
Automatic Tie-Point and Wire-frame Generation
From Oblique Aerial Imagery
Seth Weith-Glushko Advisor Carl
Salvaggio Research Proposal November 7, 2003
Digital Imaging and Remote Sensing Laboratory
2
Table of Contents

• The Problem
• Specific Aims
• Proposed Solution
• A description of the individual transforms and
  algorithms used to make up the proposed
  algorithms
• Methods
• Timetable
• Budget
• References

Digital Imaging and Remote Sensing Laboratory
3
The Problem

• Using photogrammetric techniques, tie matching
  algorithms and bundle adjustment algorithms, it
  is possible to create a 3D model using a series
  of images of an object
• These images must be around an axis (for objects)
  or ortho-rectified (for landscapes) for the
  algorithms to work
• We want to be able to use oblique imagery for
  input into the algorithms
• By using oblique imagery, more pixels are
  available to describe the features of a 3D object
  than would be in an ortho-rectified image

Digital Imaging and Remote Sensing Laboratory
4
Specific Aims

• Current algorithms only work on ortho-graphic
  (for tie-points) and axial imagery (for
  wire-frame generation)
• The goal is to draw from both photogrammetry and
  computer graphics to develop two algorithms which
  can work in unison
• A tie-point algorithm that works on oblique
  imagery
• A bundle adjustment algorithm that can use
  oblique imagery
• In the future, this algorithm would become part
  of a suite of algorithms that could generate an
  accurate 3D model of scene independent of the
  type of imagery used

Digital Imaging and Remote Sensing Laboratory
5
Proposed Solution

• The algorithm below relies heavily on the use of
  INS (Inertial Navigational System) data
• The input would be a series of oblique images
  made around a common area
• The output would be a matrix of matching points
  and a file using a common 3D format

Input A series of oblique images around a common
area
Image Processing
Ortho-rectification Transform

• Histogram processing

Converts an oblique image into an ortho-rectified
image
Digital Imaging and Remote Sensing Laboratory
6
Proposed Solution
Definition of Points
Point matching algorithm
Geometric Transform
Use the Laplacian of Gaussian (LoG) filter to
find points of high frequency (i.e. edges)
Use point distance comparison, point scale
comparison and point angle comparison to find
matching points
Using matched points, generate a geometric
transformation and use registration as indicator
of goodness of points
Digital Imaging and Remote Sensing Laboratory
7
Proposed Solution
Inverse Geometric Transform
Inverse Orthorectification Transform
Bundle Adjustment Algorithm
Perform an inverse geometric transformation using
previous transformation matrix
Convert the orthorectified image back into its
original oblique form
Using pairs of matched points, define points in
3D space
Output Matrix containing matched pairs between
images
Digital Imaging and Remote Sensing Laboratory
8
Proposed Solution
Output 3D wire-frame mesh of a scene
Interface with 3D System
Using software libraries and defined 3D points,
generate an output file
Digital Imaging and Remote Sensing Laboratory
9
Ortho-rectification Transform

• The ortho-rectification transform converts an
  oblique image into an ortho-rectified (flat)
  image by means of a linear equation
• There are four unknowns. We can use the fiducial
  points of an image as the four points we need to
  solve for the constants

Digital Imaging and Remote Sensing Laboratory
10
Image Processing

• Image processing needs to be performed because
  images with dissimilar digital count affect the
  point generation operator
• Histogram matching is performed to minimize this
  dissimilarity

Images courtesy C. Salvaggio
Digital Imaging and Remote Sensing Laboratory
11
Definition of Points

• To define points the Laplacian of Gaussian
  operator is used
• Walli found that if there is an edge in an image,
  a thresholded filtered image would show a point
  at that edge

Images courtesy K. Walli
Digital Imaging and Remote Sensing Laboratory
12
Point Matching Algorithms

• To match defined points, three algorithms are
  primarily used pixel distance match, scale
  match, angle match
• Another matching criteria is LoG maxima similarity

Digital Imaging and Remote Sensing Laboratory
13
Pixel Distance Match

• This algorithm works by comparing distances
  between pixels in a matrix

3
1 2 3 4
2
3
1 2 3 4
2
1
1
1 2 3 4
1 2 3 4
Same Elements 2 3 2
Distance
Digital Imaging and Remote Sensing Laboratory
14
Scale Match

• This algorithm works by comparing ratios of
  distances between pixels in a matrix

3
1 2 3 4
2
1 2 3 4
3
2
1
1
0 1 2 3 4 5
6 7
0 1 2 3 4
Distance Compare Ratios Ratios of
distances from like points is equal!
Digital Imaging and Remote Sensing Laboratory
15
Angle Match

• This algorithm works by comparing the angle
  formed by the triangle of 3 pixels in a matrix

3
2
1 2 3 4
3
1 2 3 4
2
1
1
1 2 3 4
1 2 3 4
Point Set 1
3
2
Point Set 2
Vertice 1
Digital Imaging and Remote Sensing Laboratory
16
Geometric Transformation

• Using the matched pixels, solutions to the affine
  polynomial problem are found
• Using the affine polynomial, one ortho-rectified
  image is geometrically transformed so that it can
  register with another ortho-rectified image.
• Quality metrics are performed to determine
  whether the registration is good. Hence, the
  matched points are good.

Digital Imaging and Remote Sensing Laboratory
17
Inverse Transformations

• The matched points are put through an inverse
  geometric transformation and an inverse
  ortho-rectification transform to return the
  points to their original oblique pixel form
• A matrix of matched points is output

Digital Imaging and Remote Sensing Laboratory
18
Bundle Adjustment Algorithm

• Bundle adjustment algorithms allow the mapping of
  2D points into 3D space using more than two
  images around a common point
• The algorithm estimates the underlying plane
  geometry of a scene

Images courtesy M. Pollefeys
Digital Imaging and Remote Sensing Laboratory
19
3D Library Interfacing

• Using these 3D points generated from the bundle
  adjustment algorithm, a triangle mesh is created
  which forms the structure of the wire-frame scene
• Also, a texture map is generated from bundle
  adjustment. This map is overlaid on the mesh
• The full model is saved to a generic 3D format

Digital Imaging and Remote Sensing Laboratory
20
Methods

• Using a programming environment, engineering code
  will be developed to determine the feasibility of
  this algorithm
• If it is feasible, quality metrics will be
  applied to determine effectiveness
• Visual analysis, absolute mean variance, and
  deviation from a polynomial model (RMSDE) can be
  used to check tie-point generation
• Visual analysis and post-photogrammetric analysis
  can be used to check wire-frame generation

Digital Imaging and Remote Sensing Laboratory
21
Timetable

• September 1, 2003 November 15, 2003
• Search for previous research, background
  knowledge
• November 15, 2003 April 1, 2004
• Development of algorithm and engineering code
• April 1, 2004 May 15, 2004
• Complete paper, poster and presentation

Digital Imaging and Remote Sensing Laboratory
22
Budget

• No money will be required for this project as the
  investigator has all of the resources he
  currently requires
• 2 credits will be required for both Winter and
  Spring Quarters
• Due to the nature of the contract, most of the
  work performed will be done on a pay basis
• Flexibility in the experimenters schedule was
  required

Digital Imaging and Remote Sensing Laboratory
23
References

• Honkavaara, Eija and Anton Hogholen. Automatic
  Tie Point Extraction in Aerial Triangulation.
  International Society for Photogrammetry and
  Remote Sensing, 16th Congress, Vienna, July 1996.
  337-342.
• Moffitt, Francis H. and Edward M. Mikhail.
  Photogrammetry. 3rd Ed. New York Harper and Row,
  1980.
• Pollefeys, M. 3D Geometry from Images. 15 Oct.
  2003. lthttp//www.esat.kuleuven.ac.be/pollefey/tu
  torial/tutorialECCV.htmlgt
• Walli, Karl C. Multisensor Image Registration
  Utilizing the LOG Filter and FWT. Diss.
  Rochester Institute of Technology, 2003.
• Wolf, Paul R. Elements of Photogrammetry. 2nd Ed.
  New York McGraw-Hill, 1983.

Digital Imaging and Remote Sensing Laboratory
24
Questions
Are there any?
Digital Imaging and Remote Sensing Laboratory