Vision Analysis for UAV Flight Control and Obstacle Avoidance

1
Vision Analysis for UAV Flight Control and
Obstacle Avoidance
Zhihai (Henry) He Dept. of Electrical and
Computer Engineering University of Missouri,
Columbia MO Ram Venkataraman Iyer
(Presenter) Department of Mathematics and
Statistics, Texas Tech University, Lubbock,
TX Phillip R. Chandler U.S. Air Force Research
Laboratory, Wright- Patterson Air Force Base, Ohio
2
Introduction
Need for vision-based flight control

• Currently, a UAV rely on GPS and gyroscope to
  measure its location, linear velocity and
  angular velocity for flight control.
• Videos, captured from on-board cameras, provide
  an additional source of information for control.
• GPS and gyroscope have measurement errors. They
  are not accurate enough for UAV flying at low
  altitudes, especially in urban areas.
• Computer vision analysis of camera videos will
  allow us to accurately estimate the linear and
  angular velocity of vehicles.

3
Vision-based Flight Control
Major Steps

• Motion Field Estimation - Estimate the motion
  field and find the pixel correspondence between
  consecutive video frames.
• Camera View Analysis - Based on camera view
  geometry analysis, determine the linear velocity
  and angular velocity of camera.

4
Vision-based Flight Control

• Existing methods
• Feature tracking
• Structure from motion

• Major problems
• High computational complexity
• Constrained environments

• Assume the image / video has some desired
  features, and build the algorithms on these
  assumptions, or tune the model parameters and
  thresholds based the image content.
• The algorithms work in constrained environments
  with assumed characteristics. They may fail in
  other types of navigation environments.

5
Vision-based Flight Control
Motion Field Analysis

• Needs to be accurate
• Need to robust in different navigation
  environments (city, rural areas, mountains,
  forest, etc)
• Need to deal with uncertainty and noise
• Needs to have low complexity for real-time
  control

6
Vision-based Flight Control
Block-based motion field analysis
-low complexity

• Global scene modeling and tracking is highly
  computation-intensive.
• Typically, in motion analysis, partition the
  image into blocks and determine them motion for
  each block based on local information.
• Uncertainty is caused by lack of sufficient
  information and image noise (or interference,
  like local moving objects).
• Uncertainty / ambiguity is the major challenge in
  accurate and robust motion field analysis.

Uncertainty
7
Uncertainty
Based on local information only, which block is
the best match?
?
?
8
Hierarchical Uncertainty Management
Structural Blocks
Texture Blocks

With different reliability levels
9
Vision-based UAV Navigation
Proposed Framework
Video frames
Non-Structural Blocks
Structural Blocks
Velocity Orientation
10
Block Classification
Structural blocks have significant amount energy
in the low-to-medium frequency bands

• Input A block
• DCT (discrete cosine transform)
• Compute ratio energy
• Rank the blocks according to their energy ratios,
  and select the top 15 as the structural blocks

20, No DC coefficients
11
Block Classification
Structural blocks
12
Motion Estimation
For a block B in frame It
A
B
Find its best match A in the previous frame It-1

Major Steps

• Extract features from both blocks
• Compute the distance between features.
• The block has the minimum distance is the best
  match, and the corresponding spatial displacement
  is the motion vector.

Distance (or similarity) Metric
Should be invariant under camera motion
translation, rotation, zoom.
13
Motion Estimation
Conventional Metric SAD Sum of Absolute
Difference
A
B
-
Invariant only under translational motion
To handle camera rotation and zoom, we introduce
one additional feature intensity profile. The
intensity profile describes the intensity
distribution in an image region.
14
Distance Metric
A
Average pixel intensity on circles around the
point
B
Zoom invariant
Rotation Invariant!
15
Distance Metric
Zoom in/out scaling factor
Distance Measure
Stretch / compress
This distance metric is Rotation and Scale
Invariant!
16
Distance Metric
Combine two metrics for robustness
Translation
Rotation / Scaling
w weighting factor. Adjusted according to the
magnitude of angular velocity.
Motion Search
Find the best match to minimize the distance
metric
The neighborhood of block B in the previous frame.
The motion vector is difference vector between
the center positions of blocks B and A.
17
Reliability-Based Motion Field Analysis

• A block may have multiple best matches (multiple
  minima).
• Because of image noise, the true motion vector
  may NOT have the minimum distance.
• More best matches gt less reliable motion
  estimation.
• We allow a block has multiple best matches in
  motion estimation.
• Assign a reliability value to each block.

Motion search path
Minimum
Reliable
Unreliable
18
Reliability-Based Motion Field Analysis
The image is equally partitioned into N blocks.
After classification, the structural blocks are
For each structural block
We find L best motion vectors
Motion vector
Distance
Select those motion vectors near the bottom
19
Reliability-Based Motion Field Analysis
Find the average the those motion vectors near
the bottom and consider it as the true motion
vector
Reliability Measure
Near 1.0
Near 0.0
20
Reliability-Based Motion Field Analysis
In camera motion estimation, the reliability will
act as a weighting factor. Those motion vectors
with lower reliability values will have less
influence in determining the camera motion.
21
Reliability-Based Motion Field Analysis
22
Angular Velocity Estimation
Refine angular motion estimation Range map for
scene objects
(X, Y, Z)
(x, y)
Camera frame
Inertial frame
23
Angular Velocity Estimation
Camera view geometry analysis
Angular velocity
Unknown
Range
Known
Motion vector in (images)
Z depends on the pixel position (x, y)
24
Angular Velocity Estimation
Combine them and get rid of Z
All known
25
Angular Velocity Estimation
A L-MSE problem
26
Range Map Estimation
For non-structure blocks, its true motion must
satisfy the view geometry equation
The only unknown valuable is Z.
Linear constraint! The true motion just be on
this line.
1/z
27
Range Map Estimation
Motion vectors for this block
If motion vector is
, then the range is given by
The fitting error is
The true motion vector should have the minimum
error
True range
True motion vector
28
Performance Evaluation

• Multi-UAV simulator
• Flight test videos

Multi-UAV simulator

• Have access to all needed information the camera
  location, linear velocity, orientation, which
  point object in the scene is mapped to which
  pixel, the range of each object.
• Inputs video, linear velocity, orientation
  noise
• Outputs Check if the vision analysis reduces the
  noise and refine the angular velocity estimation.
  Also verify if the range estimation is accurate
  or not.

29
Performance Evaluation
Flight test videos

• Have UAV GPS data, rough orientation, and target
  GPS data.
• Estimate the range of targets, and check if the
  estimation is accurate or not.

Performance Analysis

• Study how much the vision information could
  improve the angular measurements.
• Study the impact of noise in linear velocity on
  the estimation accuracy.

30
Performance Evaluation
Estimating the location (XY) and orientation
(pitch, yaw, roll)
31
Conclusion and Future Work

• We developed an robust framework for motion field
  analysis, camera view analysis, and vision-based
  flight control.
• In our future work, we shall
• Perform analytical measurement noise analysis
  (impact of noise) and system performance
  characterization.
• Utilizing temporal correlation for fast and
  robust flight control.
• Deal with local moving objects (vehicles and
  persons)
• High-level reasoning decision making handle
  strange navigation environments (stop sign, power
  lines, electrical poles, a strange wall)