Artificial VisionBased TeleOperation for Lunar Exploration

1
Artificial Vision-Based Tele-Operation for Lunar
Exploration

• Students
• Aaron Roney, Albert Soto, Brian Kuehner, David
  Taylor, Bonnie Stern, Nicholas Logan, Stephanie
  Herd

Project Mentor Dr. Giovanni Giardini
Project Advisor Prof. Tamás Kalmár-Nagy
NASA JSC Mentors Dr. Bob Savely Dr. Mike Goza
2
Project Members
3
Outline

• Motivation and Objectives
• Ego-Motion Theory
• Code Flow
• Calibration and Rectification
• Hardware
• Testing Results
• Future Work

4
Motivation

• Lunar surface exploration
• Human perspective
• In safety
• With low risk
• 3D environment reconstruction
• Self location with artificial vision system

5
Objectives
Visual Feedback System for Tele-Operations

• Vision System
• Ego-Motion estimation
• Environment reconstruction

• Tele-Operation System
• Remote control mobile unit

• Hardware and Mechanical
• Implementation

6
(No Transcript)
7
Ego-Motion Theory
8
3D Reconstruction Theory

• It is impossible to compute the 3D coordinates of
  an object with a single image
• Solution Stereo Cameras
• Disparity computation
• 3D reconstruction

Image
9
Environment Reconstruction

• Disparity map computation
• Given 2 images, it is a collection of pixel
  disparities
• Point distances can be calculated from
  disparities
• Environment can be reconstructed from disparity
  map

Left Image
Right Image
Disparity Map
10
Ego-Motion Estimation

• Main goal Evaluate the motion (translation and
  rotation) of the vehicle from sequences of images

Optical Flow Example

• Optical Flow is related to vehicle movement
  through the Perspective Projection Equation

Perspective Projection Equation

• Solving will give change in position of the
  vehicle

• Least Square solution

11
Code Flow
12
Image Processing Code
Calibration Parameters
Logitech QuickCam Deluxe
Acquire Images
Rectify Images
Ego-Motion Estimation
Sony VAIO - Pentium 4
Wireless 802.11 Network
Ground Station
13
Mobile Unit Detailed Code
Calibration Parameters
Acquire Image
Rectify Images
T 0.15 sec
T 0.5 sec
Apply Distortion Coefficient to Image Matrix
Image Parameters Gray Scale (640x480)
Snapshot Image Matrix
Rectified Image Matrix
Save Image
Wireless 802.11 Network
Ground Station
Ego-Motion Estimation
14
Ego-Motion Estimation Overview
Discard All non-Identical Points in All images
Find Features in Right Image
Right Image
Track Right Image Features in Left Image
Find Features in Left Image
Left Image
Displacement Vector (X, Y, Z, X-Rot, Y-Rot,
Z-Rot)
Find Features in New Right Image
Track Right Image Features in New Right Image
New Right Image
New Left Image
Image Feature Matrix
Find Features in New Left Image
Track Right Image Features in New Left Image
Calibration Parameters
Wireless 802.11 Network
T 3 sec
15
Calibration and Rectification
16
Calibration and Rectification

• Calibration Utilizes Matlab tools to determine
  image distortion associated with the camera

• Rectification Removes the distortion in the
  images

17
Hardware
18
Hardware
Mobile Unit
Base Station
Web Cameras
TROPOS Router
Operator Computer
Laptop
Wireless 802.11
Command Computer
Mobile Unit
Linksys Router
Wireless 802.11
19
Improvements Implemented in the System

• Improved robustness of the software
• Implemented a menu driven system for the operator
  using Matlabs network handling protocol
• Allowed pictures to be taken
• Run Ego-motion
• Sending all the results to the operator
• Graphic displaying of optical flow
• Reduced crashing
• Achieved greater mobile unit control

20
Mobile Unit
Vehicle Courtesy of Prof. Dezhen Song

• Camera support system
• 3-DOF mechanical neck
• Panoramic rotation
• Tilt rotation
• Telescopic capability

• Controlled height and baseline length

21
Testing Result
22
Test Environment
Light to simulate solar exposure
Black background to eliminate background features
Walls to eliminate stray light and side shadows
Lunar Environment
Measured displacements
23
Test Setup

• 25 pictures taken from each location (0, 5, 10
  and 15 cm) in the Z direction (perpendicular to
  camera focal plane), unidirectional movement
• Set 1 25 images located at Z0
• Set 2 25 images located at Z5
• Set 3 25 images located at Z10
• Set 4 25 images located at Z15
• The distances are measured using a tape measure
• The cameras are mounted using a semi ridged
  fixture

24
Determining the Number of Features
Results for 5 cm displacement Used all 100
images Compared each set to the previous

• But the accuracy of the results decrease

• The standard deviation decreases with the more
  features

100 Features were selected
25
Ego-Motion Example
Optical Flow Left Image
Optical Flow Right Image
26
Problems

• Images were not rectified
• Possible motion of cameras between images
• No image filtering
• Camera mounting is misaligned
• Images acquired from the right camera appear
  blurry

27
Conclusions andFuture Work

• Demonstrated
• Ego-motion estimation
• Environment Reconstruction
• Vehicle control and movement
• System integration
• Future Developments
• Filtering and improving results
• Increase the robustness of the vision system
• Create a visual 3D environment map

28
Acknowledgements

• Thanks to
• Prof. Tamás Kalmár-Nagy
• Dr. Giovanni Giardini
• Prof. Dezhen Song
• Change Young Kim
• Magda Lagoudas
• Tarek Elgohary
• Pedro Davalos