Hand Tracking for Virtual Object Manipulation

1
Hand Tracking forVirtual Object Manipulation

• Thibaut Weise

Project Supervisor Second Marker
Professor Guang-Zhong Yang Dr Duncan Gillies
2
Motivation

• Virtual and Augmented Environments
• Robotic Control
• Surgery
• Space, Deep Sea

3
State of the Art

• EM-Tracking
• Acoustic Tracking
• Data Glove
• Optical

4
Optical Approaches

• Appearance-based Approach
• Model-based Approach
• Reconstruct 3D Hand Model
• Two methods
• Minimize Model against Features
• Reconstruct from Features

5
Requirements

• Focus Grab Gesture
• ?
• Thumb and Index Finger
• Real-time
• Responsive
• Accurate

6
System Overview
Calibration
Marker-based
Hand Tracking
Marker-less
Virtual Environment
7
Reconstruction

• Triangulation
• Requires Mapping between Pixels and 3D World
  Coordinates

8
Calibration

• Optimize known World Coordinates against detected
  Image Coordinates
• Planar Approach gt Chessboard
• OpenCV Implementation
• Stereo Optimization

9
Calibration Accuracy

• Corner Detector Accuracy
• Why sub-pixel accurate?

10
X-Corner Detector

• Solution Improved X-Corner Detector

Saddle Point
Product of Eigenvalues minimised ?1?2
fxxfyy-fxy2 Gradient zero 0 fx s fxx t
fxy 0 fy s fxy t fyy Recalculate using
weighted derivatives of neighbourhood
11
Corner Detector Comparison

• Solution Improved X-Corner Detector

12
Calibration Automation

• Improved Chessboard Detector

13
Calibration Tool
14
Hand Tracking

• Marker-based Marker-less

15
Marker Tracking

• 3 Markers
• Thumb, Index Finger and Hand
• Uses Improved X-Corner Detector
• Fast
• Accurate
• Simple

16
Hand Reconstruction

• Find all potential Markers in Images
• Match Points using Epipolar Constraints

17
Hand Reconstruction

• Reconstruct all possible hand configurations
• Different markers
• Validate hand constraints

18
Hand Reconstruction

• Use best match based on temporal information
• Minimise Euclidean distance to predicted position

19
Hand Reconstruction

• Initialisation based on Orientation Heuristic

20
Marker-less Tracking

• 2 Fingertips
• Reconstruct Hand Position
• Hand Localisation
• Skin Colour Detection

21
Hand Localisation

• Background Subtraction
• -
• Skin Background

22
Hand Localisation

• Find Hand Contour

23
Hand Reconstruction

• Find Peaks
• Calculate k-curvature
• C(i)-C(i-k)TC(ik)-C(i)
• Peaks where angle gt 0o and lt 90o

24
Hand Reconstruction

• Match peaks using Epipolar Constraints ? find
  possible Finger Tips
• Calculate Finger Axis

25
Hand Reconstruction

• Reconstruct hand position
• Validate hand constraints
• Use best hand based on temporal information

26
Evaluation

• Marker-based Marker-less

27
Virtual Environment

• Virtual Tower of Hanoi
• Virtual Hand gt Grabber
• Fixed Dimensions
• Grab and Push Objects
• Perception
• Colour Coding
• Shadow

28
Implementation
Application Thread
Settings
Video / Tracking Thread
Draw / Settings
Hand Update
Virtual Environment
29
Demo
30
Conclusion

• Accurate semi-automatic Calibration
• Marker-based Tracking
• Marker-less Tracking
• Virtual Environment

31
Future Work

• Marker-based Tracking
• Handle Occlusion
• Marker-less Tracking
• Refine Skin Colour Detection
• Use Gradient

32
Questions
?

• Who wants to try?