Mobile Motion Tracking using Onboard Camera

1
Mobile Motion Tracking using Onboard Camera

• Supervisor Prof. LYU, Rung Tsong Michael
• Prepared by Lam Man Kit
• Wong Yuk Man

2
Agenda

• Motivation Objective
• Previous Work
• Improvement New components
• Sample Applications
• Conclusion
• QA

3
Motivation

• Rapid increase in the use of camera-phone
• Camera-phone can perform image processing tasks
  on the device itself
• Enhance human-computer interaction by mobile
  phone

4
Objective

• Implement real-time motion tracking on Symbian
  phone, without requiring additional hardware
• Translational motion tracking
• Rotational motion tracking
• To develop a tracking engine for other mobile
  devices developers to use

5
Previous Work

• A translational motion tracking engine
• Blocking matching
• SEA (Successive Elimination Algorithm)
• PPNM
• PDE (Partial Distortion Elimination)
• Adaptive Window Search
• Spiral Scan
• Feature selection
• A translational motion tracking engine

6
Block Matching Algorithm
7
Block Matching Algorithm

• Evaluate the "goodness" of a match by Sum of
  Absolute Difference
• Select the candidate block with the lowest error

8
Feature Selection

• A good feature block
• High variance -gt complex block
• Not in a repeated pattern
• If a good feature block is found, the performance
  of the motion tracking is improved
• We have made improvements to our existing
  algorithm

9
Old Feature Selection

• Divide the current frame into squares
• Apply Feature Selection Algorithm to each squares
• However, the best feature block may appear
  between two squares

10
New Feature Selection

• More blocks are sampled
• Sample 26x26 variances of blocks with size 15x15
  in a 54x54 window
• Search in spiral way and stop searching when
  selection criteria are matched
• Prefer to find a feature block in the center
• Prevent out of bound problem (block appear out of
  screen)

11
New Feature Selection

• If a feature block is found on the edge, the
  following will happen
• The tracking algorithm will fail to find the
  exact match
• Solution
• Apply additional constraint

12
New Feature selection

• In order to find a feature block that is not on
  the edge, we apply a checking algorithm
• Important difference in all directions gt
  interest point

13
New Feature Selection

• Now our newest feature selection algorithm
  becomes
• Find a block with a large variance -gt which
  indicate the complexity of the block
• Check if the block is on the edge or not by
  calculating the SAD between the candidate block
  and its 4 neighbors
• If either one of the SAD is small -gt the block is
  on edge -gt reject
• Else the block is not on edge

14
SSD is used instead of SAD

• SSD Sum of Squared Difference
• Last term, SAD is used as matching criteria
• SAD is commonly used because of its lower
  complexity (faster)
• But we chose to use SSD finally

15
SSD is used instead of SAD

• Experiment shows that SSD has better performance
  in accuracy and
• Algorithm with SSD run as fast as that with SAD
• Reason (of why as fast as SAD)
• Elimination effects from SEA, PPNM PDE become
  large when SSD is used Compensate the higher
  complexity of SSD

16
SSD is used instead of SAD

• SEA and PPNM lower bound is adjusted using the
  inequality (12) proposed in the following paper

J.J. Francis and G. de Jager. A Sum Square Error
based Successive Elimination Algorithm for Block
Motion Estimation (2002) 
17
Experimental Result on Symbian phone

• Frame rate
• Maximum frame rate supported by Nokia 6600 is
  about 14 frames/sec
• Running our algorithm (1/0.007 frames/sec) once
  only for each frame do not slow down the
  displaying frame rate, in other word, the frame
  rate can still be 14 frames/sec, our algorithm
  doesnt lag the display

18
Rotation Tracking Engine

• Observation and Motivation

• As we rotate the phone, the object can still be
  tracked correctly (center of object roughly
  equals center of green box)

19
Rotation Tracking Engine

• Our approach
• If two blocks are tracked at the same time, angle
  of line connecting the two blocks reflect the
  tiling angle of the phone

Two-block approach
20
Rotation Tracking Engine

• Another approach
• Track one block by simple motion tracking engine,
  then find which rotation gives the best match
• Fail if tracking object is the same for different
  angle

One-block approach
21
Rotation Tracking Engine

• Modification of motion tracking algorithm to suit
  rotation tracking
• Linear Adaptive method used in translation motion
  tracking is not used here because phones motion
  can be both linear motion and circular motion

22
Rotation Tracking Engine

• What to predict?
• Given the coordinates of the tracking blocks in
  the last 2 previous frames
• Predict the coordinates of the blocks in the next
  frame

Line L1

• A simplified mathematic problem (the following
  assumptions are approximately correct)
• All three lines pass through the circles
  center
• End points of the three lines lie on the circle

Line L2
?
?
Line L3
23
Rotation Tracking Engine

• Solution
• Angle between L2 and L3 ?
• ? Tan-1(slope(L2)) Tan-1(slope(L3))
• Coordinates of the next tracking block (xL1, yL1)
  are calculated by multiplying the column matrix
  of coordinates of the previous block with a
  rotation matrix

24
Rotation Tracking Engine

• Solution
• The prediction of the position of the next
  tracking block should also take the translational
  movement into account
• Horizontal displacement Tx
• Tx ( xL21 xL22 xL31 - xL32 )/2
• Vertical displacement Ty
• Ty ( yL21 yL22 yL31 - yL32 )/2
• Coordinates of the next tracking block (xL1, yL1)
  is calculated by

25
Rotation Tracking Engine

• A simple drawing to show the detected rotation
  angle of the phone

26
Rotation Tracking Engine

• Reducing error by using level
• Apply threshold on output, increase/decrease one
  level only when change is large
• To give less sensitive but more desirable output
  for game
• e.g. skiing game skier face only to 7 directions
• Reduce difficulty, increase reliability
• A small rotation or a small detection error will
  not increase/decrease one level

27
Conditions to be a good background for both
engines

• Objective
• Max. performance measurement
• Increase usability
• Condition
• Feature selection can always find a good feature
  point
• Within certain distance
• No repeat pattern
• Very distinct pattern
• Pattern is nearly the same when rotated. E.g.
  Circle which is good for rotation detection

28
Virtual Mouse

• An application that make full use of the
  translational motion detection engine
• Remote control the mouse of PC by Symbian phone
  using motion tracking
• Advantages
• It allows input in all directions
• It provides high levels of control
• Joystick can still be used as rough moving while
  camera input can be used as fine translation

29
Virtual Mouse

• Server
• In server side (Windows), the Bluetooth is
  configured to provide RFComm Service and server
  regards the Bluetooth transmission port as Comm.
  Port
• Server receives message from that Comm. Port
• Call function in MouseAction.h to make the mouse
  move and trigger mouse click event

30
Virtual Mouse

• Client
• Search for Bluetooth device nearby
• Connect to the selected Bluetooth device
• Every time motion tracking algorithm finishes,
  results are sent to server (12 times/sec)
• Joystick and keypad can also be used to control
  mouse in PC (Multi-button mouse)

31
Car Racing Game

• A Car Racing Game is developed using the motion
  tracking engine.
• Game lags. It is because
• The engine takes time to find the motion vector
• CPU speed of the Symbian phones are low
• The game engine thread uses most of the CPU power
  while the thread for updating the screen cannot
  proceed
• Solutions
• Double Buffering
• Direct Screen Access
• Single Thread

32
Double Buffered Area

• Double-buffering two complete color buffers
• One is displayed while the other is calculating
  next scene to be drawn,
• When the drawing of the next frame is completed,
  the content of the back buffer is copied to the
  front screen

Computing Frame 2
Front Screen
Show Frame 2
Frame 2 ready
Buffer 1
Buffer 2
copy
33
Direct Screen Access

• In Symbian, traditional drawing requires the help
  of Window Server
• Overhead in context switching
• Lower speed
• Access the screen directly in 3 ways
• Creating and using CfbsScreenDevice
• Accessing screen memory directly.
• Using CdirectScreenAccess.
• Using DSA can bypass the Window Server
• Get rid of context switching
• Faster

34
Traditional Graphic Programming
35
Direct Screen Access
36
Single Thread

• Prevent context switching
• No need to do synchronization

37
Car Racing Game

• With the use of efficient graphic programming
  algorithms, the tracking engine will not affect
  the performance of the game
• It shows that we can use the engine to develop
  other applications without performance degradation

38
Skiing Game

• Sample program from the book Developing Series
  60 Applications A Guide for Symbian Os C
  Developers
• Based on this game, we plug in our rotation
  tracking engine into it
• The game becomes more interactive
• Rotate the phone to control the angle of the skier

39
Skiing Game
40
Skiing Game

• The process of using the engine is very simple
• Create an instance of the tracking engine
• Call the function of the engine to find the
  motion vector
• Use the motion vector for the game logic

41
Skiing Game
42
Demo
43
Experimental Result on Symbian phone

• Testing Environment
• Platform
• Symbian Phone Nokia 6600
• Algorithm
• Our final (hybrid-type) Algorithm (Block matching
  algorithm featured with Adaptive Window, Spiral
  Scan, SEA, PPNM and PDE method)
• Algorithm parameter
• Block size         17 x 17 (pixels)
• Search window size     16 x 16 (pixels)
• Number of block to track in each run of algorithm
  1 block ONLY

44
Experimental Result on Symbian phone

• Testing Result (avg time to run)
• Final algorithm
• 7ms
• Final algorithm without adaptive window method
  and SEA, PPNM method
• 22ms
• Full Exhaustive Search algorithm (the most
  simplest one)
• 55ms

45
Q A