Wii3D

1
Wii3D Extending the Wiimote into 3D

• Final Honours Presentation

Principal Investigator João Lourenço Supervisor
Dr Hannah Thinyane
2
Presentation Outline

• Introduction
• Design and Implementation
• User Study
• Future Work
• Conclusion

3
Introduction

• Problem Statement
• Research Motivation and Goals

4
Introduction

• Problem Statement
• Research Motivation and Goals

5
Problem Statement

• Migration from 2D output to 3D output
• 3D TV
• Surround sound speakers
• Suggests a need for migration from 2D input to 3D
  input
• Microsoft Kinect (previously Project Natal)
• Wii3D

6
Introduction

• Problem Statement
• Research Motivation and Goals

7
Research Motivation and Goals

• Intuitive 3D Interaction Techniques are needed
• Develop 3D Interaction using Nintendo Wii Remote
• Compare Proposed interaction technique with
  existing techniques

8
Design and Implementation

• Hardware
• Software
• OS and Frameworks
• Bluetooth
• Libraries
• Point Tracker
• Gestures and Gesture Recognition
• User Interface

9
Design and Implementation

• Hardware
• Software
• OS and Frameworks
• Bluetooth
• Libraries
• Point Tracker
• Gestures and Gesture Recognition
• User Interface

10
Hardware

• 2 Nintendo Wii Remotes
• Personal Computer with Bluetooth Capabilities
• 2 WiiGloves

11
Nintendo Wii Remotes

• Layout Maximizing Volume

12
Nintendo Wii Remotes

• Layout Maximizing Efficiency

13
WiiGlove

• Circuit
• 2 Configurations
• 2 IRs per hand
• 4 IRs on one hand

14
Design and Implementation

• Hardware
• Software
• OS and Frameworks
• Bluetooth
• Libraries
• Point Tracker
• Gestures and Gesture Recognition
• User Interface

15
OS and Frameworks

• Windows 7
• Microsoft C.NET 4.0

16
Design and Implementation

• Hardware
• Software
• OS and Frameworks
• Bluetooth
• Libraries
• Point Tracker
• Gestures and Gesture Recognition
• User Interface

17
Bluetooth

• Microsoft Windows built-in Bluetooth Stack

18
Design and Implementation

• Hardware
• Software
• OS and Frameworks
• Bluetooth
• Libraries
• Point Tracker
• Gestures and Gesture Recognition
• User Interface

19
Libraries

• WiimoteLib 1.7.0.0 Wiimote Connection
• Accord.NET 2.1.1.0 Statistics
• Math.NET Iridium 2008.8.16.470 Linear algebra
• 32feet.NET 2.2.0.0 Bluetooth Connection

20
Design and Implementation

• Hardware
• Software
• OS and Frameworks
• Bluetooth
• Libraries
• Point Tracker
• Gestures and Gesture Recognition
• User Interface

21
Point Tracker

• Point tracking and probabilistic estimation were
  required to ensure that the system would handle
  multiple inputs (multitouch) and continue to
  function even in the event of the loss of points.
  A loss of points can occur for several reasons
• An infra-red source can leave the field of view
  of the Wii Remote
• An infra-red source can move behind another
  infra-red source
• Two infra-red sources can come together and merge

22
Point Tracker

• Several Components
• Stereoscopic Triangulation Solver
• Distance Minimizer
• Predictive Interpolator

23
Stereoscopic Triangulation

• Achieved by solving for the point at the near
  intersection of two rays
• Camera intrinsic properties needed
• Viewport width
• Viewport height
• Horizontal FOV
• Vertical FOV

24
Stereoscopic Triangulation
25
Stereoscopic Triangulation

• Using the fields of view, a vector from the
  camera position to the detected point was
  obtained. These vectors can be thought of as rays
  from the infra-red source to the camera.
• s the horizontal field of view
• ? be the vertical field of view
• x be the horizontal component of the two
  dimensional coordinate
• y be the vertical component of the two
  dimensional coordinate

26
Stereoscopic Triangulation
27
Stereoscopic Triangulation

• With the two rays from the two Wii Remote
  Cameras, the intersection of the two rays can be
  calculated using linear algebra.

28
Stereoscopic Triangulation
29
Stereoscopic Triangulation

• Physical System rays would never perfectly
  intersect
• Need to take an estimate
• Estimate distance along first vector
• Estimate distance along second vector
• Average the two points and make note of the
  distance between them

30
(No Transcript)
31
(No Transcript)
32
Distance Minimization

• The points that are reported by the two Wii
  Remotes are not necessarily sorted in the same
  order

33
Distance Minimization

• Pairwise permutation of points in matrix form
• Minimize distance error per row

34
Predictive Interpolation

• Try to predict where a point should be expected
  points
• Instantaneous velocity
• Average velocity over a period
• Minimize the errors between the observed and
  expected points

35
Predictive Interpolation
36
Predictive Interpolation

• The selected point per row is used as the point
  that the Wii3D System reports

37
Design and Implementation

• Hardware
• Software
• OS and Frameworks
• Bluetooth
• Libraries
• Point Tracker
• Gestures and Gesture Recognition
• User Interface

38
Gestures and Gesture Recognition

• Supported gestures
• Recognition methods

39
Click Gesture

• A finger going down and forward, and then up and
  backward

40
Pan/Scroll Gesture

• Two fingers that are close together moving in
  similar directions

41
Zoom Gesture

• Two fingers moving in opposite directions with
  the distance between them increasing

42
Rotate Gesture

• Two fingers moving in opposite directions with
  the distance between them constant

43
Circle Gesture

• Hidden Markov Model Gesture

44
Multitouch Gestures

• Number of points used for each gesture has been
  linked to the intensity of the gesture

45
Recognition

• Movement polling
• Simple gestures Finite State Automata
• Complex Gestures Hidden Markov Models

46
Design and Implementation

• Hardware
• Software
• OS and Frameworks
• Bluetooth
• Libraries
• Point Tracker
• Gestures and Gesture Recognition
• User Interface

47
User Interface

• States

48
User Interface

• Upon start-up, the Wii3D System attempts to
  connect to the Wii Remotes

49
User Interface

• Two possible errors not 2 Wiimotes connected or
  no Wiimotes found

50
User Interface

• Search for devices

51
User Interface

• Found devices

52
User Interface

• Allow the devices to install

53
User Interface

• Connection success

54
User Interface

• Start polling

55
User Interface

• Currently polling

56
User Interface

• Events received

57
User Study

• Goals of the Experiment
• Design and Methodology
• Tasks
• Results

58
User Study

• Goals of the Experiment
• Design and Methodology
• Tasks
• Results

59
Goals of the Experiment

• To investigate whether the Wii3D System is a
  useful system
• To investigate whether the Wii3D System could be
  used as an alternative to the mouse and keyboard

60
User Study

• Goals of the Experiment
• Design and Methodology
• Tasks
• Results

61
Design and Methodology

• 20 participants
• 15 minutes per user
• User consent required
• No individual data stored, only collated results
• Supervised by researcher

62
User Study

• Goals of the Experiment
• Design and Methodology
• Tasks
• Results

63
Tasks

• Introductory Questions
• Tasks
• Overview Questions

64
Introductory Questions

• How long have you been using a mouse and keyboard
  for?
• 8 years
• How often do you use a mouse and keyboard?
• 3 hours/day

65
Task 1 Pointer Movement

• Move pointers over targets

66
Task 2 - Clicking

• Move over targets
• Click gesture

67
Task 3 Panning/Scrolling

• Pan/Scroll and Image

68
Task 4 Zooming

• Zoom into and out of an image

69
Task 5 - Rotating

• Rotate an image about the Z and X axes

70
Task 6 Complex Gesture

• Hidden Markov Models were used
• Circle gesture

71
Overview Questions

• Would you use the Wii3D System as an alternative
  to the mouse and keyboard? Why or why not?
• If no, do you believe that the Wii3D System would
  be useful to someone else? Why?
• Would you add/remove/change anything in the Wii3D
  System? Why?
• You have undoubtedly had more experience using a
  mouse and keyboard than this type of system. What
  tasks would you find Wii3D more suitable for than
  a mouse and keyboard?

72
User Study

• Goals of the Experiment
• Design and Methodology
• Tasks
• Results

73
Results

• Preliminary Observations
• Principal Results
• Participant Subjective Feedback
• Other Findings

74
Preliminary Observations

• Confusion between rotation and zoom gestures
• People are more used to mouse and keyboard
• Hidden Markov Models not performing efficiently
• Hidden Markov Models only successfully
  recognizing a circle gesture 50 of the time

75
Principal Results

• None of the users would use Wii3D as a
  replacement for a mouse and Keyboard
• Lack of typing support
• Gestures are not always recognized
• Accuracy of the Wii3D System pointers was not as
  good as a mouse

76
Principal Results

• The majority of the participants stated that they
  thought that the Wii3D System would be useful

77
Movement

• Mouse better for a single pointer
• Wii3D better when there are more pointers

78
Clicking

• Mouse better for single pointer clicks
• Wii3D slightly better for multiple pointer clicks

79
Panning/Scrolling

• Panning and scrolling is preferred with a mouse
• Average and above ratings for Wii3D

80
Zooming

• Zooming is preferred with the mouse
• Best ratings out of all the gestures for Wii3D

81
Rotating

• Preferred, again, is the mouse
• Many users made zoom gestures instead

82
Complex Gesture

• Circle gestures were recognized correctly 50 of
  the time using Hidden Markov Models
• This result is reflected in the ten participants
  who rated this task as Very Unintuitive

83
Participant Subjective Feedback

• Users thought that the Wii3D System would be
  useful for gaming and other three dimensional
  interactions
• The users want typing functionality

84
Other Findings

• The longer people have used computers, the more
  time they spend on them daily

85
Future Work

• Improvements
• Additions

86
Future Work

• Improvements
• Additions

87
Improvements

• Multiple Wii Remotes
• Acceleration in predictive interpolation
• Dynamic Hidden Markov Models
• Gaussian Mixture Models in HMM (continuous)

88
Future Work

• Improvements
• Additions

89
Additions

• Gestures
• Double Click
• Flick
• Scaling an Object
• Typing
• Artificial Neural Networks for Recognition

90
Conclusion

• There is a use for the Wii3D System
• The Wii3D System would not work as a replacement
  for the traditional mouse and keyboard

91
Questions/Comments