Vision-Based Interactive Systems

1
Vision-Based Interactive Systems

• Martin Jagersand
• c610

2
Applications for vision in User Interfaces

• Interaction with machines and robots
• Service robotics
• Surgical robots
• Emergency response
• Interaction with software
• A store or museum information kiosk

3
Service robots

• Mobile manipulators, semi-autonomous

DIST TU Berlin KAIST
4
TORSO with 2 WAMs
5
Service tasks
This is completely hardwired! Found no real task
on WWW
6
But

• Maybe first applications in tasks humans cant do?

7
Why is humanlike robotics so hard to achieve?

• See human task
• Tracking motion, seeing gestures
• Understand
• Motion understanding Translate to correct
  reference frame
• High level task understanding?
• Do
• Vision based control

8
Types of robotic systems
Preprogrammed systems
Autonomy
Programming by demonstration
Tele-assistance
Supervisory control
Generality
9
Interaction styles
If A then end
Conventional

• Low bandwidth interaction
• Partial or indirect system state displayed
• User works from internal mental model

10
Interaction styles
Direct Manipulation

• High bandwidth interaction
• Interact directly and intuitively with objects
  (affordance)
• See system state (visibility)
• (Reversible actions)

11
Examples of Direct Manipulation

• Drawing programs e.g. Mac Paint
• Video games, flight simulator
• Robot/machine teaching by showing
• Tele-assistance
• Spreadsheet programs
• Some window system desktops
• But can you always see effects (visibility)?

12
xfig drawing program

• Icons afford use
• Results visible
• Direct spatial action-result mapping

matlab drawing
line(10, 20,30, 85) patch(35, 22,15, 35,
C) C complex structure text(70,30,'Kalle')
Potentially add font, size, etc
13
Why direct manipulation?

• Recognition quicker than recall.
• Human uses the world as memory/model
• Human skilled at interacting spatially

How quick is direct?

• Subsecond! Experiments show human performance
  decreased at 0.4s delay.

14
Vision and Touch based UI
Supports Direct Manip!

• Typical UI today Symbolic, 1D (slider), 2D
• But human skilled at 3D, 6D, n-D spatial
  interaction with the world

15
Seeing a task

• Tracking movement
• See directions, movements in tasks
• Recognizing gestures
• Static hand and body postures
• Combination Spatio-temporal gestures

16
Tracking movement

• Tracking the human is hard
• Appearance varies
• Large search space, 60 parameters
• Unobservable Joint angles have to be inffered
  from limb positions, clothing etc.
• Motion is non-linear.
• Difficult to track 3D from 2D image plane info
• Self occlusion of limbs

17
Trick 1Physical model

• Reduce number of DOFs by coupled model of
  articulated motion (Hedvig, Mike)

18
(No Transcript)
19
Trick 2Use uniqueness of skin color

• Can be tracked at real time

20
Gestures

• Identifying gestures is hard
• Hard to segment hand parts
• Self occlusion
• Variability in viewpoints

21
Trick 3Scale space

• Define hand gesture in course to fine terms

22
Trick 4Variability filters
23
Programming by Demonstration

• From assembly relations
• From temporal assembly sequence
• Segmenting manipulation sequence into parts
  (subtasks) is hard
• Using a gesture language

24
Tele-assistance

• Gestures context

25
Robust manipulations
26
Conclusions

• Most aspects of Robot see robot do are hard
• Conventional methods are
• Incapable of seeing task
• Incapable of understanding whats going on
• Incapable of performing human manipulation tasks
• Uncalibrated methods are more promising