Vision-based Interaction

1
Vision-based Interaction

• Scott Klemmer
• 11 November 2004

2
cs547 Michael Shiloh

• Teleo Rapid Prototyping Toolkit
• Making Things

3
The first vision-based interface

• Myron Krueger used computer vision to create
  Responsive Environments (1970s)
• Reaction is the Medium
• http//www.artmuseum.net/w2vr/timeline/videoplace_
  video.html

4
How it works

• Video and background are separated in analog
  using chroma key techniques(think broadcast
  news)
• The first and last points of each raster are
  stored in the computer, and represent the
  persons outline

5
Vision-based UIs Verbs

• Detecting and Tracking elements of a certain type
  in a scene
• Capturing contents of detected objects
• Recognizing individual members in an object class

6
Vision-based UIs Verbs

• Detecting and Tracking elements of a certain type
  in a scene

7
Vision-based UIs Verbs

• Capturing contents of detected objects

8
Vision-based UIs Verbs

• Recognizing individual members in a class

9
Vision-based UIs Nouns

• People (one or multiple)
• Bodies
• Faces
• Hands
• Documents
• Objects

10
Vision-based UIs Nouns

• People (one or multiple)
• Bodies
• Faces
• Hands
• Documents
• Objects

11
Vision-based UIs Nouns

• People (one or multiple)
• Bodies
• Faces
• Hands
• Documents
• Objects

12
Background Subtraction
I N F R A S T R U C T U R E
13
Image Moments (of Inertia)

• 0th moment is mass(total number of pixels)

14
Image Moments (of Inertia)

• 1st moment is center

15
Image Moments (of Inertia)

• 2nd moment is orientation

16
Tools for Vision apps

• Intels OpenCV
• C API to highly optimized image processing
  functions (threshold, dilate, optical flow, )
• http//www.intel.com/research/mrl/research/opencv
• Fast to run! Slow to develop
• Great for vision folks too low-level for app
  folks
• Papier-Mâché
• Java API (and to some extent visual UI) for
  vision (and other physical input)
• http//guir.berkeley.edu/papier-mache
• Fast to develop! Slow to run
• Great for app folks 5 fps can sometimes be too
  slow

17
Good Vision Books

• Computer Vision A Modern Approach
• David Forsyth and Jean Ponce (2003)
• Fantastic book but goal is more theoretical
  understanding than practical application
• Robot Vision
• Berthold Horn (1987)
• More focused on apps and interactive algorithms
• Somewhat out of date

18
iRoom demo after class