Ink, Audio and Video

1
Working with Natural Data Types

• Ink, Audio (and Video)

2
Natural Data Types

• As we move off the desktop, means of
  communication mimic natural human forms of
  communication
• Writing, Speaking, Seeing
• Ink, Audio, Video
• Understanding these data types leads to
  interesting applications
• e.g., capture applications

3
Uses of Natural Data Types

• Recognition
• Translate into other information (words or
  description of scene)
• Dealing with errors
• Raw data
• as data use for human consumption (to read,
  hear or see)
• Often linked with other context (time) in capture
  applications

4
Pen Computing

• Use of pens has been around a long time
• Light pen was used by Sutherland before Engelbart
  introduced the mouse
• Resurgence in 90s
• Types of pens
• Passive (same as using a finger)
• Active (pen provides some signal)

5
Example Pen Technologies

• Passive
• Touchscreen (e.g., PDA, some tablets)
• Contact closure
• Vision techniques
• Active
• Pen emits signal(s)
• e.g. IR ultrasonic
• Where is sensing? Surface or pen?

6
Questions about Pens

• What operations detectable
• Contact up/down?
• Drawing/Writing?
• Hover?
• Pressure?
• Modifiers? (like mouse buttons)
• Which pen used?
• Eraser?
• Differences between pen and mouse?

7
Example Sony Ericsson Chatpen

• Reads dot pattern on paper
• - requires special paper
• Transmits via Bluetooth

8
Example mimio

• Active pens
• IR ultrasonic
• Portable sensor
• Converts any surfaceto input surface
• Can chain theseto create big surface
• http//www.mimio.com

9
Pen input

• Free-form ink
• Soft keyboards
• Recognition systems
• generalize to gesture-based systems

10
Toolkits for Pen-Based Interfaces

• UC Berkeley SATIN toolkit (UIST 2000)
• http//guir.berkeley.edu/projects/satin
• MS TabletPC
• Descendant of MS Windows for Pen Computing
• MS Pocket PC, CE.net
• Apple Newton OS
• GO PenPoint
• Palm Developer environments
• GDT (Long, Berkeley) Java-based trainable
  unistroke gesture recognizer
• OOPS (Mankoff, GT) error correction based on
  subArctic

11
SATIN (UIST 2000)

• Pen input for informal input
• Sketching (others have investigated this)
• Common toolkit story
• Gee, X sure is a neat class of apps!
• Golly, making X apps is tough!
• Heres a toolkit to build X things easily!

12
The SATIN Toolkit

• The application space
• Informal ink apps
• Beyond just recognition
• Pen look-and-feel
• Abstractions
• Recognizers
• Interpreters
• multi-interpreters

13
Audio

• Natural appeal of sound
• Off-the desktop
• hands-free
• display not necessary
• effective for a short distance

14
How sound is created

• Sound is created when air is disturbed, usually
  by vibrating object(s) causing ripples of varying
  air pressure propagated by the collision of air
  molecules.

15
How sound is perceived

• Characteristics of sound
• Volume
• Pitch
• Features of sound wave
• Amplitude
• Volume amplitude of a sound wave
• Frequency
• Pitch frequency of a single sound wave

16
Example
17
When multiple sounds occur
18
Sampling sound

• Note sampling rate affects accurate
  representation of the sound wave

19
Acquiring audio
20
Generating audio
21
Features in speech
22
Audio features

• Pauses
• Analogy to mouse up/down/drag
• Who is speaking
• Turn-taking
• How is someone speaking
• Prosody, affect
• What is being said
• Recognition of words

23
Challenges with Audio

• Provide support for handling audio in manner
  similar to text
• Simplify acquisition, segmenting, storing,
  playback
• Open air audio (in/out) results in messy audio
  from ambient noise and feedback
• Support noise reduction, enhance audio quality,
  echo cancellation

24
Noise reduction

• Random noise is hard to predict

Noise Filter
f (t)
f(t)
25
Echo cancellation

• Software hardware exist, but are hard for
  developers to easily add to application
• Random noise is hard to predict, but echoes are
  not so random

Echo Canceller
f(t)
f (t)
26
Other interesting audio work

• Auditory icons or earcons
• sound that identifies objects actions
• Automatic transformation of GUI interfaces to
  auditory interfaces
• Mercator (Mynatt, 1995)
• Sound localization sound cones

27
Creating UIs with audio

• Audio
• Java Media Framework
• http//java.sun.com/products/java-media/jmf/
• Speech
• UC Berkeley SUEDE toolkit
• http//guir.berkeley.edu/projects/suede/
• CSLU toolkit
• http//cslu.cse.ogi.edu/toolkit/

28
Next Time

• Design Patterns for UI Construction