Natural Language, Pen

1
Natural Language, Pen Gesture, and Nonspeech
Audio
2
Agenda

• Questions
• Talk announcement
• More on poster session
• Natural language interfaces
• Nonspeech audio
• Pen gesture

3
Talk announcement

• Monday 6/14, 4-500PM, TSRB Auditorium
• Daniel Rosenberg, VP of RD in Usability and
  Interface Design at Oracle
• Overview of lab, organization, and projects
• Largest single HCI organization in Silicon Valley
• Please come to this talk!

4
Poster session

• Friday
• Present prototypes and domain, collect feedback
• Us in here at 1200 (come early to set up!),
  other class at 1040 in 102 go see their
  posters

5
Natural input

• Universal design
• Take advantage of familiarity, existing knowledge
• Alternative input output
• Multi-modal interfaces
• Getting off the desktop

6
Natural Language

• Natural input (written and spoken)
• Expression and ambiguity
• Recognition errors
• Interaction dialogue is really difficult

7
Natural language and speech

• Written/typed input
• Written/typed output
• Speech input
• Speech output

8
Natural Language Domains

• Conceptual total set of objects/actions
• Knows about managers and salaries
• Functional what can be expressed
• What is the salary of Joes manager? Who is
  Marys manager?
• Syntactic variety of forms
• What is the salary of the manager of Joe?
• Lexical word meanings, synonyms, vocabulary
• What were the earnings of Joes boss

9
Advantages

• Easy to learn/remember
• Fewer transfer problems (rm or delete)
• Enormous potential (direct representation)
• Fast, efficient (expressability)
• Little screen real estate

10
Disadvantages

• Assumes domain knowledge
• Requires confirmation/clarification
• Error-prone input (typing, voice)
• Unrealistic expectations
• Generate mistrust/anger

11
Design Guidelines

• Use restricted language
• Tailor to user (Wizard of Oz)
• Phrase system output in valid forms
• Terse and clear dialogue
• Visual record of input/output
• Support editing and reuse

12
Voice Recognition

• Speaker independent/dependent
• Discrete or continuous
• Vocabulary size
• Lots of tradeoffs

13
Recognition Errors

• Substitution (detect and misinterpret)
• Rejection (detect but not recognize)
• Insertion (false detection)
• Deletion (no detection)

14
Applications

• Eyes and hands busy (mobility)
• Discrete voice menus (yes no)
• Dictation (speaking vs. writing)
• Interactive
• representing context
• clarify, correcting errors
• Stressful (cockpit)

15
Speech as output

• Speech synthesis
• prosody is again the problem
• digitized or synthesized
• tradeoffs in speed, naturalness and
  intelligibility

16
Speech as data

• Inherently sequential
• methods for scanning and indexing

17
Non-speech audio

• Traditionally used for warnings, alarms or status
  information
• Sounds provide information that help reduce
  error. Eg typing, video games
• Multi-modal interfaces

18
Non-speech v. speech

• Temporal difference
• Universality
• Learning time

19
Additional benefits of non-speech audio

• Good for indicating changes, since we ignore
  continuous sounds
• Provides secondary representation
• Supports visual interface

20
Auditory icons

• Use natural sounds to represent objects and
  actions in the interface
• Not beeps, but real sounds that correspond
  semantically
• SonicFinder examples folder, email file size,
  trashcan, copying
• What are some examples we know?

21
Earcons

• Synthetic sounds
• Structured combination of notes
• Alter pitch, volume, timbre, and volume to convey
  meaning

22
Auditory icons v. earcons

• Tradeoff in semantic association
• Auditory icons easier to learn, but there may not
  be an appropriate one

23
Pen and Gesture

• Another natural input
• Relies on recognition
• Does not imply reciprocal output

24
Soft Keyboards

• common on small mobile devices
• many varieties
• tapping interfaces
• simple gesture recognizers

25
T9 (Tegic Communications)

• Alternative tapping interface
• phone layout plus dictionary
• mobile phone interface

26
Multipress/multi-tap

• Maps multiple characters to each key
• Cycle through characters using multiple keypresses

27
Handwriting recognition

• On-line or off-line
• Individual variation
• Variation within single users handwriting
• Requires training
• Difficult to distinguish individual letters

28
Unistrokes

• Single stroke handwriting recognizer
• Xerox PARC -- PARC Tab
• Took advantage of screen borders

29
Graffiti

• Palm Pilot
• Very similar to Unistroke
• Physical design matters
• Recently lost lawsuit to PARC

30
Quickwrite (Perlin)

• Single stroke
• Memorize?

31
Cirrin (Mankoff)

• Word-level single stroke recognizer

32
Mixing modes of pen use

• Users want free-form and commands
• or commands vs. text
• How to switch between them?
• (1 mode) recognize which applies
• (2 modes) visible mode switch
• (1.5 modes) special pen action switches

33
Correcting recognition errors

• Really slows effective input
• word-prediction can prevent errors
• Various strategies
• repetition (erase and write again)
• n-best list

34
Other interesting applications

• Signature verification
• Note-taking
• group (NotePals by Landay _at_ Berkeley)
• student (StuPad by Truong _at_ GT)
• meetings (Tivoli and other commercial)
• Sketching systems
• early storyboard support (SILK, Cocktail Napkin)

35
Gesture Recognition

• Tracking 3D hand-arm gestures
• fiber optic, e.g. dataglove
• magnetic tracker, e.g. Polhemus
• Perceptual user interfaces
• emerging area
• mainly computer vision researchers

36
Advantages

• Using Wizard of Oz techniques, we find
• Consistent gestures for certain commands
• Easy to learn
• Remembered over time

37
Upcoming

• Part 2 poster session
• Direct manipulation for Monday (will put pdf on
  web syllabus)