Talk v1.0

1
Gesture Toolkits

• Talk v1.0
• Gopi Flaherty
• April 15, 2003

2
Boring Gesture Systems

• Graffiti on Palm
• Extensions for X11
• Windows for Pen Computing

3
Graffiti

• Keyboard-like behavior
• Simple input model with streams of characters
• Most applications behave as if they have a
  keyboard
• Some Palm devices have keyboards instead of
  graffiti - one even comes in a graffiti or
  keyboard version - and apps dont notice

4
Graffiti

• Graffiti 2.0 different alphabet, more two stroke
  characters
• When the user enters a double stroke gesture, the
  app receives a letter, a backspace, and the next
  letter
• API documentation suggests hardcoding the list of
  these gestures, and implementing a delay if this
  behavior is a problem

5
xstroke

• Simple extension to X11
• Has a configuration file with a 3x3 grid.
  Gestures are defined based on the order you hit
  squares in the grid
• Behaves similarly to Graffiti
• Toolkit?

6
Slightly more advanced

• Gestures for simple commands
• Some simple gesture systems - possibly graffiti -
  let you have gestures for commands
• Generally very limited, also requires that
  gesture system knows about commands, and most
  simple gesture systems dont get very involved

7
Are these approaches ever useful?

• Existing apps - console UNIX apps via graffiti or
  xstroke is better than trying to make input
  fields
• Retrofitting an app such that half of it is
  gesture-based frequently has problems
• Good pen/gesture input requires fundamentally
  different ways of interacting and programming

8
Middle of the road

• Systems that provide some gesture resources but
  only at a more basic level
• Useful widgets, but with only limited
  configurability
• Usually a fixed set of gestures that are
  recognizable

9
Newton Widgets

• Text entry widget is very configurable
• Phone, Date, Time, Address and Name are field
  types

10

• Text entry widgets have many callback methods
  that can be utilized for more configurability
• viewWordScript is called when a word is
  recognized
• viewChangedScript is called to indicate that the
  text in the entry has changed
• viewGestureScript indicates that a gesture has
  been entered - this would be a raw, unrecognized
  gesture

11

• Gesture widgets are primarily forms-based
• Internal implementation is opaque
• Application isnt involved in recognition
  normally
• Becoming slightly more involved in gesture
  processing appears to be possible through some
  reverse engineering

12
The Interesting Stuff

• Why do we need toolkit support?
• There are hundreds of useful apps for the Newton
• Most apps seem quite content to use the standard
  widgets
• Newton widgets are clearly optimized for pen and
  gesture input, despite not being very involved in
  the gesture parsing

13

• Fundamentally, the widgets are still very much
  forms-based
• Intelligent dynamic forms can be extremely
  interactive, but they are still forms and still
  limited
• Hacking widgets to do things they werent
  intended to do can get tedious and slow

14
Common apps spreadsheet

• What is the cursor?
• Write gestures centered on cell
• Edit line on the top
• Early newton app viewGestureScript -
  viewWordScript?

15
Common appsWord Processor

• Standard text entry fields format text
  themselves, with limited flexibility
• Even basic word processors have quite complex
  custom formatting needs - justified text, tab
  editing, headers and footers
• It can be hard to combine custom rendering code
  with standard gesture widgets

16
Results

• If a toolkit is to provide easy support for
  complex apps, it needs to be extremely
  transparent
• Applications must be able to override as much or
  as little as is necessary for them to achieve
  their UI goals

17
Microsoft TabletPC

• Hybrid system, designed to allow handwriting in
  apps without modification
• Many basic recognition features can be used in
  apps that work with or without the pen
• Provides more advanced features and tighter pen
  integration that is harder to integrate with
  non-pen interaction

18
PenInputPanel
19
PenInputPanel

• Object that can be attached to any input field in
  your app
• Allows app to override default behaviors

20
InkEdit

• Superclass of RichEdit or RichTexBox
• Provides far more control
• Applications can register controls to receive
  various predefined gestures

21
Pre-Defined Gestures
22
Future Gestures 1
Infinity Switch in and out of gesture mode
Cross Delete Paragraph mark New paragraph
Section New section Bullet Application-specifi
c Bullet-cross Application-specific Squiggle
Bold Swap Exchange content Openup Open up
space between words Closeup Close up extra
space Rectangle Selects enclosed content
Circle-tap Application-specific Circle-circle
Application-specific Circle-cross
Application-specific Circle-line-vertical
Application-specific Circle-line-horizontal
Application-specific Plus Paste
Double-up Scroll up Double-down Scroll down
23
Future Gestures 2
Double-left Scroll left Double-right Scroll
right Triple-up Page up Triple-down Page down
Triple-left Application-specific Triple-right
Application-specific Bracket-over
Application-specific Bracket-under
Application-specific Bracket-left Start of
selection Bracket-right End of selection
Brace-over Application-specific Brace-under
Application-specific Brace-left Start of
discontinuous selection Brace-right End of
discontinuous selection Triple-tap
Application-specific Quadruple-tap
Application-specific
24
Custom Recognizers

• Custom Recognizer objects can be registered to
  Ink objects
• An InkCollector object collects gestures, which
  can be passed to a custom recognizer, or to the
  original Microsoft one, or both in parallel

25
For each recognizer
enum RecognizerCapabilities DontCare 1,
Object 2, FreeInput 4, LinedInput
8, BoxedInput 16, CharacterAutoCompletio
nInput 32, RightAndDown 64,
LeftAndDown 128, DownAndLeft 256,
DownAndRight 512, ArbitraryAngle 1024,
Lattice 2048
26
Recognition Ambiguity

• Egg Freckles?
• One of the largest challenges with computer
  recognition of any type
• Dealing with recognition ambiguity can be the
  difference between a pen system working or not
  working

27
GraffitiWhat ambiguity?

• Sidesteps the issue completely
• You write the stroke the way youre supposed to
• If it mis-recognizes, the assumption is that it
  was your fault
• Blaming the user is argued by some to be the
  cause of its success

28
Different kinds of ambiguity

• Basic input What letter or word did the user
  type?
• One of the easier types of ambiguity to deal with
• The recognizer provides an n-best list, select
  the top, and you provide a UI to making alternate
  selections from the list

29
Simple Ambiguity

• This can be somewhat more challenging with
  multiple recognizers how do you intermingle
  their respective n-best lists?
• If gestures can perform actions, the UI can be
  difficult how do you present a choice of cut,
  copy or the word hello to the user?
• Newton had interesting choice in n-best do
  letter by letter, ignoring dictionary weighting.
  Produced a new n-best list

30
Ambiguity and time

• If gesture recognition is being used in a
  time-constrained environment, it brings up other
  interesting challenges
• Resolving ambiguities is distracting and slow,
  but retaining n-best information indefinitely can
  increase storage space immensely
• Lazy ambiguity resolution cant be used for
  action gestures without extreme penalties

31
The lazy approach

• Later versions of the Newton OS let you do lazy
  recognition
• When text is written, its retained as the
  original glyphs, segmented into words and
  reflowed with word wrap as normal text
• At any stage in the future, the user can request
  that text be recognized
• Retain as glyphs is also an option in the n-best
  correction widget

32
DENIM and SILK

• Very lazy recognition
• Somewhat similar to the Newtons keep it as
  glyphs, but significantly more structured than
  simple text editing
• One of the design goals is to deal with the
  problem of recognition systems demanding
  immediate ambiguity resolution - very distracting

33
Meta-ambiguity

• What about segmentation?
• Most shipping recognition systems do segmentation
  of strokes into individual collections, and do
  not allow any future re-analysis of the decision
• Allowing segmentation ambiguity is difficult from
  a UI perspective - double tapping a word is easy,
  but what about trying to resolve a pair of words

34
Segmentation

• Another aspect - who gets the gesture?
• On a screen with forms, how do you resolve which
  text entry field gets the text?
• Approaches I have seen are the simple ones -
  theres a bounding box, strokes in the box are
  for that widget
• This reduces the density of widgets and requires
  careful design

35
Was it a gesture?

• A common problem in poorly designed Newton apps
  text entry box right next to control widgets
• Write a stroke - start too close to the window
  drag area and the window moves instead
• Nearly impossible to resolve, since control
  widgets demand immediate feedback, but a serious
  problem on small screens

36
Application Support

• How much support is needed in the app for dealing
  with ambiguity?
• Cut and paste on the Newton retains n-best
  objects
• In theory, nearly infinite knowledge
  possibilities
• Which text field wants the input?

37
Application Intelligence

• Imagine two text fields One is a city, the other
  is a county
• Its generally impractical to have a complete
  list of all of these, but a possible list can
  exist
• If there is ambiguity about which field an entry
  is for, the applications knowledge about likely
  values can add increased weighting knowledge

38
Specifying Gestures By Example

• Dean Rubine

39
Basic outline

• Provide a series of samples for each gesture
• Works exclusively with single strokes
• Eager recognizes during the stroke to increase
  responsiveness
• Drawing ink on screen is easy - so process during
  the stroke
• Battery systems slow processor during strokes

40
Providing Integrated Toolkit Level Support for
Ambiguity in Recognition Based Interfaces

• Jennifer Mankoff, Scott E. Hudson, Gregory D.
  Abowd

41
Basic ambiguity
42
Segmentation Ambiguity
43
Shortcomings in other systems

• Nobody seems to deal with target ambiguity
• Segmentation ambiguity is also rarely tackled
• Application-specific knowledge is rarely used

44
Mediation Toolkit

• Toolkit identifies ambiguous events and sends
  them to the mediation subsystem
• Each mediator receives the event in turn
• Mediators can ignore, partially resolve, or defer
  their decision to the future
• Some mediators have a UI, some do not
• Also allows much lazier mediation