High Performance Pen Interfaces

1
High Performance Pen Interfaces

• Ken Hinckley
• Microsoft Research
• October 22nd, 2004
• kenh_at_microsoft.com
• http//research.microsoft.com/kenh

2
Pen Computing Now?
3
Project 1 Stitching

• Ken Hinckley
• Gonzalo Ramos
• Francois Guimbretiere
• Patrick Baudisch
• Marc Smith

4
Wireless Networks

• Still an immature technology
• Flaky, hard for users to configure, slow,
• Most research has focused on small-screen
  presentation of web pages
• What are the completely new unique affordances
  of wireless connectivity?
• Odd phrase not what it is, but what it lacks

5
Wireless Network 2004 Horseless Carriage 1904?

• Are Wireless Networks really just wired networks
  without the wires?
• Or are they something completely different?

6
Wireless Device Soup Which links does the user
want?
7
Name That Device

• Users need techniques to intuitively form bridges
  between devices
• How do users name the devices to connect?
• What is type / purpose of the connection?
• Parameters? (Who copies what, to where?)

8
Stitching

• Pen stroke that spans displays
• Move the pen
• Cross over bezel
• Finish stroke on nearby tablet
• System infers connection

9
Central Design Questions

• How is a connection established?
• What type of connection is required?
• What information is shared?
• What is the spatial relationship between devices?
• How do users share physical space?
• Explored in context of photo sharing app

10
VIDEO

• AVI04
• ACM Advanced Visual Interfaces 2004

11
1. Establishing a Connection

• Recognizes timing dynamics of pen trajectory
• There is nothing special about the pen!
• Wireless signal strength determines nearby devices

12
2. Type of Connection

• User selects a multi-device command
• e.g., Pie menu at end of stitching gesture

13
3. What Information to Share?

• Many commands may need operands
• Lasso to select, continue to other display to
  stitch.

14
4. What is Spatial Relationship Automatic Screen
Calibration

• Fits a line to absolute coords of users pen
  stroke

15
5. How Do Users Share Physical Space?

• Sociology?Proxemics How people use space
• Invisible bubble that surrounds an individual

16
6. How Do Users Share Physical Space?

• Sociology?Proxemics How people use space
• Invisible bubble that surrounds an individual

• Homework assignment
• Sit right next to someone at airport (when it is
  not necessary to do so)
• Time how long it is before the other person
  leaves

17
5. How Do Users Share Physical Space?

• Sociology?Proxemics How people use space
• Invisible bubble that surrounds an individual

18
(No Transcript)
19
(No Transcript)
20
Proxemic Lessons for Stitching

• Dont require contact touching is taboo
• Do they have to be right next to one another?
  10-40cm
• wide screen would be nice for collaboration
• no two faces trying to peek at only one screen

21
Proxemic Lessons for Stitching

• Dont require contact
• Do they have to be right next to one another?
  10-40cm
• wide screen would be nice for collaboration
• no two faces trying to peek at only one screen
• Gives users flexibility to be involved, or not
  mood, task,
• Intimate Spaces Combining screens. For close
  collaboration with friend or trusted colleague
• Personal Spaces Tablets can be separated by up
  to 30 yet still possible to stitch to give files
  to colleague, etc.
• Social Spaces Once connected, transporter can
  be used to give files to a user beyond arms reach

22
Proxemic Lessons for Stitching

• Dont require contact
• Do they have to be right next to one another?
  10-40cm
• wide screen would be nice for collaboration
• no two faces trying to peek at only one screen
• Gives users flexibility to be involved, or not
• Intimate Spaces Combining screens. For close
  collaboration with friend or trusted colleague
• Personal Spaces Tablets can be separated by up
  to 30 yet still possible to stitch to give files
  to colleague, etc.
• Social Spaces Once connected, transporter can
  be used to give files to a user beyond arms
  reach
• Support varying relative body orientation

23
Proxemics Orientation of Spaces

• Communication patterns change wrt orientation
• Stitching supports
• Shoulder-to-shoulder
• Face-to-face
• At 90 angles

Relative frequency
24
Ongoing work

• Multiple Devices Cooperative Stitching

25
Ongoing work

• Multiple Devices Cooperative Stitching

26
Project 2 Scriboli

• Ken Hinckley
• Patrick Baudisch
• Gonzalo Ramos
• Francois Guimbretiere

27
Round trip problem on Tablet PC

• 4 trips ( some xtra) for copy/paste

28
Round trip problem

• 4 trips ( some xtra) for copy/paste

29
Round trip problem

• 4 trips ( some xtra) for copy/paste

30
Round trip problem

• 4 trips ( some xtra) for copy/paste

31
Round trip problem

• 4 trips ( some xtra) for copy/paste

32
Round trip problem

• 4 trips ( some xtra) for copy/paste

33
Round trip problem

• 4 trips ( some xtra) for copy/paste

34
Round trip problem

• 4 trips ( some xtra) for copy/paste

35
Round trip problem

• No right click, kbd shortcuts to avoid this!

36
GEdit Kurtenbach Buxton
Delete one obj.
Copy add final C to move
Delete group end stroke inside lasso

• One of the first works considering pen phrases
• Every stroke is gesture
• Only 3 cmds
• Group vs. single obj. inconsistent

Move end stroke outside. Objects move on PenUp.
37
SKETCH Brown University

• Adding 1 more gesture breaks everything
• Specific to 3D, difficult to learn how to use

38
Marking Menus

• Prompted Blind Scale independence
  
• Self-revealing prompts w/ available cmds
• Rapid expert use blind marking
• Menu appears only after 1/3 second delay
• Size of mark does not matter

39
FlowMenu Francois Guimbretiere

• Integrates command selection with continuous
  parameter manipulation
• But no blind marking, size of gesture matters

Crossing in?out chooses Item submenu
Crossing out?in chooses Move command
Dragging continues
40
Tivoli

• Tivoli project _at_ Xerox PARC
• We wanted quick, informal feel of scribbling

41
Learning Cursive

• building blocks of writing as basis for pen UI?
• not natural handwriting recognition

42
The Deep Structure of Handwriting

• The scribble of Scriboli
• Phasic muscle contractions
• fast, brief, usually repetitive
• Pre-programmed, internally driven
• The conversion of those Os to real words
  and sentences happens when small variations in
  the amount of verticality or laterality are
  introduced into the movement. The extreme
  orderliness and predictability of individual
  handwriting is due to this mode of generating and
  controlling the movement. F. R. Wilson, The
  Hand, p. 163

43
Design Philosophy of Scriboli

• RAPID
• Fast motions, repeatable, no waiting / dwelling,
  minimal demands on visual attention
• UNAMBIGUOUS
• No guessing games
• No reco unless I tell the system otherwise
• (There is almost no reco in Scriboli)
• EXPRESSIVE
• Supports many commands, easy to grow design
• Mechanisms for a wide span of pen apps, not
  restricted to specific niche application

44
Grammar for pen input

• Scriboli proposes fundamental building blocks
• Links together object, verb, indirect object in
  fast, fluid, unambiguous command phrases

45
Ink vs. Gesture

• Who does the recognition?
• Ink strokes intended primarily for human
• Gesture strokes intended for computer
• Virtually all pen systems make this distinction
  at some level
• Very seductive to attempt to do automatically
• Many systems spend most or all effort just on
  this
• Any pen gesture might occur in freeform input
• Scriboli uses explicit ink/gesture mode switch

46
Ink/Gesture study w/ U. Washington

• 5 Techniques
• Barrel Button
• Taphold
• NPH button
• Pen Pressure
• Pen Eraser

47
Grammar for pen input

• Scriboli proposes fundamental building blocks
• Links together object, verb, indirect object in
  fast, fluid, unambiguous command phrases

48
Delimiters Single-stroke lasso selection
marking

• Initial part of stroke is the scope
• Pigtail is the delimiter
• Tail interpreted as the mark

49
What is a Delimiter?

• A delimiter is something different about the
  pen input stream
• Determines lexical structure of pen stroke
• Pigtail only intended as delimiter during
  Gesture mode
• Not an automatically inferred mode switch

50
Grammar for pen input

• Scriboli proposes fundamental building blocks
• Links together object, verb, indirect object in
  fast, fluid, unambiguous command phrases

51
Direct manipulation phase

• Copy-paste in place
• Zero round trips
• Continuing past outer border turns into dragging
  (like FlowMenu)
• A single stroke supports
• Scope
• Command activation
• Dragging (direct manip)
• FlowMenu cannot support all 3

52
Direct manip phase preserves scale-independence

• Possible to have both
• boundary crossing (like FlowMenu)
• scale-independence (like marking menus)
• Scale-ind. only matters for blind marking
• Dragging naturally lasts longer than 1/3 second.

53
Grammar for pen input

• Scriboli proposes fundamental building blocks
• Links together object, verb, indirect object in
  fast, fluid, unambiguous command phrases

54
Scriboli Video

• VIDEO

55
Delimiters performance study

• Pigtail
• Timeout
• draghold during lasso
• Multi-stroke
• acquire selection handle after lasso
• Button
• Hit button during lasso to delimit scope vs. mark
• RI Repeated Invocation
• Same command 24x

56
Learning effects

• Timeout plateaus quickly
• Button gets worse
• Pigtail, multi-strokefollow similar trend
• Pigtail problems
• Fail to accept
• Heading in wrong direction
• Curl too much ? choose wrong direction

57
Pigtail self-revelation technique

• Timeout easy to learn but tedious
• Pause ?Scriboli shows how to draw pigtail
• Allows experienced users to shortcut timeout
• Helps novices learn the pigtail (?)

58
ONGOING Phrasing together multiple gesture
strokes

• Automatic means hard to group arbitrary strokes
  across various application domains
• Time delay Time outs are lame
• Scriboli uses muscular tension on Gesture button
  (lit. may help reduce mode errors)

59
Separating the pig from the tail

• symbol indicatesgesture is extension
  ofcurrent command phrase
• Solves screen edge problem very nicely

60
Disjoint scopes
61
Grammar for pen input

• Scriboli proposes fundamental building blocks
• Links together object, verb, indirect object in
  fast, fluid, unambiguous command phrases

62
Crossing Scope (line)

• Draw stroke through object
• Crossing can have different semantics than
  lassoing
• Brings up crossing menu
• e.g. the crossing line specifies right edge of
  icon for alignment

63
Alignment (Multiple to edge)
64
Alignment (Multiple to edge)
65
Alignment (Multiple to edge)
66
Crossing Scopedesign properties

• For cmds that require edge and/or axis
• Only possible to invoke withrequisite spatial
  information
• Limits possible errors of syntax
• Tremendous economy of design
• e.g. 1 Align cmd instead of 6

67
Alignment (Individual Edges)
68
Alignment (Individual Edges)
69
Alignment (Individual Edges)
70
Alignment (Individual Edges)
71
Alignment (Various edges)
72
Alignment (Various edges)
73
Alignment (Various edges)
74
Alignment (Various edges)
75
Alignment (Various edges)
76
Alignment (Various edges)
77
Grammar for pen input

• Scriboli proposes fundamental building blocks
• Links together object, verb, indirect object in
  fast, fluid, unambiguous command phrases

78
Future work

• Are there building blocks missing?
• What are alternative designs for each part?
• What other phrases can be envisioned?

79
Questions?