Jason Hong and James Landay

1

• Jason Hong and James Landay
• University of California Berkeley
• Group for User Interface Research

2
Motivation

• Sketching is useful in many settings

3
Survey of Informal Ink Apps

• Creative / Design
• SILK
• Music Notepad
• Knight
• Elec. Cocktail Napkin
• Teddy
• PatchWork

• Capture
• ZenPad
• NotePals
• Dynomite
• FiloChat
• Others
• Pegasus
• XLibris
• PerSketch
• Kramer's Patches

4
Example - Elec. Cocktail Napkin
Gross and Do (UIST96)
5
Example - Flatland
Igarashi, Edwards, LaMarca, and Mynatt (AVI2000)
6
Common Features

• Pen input as ink or gesture
• Stroke is pen input from dragging
• Ink is a stroke that stays on-screen
• Gesture is a stroke that activates a command
• Recognize ink as objects
• Transformation / clean-up of ink
• Immediate / deferred processing of ink

erase me
7
Common Features

• Contain objects in addition to ink
• Selecting and moving objects
• Grouping of objects
• Layering of objects

8
Related Work

• Commercial Software Support
• Newton OS
• PenPoint
• Microsoft Windows for Pen Computing
• Windows CE
• Palm OS
• Problems
• Form-based and handwriting interfaces
• Not easily extensible for building informal ink
  apps

9
Related Work

• Research Software Support
• ArtKit (Henry et. al. 1990)
• Garnet Pen Int. (Landay et. al. 1993)
• Patches (Kramer, 1994)
• OOPS (Mankoff et. al. 2000)
• Problems
• Need more reusable libraries for handling and
  processing strokes
• Need more extensibility so lots of kinds
  of informal ink apps can be built

10
Problems

• Software infrastructure support for informal ink
  apps rudimentary
• Pen-based interaction techniques difficult to
  implement
• Similar to GUI development in 1980s
• Lots of applications interaction techniques
• Need cohesive frameworks toolkits

11
Goals of SATIN

• Toolkit support for informal ink apps
• Provide reusable mechanisms for handling
  processing strokes
• Separate mechanism from policy
• Be extensible for new kinds of apps
• Reusable widgets for pens
• Distribute this toolkit for use

12
Overview

• Motivation
• Survey of Informal Ink Applications
• Two Applications Built with SATIN
• SATIN Architecture Overview
• Conclusions

13
DENIM
14
SketchySPICE
15
SketchySPICE
16
SketchySPICE
17
Some Metrics
SATIN DENIM Sketchy SPICE
source files 180 76 7
methods 2192 642 63
classes 220 131 32
lines of code 20000 9000 1000
time 1 person 18 months 3 people 3 months 1 person 3 days
18
Overview

• Motivation
• Survey of Informal Ink Applications
• Two Applications Built with SATIN
• SATIN Architecture Overview
• Conclusions

19
Architecture Overview
Application
SATIN relies on Java2D and Swing
Applications use SATIN, can use Java2D and Swing
SATIN
Swing
Java2D
Java Core Classes
Java Virtual Machine
20
Architecture Overview
SATIN can be broken into twelve concepts
Recognizers
Views
Transitions
Interpreters
Scenegraph
Rendering
Stroke Libraries
Notification
Events
Widgets
Clipboard
Command
21
Recognizers

• Problem - many recognition techniques
• Recognizers let new recognition technologies be
  plugged into SATIN
• Given a stroke, return n-best list
• No actions taken
• SATIN comes with Rubine's Recognizer (implemented
  by Chris Long)

Recognizer
22
Interpreters

• Problem many ways of handling strokes
• Interpreters let new ways of handling strokes be
  plugged into SATIN
• e.g. straighten a line
• e.g. issue a command

Ink Interpreter
Gesture Interpreter
23
Interpreters and Recognizers

• Decoupled recognition (Recognizers) from actions
  (Interpreters)
• Interpreters can use recognizers
• Both are modules that can be plugged into apps
  built with SATIN

Interpreter
Recognizer
Cut
24
Multi-Interpreters

• Problem Hard to combine and extend existing
  interpreters
• Multi-Interpreters are a collection of
  interpreters dispatch policy
• e.g. dispatch to chain of interpreters
• e.g. disable some interpreters on context

Default Multi Interpreter
Intrp A
Intrp Z
Intrp B

Semantic Zoom Multi Interpreter
Intrp A
Intrp Z
Intrp B
25
Scenegraph
Sheet
Denim Label (Patch)
Denim Sketch (Patch)
Phrase (Patch)
Timed Stroke

26
Stroke Dispatching

• Needed an approach that enabled
• Individual objects handle strokes on own
• Priority to gestures over ink
• Default policy is top-down dispatch
• Process stroke with its gesture interpreter
• Re-dispatch stroke to one of its children
• Process stroke with its ink interpreter
• Handle stroke in object itself

27
Stroke Dispatching
Sheet
Denim Label (Patch)
Denim Sketch (Patch)
Phrase (Patch)
Timed Stroke
28
Stroke Dispatching
1. Process stroke with gesture interpreter
Sheet
Sheet
Denim Label (Patch)
Denim Sketch (Patch)
Phrase (Patch)
Timed Stroke

Default Multi Interpreter
Hold Select Intrp
Standard Gesture Intrp
29
Stroke Dispatching
Sheet
Denim Label (Patch)
Denim Sketch (Patch)
Phrase (Patch)
Timed Stroke
30
Stroke Dispatching
1. Process stroke with gesture interpreter
Sheet
Sheet
Denim Label (Patch)
Denim Sketch (Patch)
Phrase (Patch)
Timed Stroke

Default Multi Interpreter
Hold Select Intrp
Standard Gesture Intrp
31
Stroke Dispatching
2. Re-dispatch stroke to children
Sheet
Denim Label (Patch)
Denim Sketch (Patch)
Denim Sketch (Patch)
Phrase (Patch)
Timed Stroke
32
Stroke Dispatching
1. Process stroke with gesture interpreter
Sheet
Denim Label (Patch)
Denim Sketch (Patch)
Denim Sketch (Patch)
Phrase (Patch)
Timed Stroke
33
Stroke Dispatching
2. Re-dispatch stroke to children
Sheet
Denim Label (Patch)
Denim Sketch (Patch)
Denim Sketch (Patch)
Phrase (Patch)
Timed Stroke
34
Stroke Dispatching
3. Process stroke with ink interpreter
Sheet
Denim Label (Patch)
Denim Sketch (Patch)
Denim Sketch (Patch)
Phrase (Patch)
Timed Stroke
Phrase Interpreter
35
Stroke Dispatching
4. Handle stroke in object
Sheet
Denim Label (Patch)
Denim Sketch (Patch)
Denim Sketch (Patch)
Phrase (Patch)
Timed Stroke
36
Stroke Dispatching
Sheet
Denim Label (Patch)
Denim Sketch (Patch)
Phrase (Patch)
Timed Stroke
37
Example - Cut Shallow
38
Example - Cut Deep
39
Reusable Stroke Libraries
Straighten
Merge
40
Reusable Stroke Libraries
Split
Simplify
41
Widgets

• Pie Menus
• Pen PLAF
• Swing "Pluggable Look And Feel"
• Larger and clickable sliders
• Single-click
• file opener

42
Conclusions

• SATIN - Java toolkit for informal ink apps
• Extensible thru Recognizers, Interpreters,
  and Multi-Interpreters
• Separates mechanism from policy for strokes
• Offers reusable libraries for manipulating ink
• Reusable widgets for pens

43
Download SATIN
Download SATIN at
http//guir.berkeley.edu/projects/satin
http//guir.berkeley.edu
44
Extra Slides
45
DENIM
46
DENIM
47
Survey of Informal Ink Apps

• Elec. Cocktail Napkin
• Flatland
• Pegasus
• FiloChat
• Zenpad
• PerSketch
• Teddy
• PatchWork

• XLibris
• NotePals
• Dynomite
• SILK
• Tivoli
• Kramer's Patches
• Music Notepad
• Knight

48
Architecture Overview
SATIN can be broken into twelve concepts

• Recognizers
• Interpreters
• Scenegraph
• Stroke Libraries
• Widgets
• Views

• Rendering
• Transitions
• Clipboard
• Notifications
• Commands
• Events