quill: A UI Design Case Study

1
quill A UI Design Case Study

• A. Chris Long
• Postdoctoral Research Fellow
• HCI Institute

2
Administrivia

• Turn in Homework 1
• Homework 2 assignment on course website

3
Outline

• Gestures
• quill Design
• quill Architecture Implementation
• Object-Oriented Design
• Long-running Tasks and Interfaces
• Recognition-based UIs
• How Gesture Recognition Works
• More quill Implementation

4
Gestures

• Mark that invokes a command

5
Gestures

• Mark that invokes a command

delete me
6
Gestures

• Mark that invokes a command
• Benefits
• Fast
• Save screen space
• Avoid aerobic interface
• Specify operation and operand(s) simultaneously

delete me
7
Problems With Gestures

• For users
• Computer often misrecognizes gestures
• Difficult to learn and remember gestures
• For designers
• Gestures are hard to design

8
quills Place in the World

• Who Designer of pen-based UI
• When During design phase
• What Designing gestures
• Why To design better gestures

Design
Prototype
Test
9
Another View of quill
10
Designing quill

• Design low-fi prototype
• Pilot test low-fi
• Redesign
• Implement Java version

11
Low-fi Prototype
12
Current Version

• Tree view
• Expanded view
• Suggestion for improvement

13
Recognition Problem
14
Gestures, Packages, and Groups, Oh My!
Gesturepackage
MyApplication
Gesturegroup
Edit
View

Gesturecategory
copy

cut
Gesture

15
quill Architecture

• JDK Core
• Gesture Objects
• Gesture
• Gesture Category
• Recognizer
• Gesture Analyses

• Widgets

16
quill Architecture
Widgets Widgets Widgets Widgets Widgets
Gesture Analyses Gesture Analyses
Gesture Recognition Objects Gesture Recognition Objects
Swing
Core JDK Core JDK Core JDK Core JDK Core JDK
17
Implementation Overview

• 93 classes, 26,000 lines

• Java/Swing

18
Object Oriented Design

• Gesture Objects
• Gesture
• GestureCategory
• GestureGroup
• GesturePackage

• Gesture Display Widgets
• GestureDisplay
• GestureInteractor
• GestureCategoryDisplay
• GestureGroupDisplay
• GesturePackageDisplay
• GestureCategoryThumbnailDisplay

19
OO Design
GestureObject
GestureContainer
DefaultGestureObject
AbstractGestureContainer
GestureCategory
GestureGroup
GesturePackage
20
Analyzing Gestures

• Problem Analysis may take a long time
• Solutions?

21
Analyzing Gestures

• Problem Analysis may take a long time
• Strategies for analyzing gestures
• Run analysis in foreground, disable all actions
• Analyze in background
• Disable all conflicting actions
• Allow anything, cancel analyses
• Allow anything

22
Multithreading

• Benefits
• ?
• Drawbacks
• ?

23
Multithreading

• Benefits
• Computation in background
• Can do very complex tasks
• User can perform any action, any time
• Drawbacks
• Must synchronize data across threads
• Deadlock
• Race conditions

24
Multiple Threads in Java

• Thread class
• Start separate thread that runs any method
• synchronized keyword
• Prevents a method from being called or object
  from being accessed by more than one thread at
  once
• SwingWorker class
• Easy to run background thread that produces a
  result

25
Recognition-Based UIs

• Examples
• Pen gesture
• Handwriting
• Speech
• Advantages
• ?
• Disadvantages
• ?

26
Recognition-Based UIs

• Advantages
• Natural input
• Informal
• Multiple interpretations
• Disadvantages
• No clear separation of data and commands
• Ambiguous commands
• Individual differences
• Multiple interpretations

27
Dealing with Recognition

• Confirm dangerous operations
• Multiple modes may help disambiguate
• Speech pen, for example
• Provide undo/redo

28
How Does Recognition Work?

• Training
• Measure geometric features of gestures
• Compute average feature values for each type of
  gesture
• Compute how important each feature is

• Recognition (a.k.a. Classification)
• Measure geometric features of unknown gesture
• Compare with features of known gestures
• Pick closest known gesture

29
Rubines Feature Set

• Initial angle
• Length of bounding box diagonal
• Angle of bounding box diagonal
• Distance between first and last points
• Angle between first and last points
• Length
• Total angle traversed

30
Rubines Feature Set (cont.)

• Sum of absolute value of angle at each point
• Sum of squared angle at each point (sharpness)
• Square of maximum speed
• Duration

31
Weakness of Feature-based Recognition

• Feature set has to be right

32
Feature-based Training

• Mean feature vector
• Per-class covariance matrix

33
Training (cont.)

• Average covariance matrix
• Weights estimation

34
Feature-based Classification

• For all gesture classes c, compute
• Return class with max vc

35
quill Implementation Issues (contd)

• Suggestions (contd)
• When do they disappear?
• User says to ignore
• No longer apply
• When are two notices the same?
• At time A, gestures X and Y are too similar
• At time B, something happens and they arent
  similar
• At time C, they are similar again
• Display notice again?

36
Implementation Issues (contd)

• Annotations
• Special purpose vs. generic
• How to compose?
• Gesture display
• Desktop windows
• Tiled
• Standard MDI
• Display factory

37
Implementation issues (contd)

• Generic properties
• void GestureObject.setProperty(String name,
  Object value)
• Object GestureObject.getProperty(String name)
• What happens to properties like isMisrecognized
  when copy paste?
• Persistent vs. transient

38
Implementation issues (contd)

• Generic properties
• void GestureObject.setProperty(String name,
  Object value)
• Object GestureObject.getProperty(String name)
• What happens to properties like isMisrecognized
  when copy paste?
• Persistent vs. transient

39
Implementation Issues (contd)

• Menus enable/disable items, and dispatching
  commands
• Context-dependent menu items
• Context-dependent gesture drawing
• Solution Chain of command pattern that follows
  selection

40
Questions?

• Other topics
• Grad school
• Berkeley (city or university)
• Speech UIs
• More on pen-based UIs

41
The End

• Design Patterns Elements of Reusable
  Object-Oriented Software
• by Erich Gamma, Richard Helm, Ralph Johnson, and
  John Vlissides
• http//www.cs.cmu.edu/chrisl/
• http//guir.berkeley.edu/projects/quill/

42
Extras
43
Criticisms

• Recognition is due to poor recognizers, which
  will eventually be perfect
• Not for a long time, if ever
• Learning and memory predictions still useful
• Recognition will never be good enough
• Already good enough for many users
• Is improving over time
• Typing is faster and more accurate than writing
• Pens wont replace keyboards, but are better in
  many situations

44
Criticisms (cont.)

• Gestures will always be too hard to learn and
  remember, and will never be useful enough
• Gestures very often used on paper in many domains
• Design tool doesnt guarantee good gestures
• Will provide useful advice
• Will shorten design cycle
• There are only a small number of common gestures
• Application/domain-specific gestures
• Personal shorthand

45
Criticisms (cont.)

• Marking menus are better than free-form gestures
• Cant specify additional information with
  marking-menus
• Shapes are arbitrary
• All straight lines and 45/90º angles

46
Criticisms (cont.)

• Whats wrong with traditional GUI interaction
  techniques?
• Pen ? mouse
• Pen
• Finer control
• Has other dimensions (e.g., pressure, tilt)
• Mouse
• More buttons (sometimes)

47
Pen vs. Mouse

• Kato, et al
• Pen significantly faster more accurate than
  mouse for precise dragging
• Pen significantly faster for pointing
• Pen has directional dependencies mouse doesnt
• Mackenzie, et al
• Pen slightly faster for pointing
• Mouse slightly faster for dragging