Interaction Devices

1
Interaction Devices

• Human Computer Interaction
• CIS 6930/4930
• Section 4188/4186

2
Interaction Performance

• 60s vs. Today
• Performance
• Hz -gt GHz
• Memory
• k -gt GB
• Storage
• k -gt TB
• Input
• punch cards -gt
• Keyboards, Pens, tablets, mobile phones, mice,
  digital cameras, web cams
• Output
• 10 character/sec
• Megapixel displays, color laser, surround sound,
  force feedback, VR
• Substantial bandwidth increase!

3
Interaction Performance

• Future?
• Gestural input
• Two-handed input
• 3D I/O
• Others voice, wearable, whole body, eye
  trackers, data gloves, haptics, force feedback
• Engineering research!
• Entire companies created around one single
  technology
• Current trend
• Multimodal (using car navigation via buttons or
  voice)
• Helps disabled (esp. those w/ different levels of
  disability)

4
Keyboard and Keypads

• QWERTY keyboards been around for a long time
• (1870s Christopher Sholes)
• Cons Not easy to learn
• Pros Familiarity
• Stats
• Beginners 1 keystroke per sec
• Average office worker 5 keystrokes (50 wpm)
• Experts 15 keystrokes per sec (150 wpm)
• Is it possible to do better? Suggestions?

5
Keyboard and Keypads

• Look at the piano for possible inspiration
• Court reporter keyboards (one keypress multiple
  letters or a word)
• 300 wpm, requires extensive training and use
• Keyboard properties that matter
• Size
• large - imposing for novices, appears more
  complex
• mobile devices
• Adjustable
• Reduces RSI, better performance and comfort
• Mobile phone keyboards, blackberry devices, etc.

6
Keyboard Layouts

• QWERTY
• Frequently used pairs far apart
• Fewer typewriter jams
• Electronic approaches dont jam.. why use it?
• DVOARK (1920s)
• 150 wpm-gt200 wpm
• Reducing errors
• Takes about one week to switch
• Stops most from trying
• ABCDE style
• Easier for non-typists
• Studies show no improvement vs. QWERTY
• Number pads
• Whats in the top row?
• Look at phones (slight faster), then look at
  calculators, keypads
• Those for disabled
• Split keyboards
• KeyBowls orbiTouch (screenshot)
• Eyetrackers, mice

7
Keys

• Current keyboards have been extensively tested
• Size
• Shape
• Required force
• Spacing
• Speed vs. error rates for majority of users
• Distinctive click gives audio feedback
• Why membrane keyboards are slow (Atari 400?)
• Environment hazards might necessitate
• Usually speed is not a factor

8
Keys Guidelines

• Special keys should be denoted
• State keys (such as caps, etc.) should have
  easily noted states
• Special curves or dots for home keys for touch
  typists
• Inverted T Cursor movement keys are important
  (though cross is easier for novices)
• Auto-repeat feature
• Improves performance, but only if repeat is
  customizable (motor impaired, young, old)
• Two thinking points
• Why are home keys fastest to type?
• Why are certain keys larger? (Enter, Shift, Space
  bar)
• This is called Fitts Law

9
Keypads for small devices

• PDAs, Cellphones, Game consoles
• Fold out keyboards
• Virtual keyboard
• Cloth keyboards (ElekSen)
• Haptic feedback?
• Mobile phones
• Combine static keys with dynamic soft keys
• Multi-tap a key to get to a character
• Study Predictive techniques greatly improve
  performance
• Ex. LetterWise 20 wpm vs 15 wpm multitap
• Draw keyboard on screen and tap w/ pen
• Speed 20 to 30 wpm (Sears 93)
• Handwriting recognition (still hard)
• Subset Graffiti2 (uses unistrokes)

10
Pointing Devices

• Direct manipulation needs some pointing device
• Factors
• Size of device
• Accuracy
• Dimensionality
• Interaction Tasks
• Select menu selection, from a list
• Position 1D, 2D, 3D (ex. paint)
• Orientation Control orientation or provide
  direct 3D orientation input
• Path Multiple poses are recorded
• ex. to draw a line
• Quantify control widgets that affect variables
• Text move text
• Faster w/ less error than keyboard
• Two types (Box 9.1)
• Direct control device is on the screen surface
  (touchscreen, stylus)
• Indirect control mouse, trackball, joystick,
  touchpad

11
Direct-control pointing

• First device lightpen
• Point to a place on screen and press a button
• Pros
• Easy to understand and use
• Very fast for some operations (e.g. drawing)
• Cons
• Hand gets tired fast!
• Hand and pen blocks view of screen
• Fragile
• Evolved into the touchscreen
• Pros Very robust, no moving parts
• Cons Depending on app, accuracy could be an
  issue
• 1600x1600 res with acoustic wave
• Must be careful about software design for
  selection (land-on strategy).
• If you dont show a cursor of where you are
  selecting, users get confused
• User confidence is improved with a good lift-off
  strategy

12
Direct-control pointing

• Primarily for novice users or large user base
• Case study Disney World
• Need to consider those who are disabled,
  illiterate, hard of hearing, errors in usage (two
  touch points), etc.

13
Indirect-Control Pointing

• Pros
• Reduces hand-fatigue
• Reduces obscuration problems
• Cons
• Increases cognitive load
• Spatial ability comes more into play
• Mouse
• Pros
• Familiarity
• Wide availability
• Low cost
• Easy to use
• Accurate
• Cons
• Time to grab mouse
• Desk space
• Encumbrance (wire), dirt
• Long motions arent easy or obvious (pick up and
  replace)
• Consider, weight, size, style, of buttons,
  force feedback

14
Indirect-Control Pointing

• Trackball
• Pros
• Small physical footprint
• Good for kiosks
• Joystick
• Easy to use, lots of buttons
• Good for tracking (guide or follow an on screen
  object)
• Does it map well to your app?
• Touchpoint
• Pressure-sensitive nubbin on laptops
• Keep fingers on the home position

15
Indirect-Control Pointing

• Touchpad
• Laptop mouse device
• Lack of moving parts, and low profile
• Accuracy, esp. those w/ motor disabilities
• Graphics Tablet
• Screen shot
• comfort
• good for cad, artists
• Limited data entry

16
Comparing pointing devices

• Direct pointing
• Study Faster but less accurate than indirect
  (Haller 84)
• Lots of studies confirm mouse is best for most
  tasks for speed and accuracy
• Trackpoint lt Trackballs Touchpads lt Mouse
• Short distances cursor keys are better
• Disabled prefer joysticks and trackballs
• If force application is a problem, then touch
  sensitive is preferred
• Vision impaired have problems with most pointing
  devices
• Use multimodal approach or customizable cursors
• Read Vanderheiden 04 for a case study
• Designers should smooth out trajectories
• Large targets reduce time and frustration

17
Example

• Five fastest places to click on for a
  right-handed user?

18
Example

• What affects time?

19
Fittss Law

• Paul Fitts (1954) developed a model of human hand
  movement
• Used to predict time to point at an object
• What are the factors to determine the time to
  point to an object?
• D distance to target
• W size of target
• Just from your own experience, is this function
  linear?
• No, since if Target A is D distance and Target B
  is 2D distance, it doesnt take twice as long
• What about target size? Not linear there either
• MT a b log2(D/W 1)
• a time to start/stop in seconds (empirically
  measured per device)
• b inherent speed of the device (empirically
  measured per device)
• Ex. a 300 ms, b 200 ms/bit, D 14 cm, W 2
  cm
• Ans 300 200 log2(14/2 1) 900 ms
• Really a slope-intercept model

20
Fittss Law

• MT a b log2(D/W 1)
• a time to start/stop in seconds (empirically
  measured per device)
• b inherent speed of the device (empirically
  measured per device)
• Ex. a 300 ms, b 200 ms/bit, D 14 cm, W 2
  cm
• Ans 300 200 log2(14/2 1) 900 ms
• Question If I wanted to half the pointing time
  (on average), how much do I change the size?
• Proven to provide good timings for most age
  groups
• Newer versions taken into account
• Direction (we are faster horizontally than
  vertically)
• Device weight
• Target shape
• Arm position (resting or midair)
• 2D and 3D (Zhai 96)

21
Very Successfully Studied

• Applies to
• Feet, eye gaze, head mounted sights
• Many types of input devices
• Physical environments (underwater!)
• User populations (even retarded and drugged)
• Drag Drop and Point Click
• Limitations
• Dimensionality
• Software accelerated pointer motion
• Training
• Trajectory Tasks (Accot-Zhai Steering Law)
• Decision Making (Hicks Law)
• Results (what does it say about)
• Buttons and widget size?
• Edges?
• Popup vs. pull-down menus
• Pie vs. Linear menus
• iPhone/web pages (real borders) vs. monitormouse
  (virtual borders)
• Interesting readings

22
Precision Pointing Movement Time

• Study Sears and Shneiderman 91
• Broke down task into gross and fine components
  for small targets
• PPMT a b log2(D/W1) c log2(d/W)
• c speed for short distance movement
• d minor distance
• Notice how the overall time changes with a
  smaller target.
• Other factors
• Age (Pg. 369)
• Research How can we design devices that produce
  smaller constants for the predictive equation
• Two handed
• Zooming

23
Novel Devices

• Themes
• Make device more diverse
• Users
• Task
• Improve match between task and device
• Improve affordance
• Refine input
• Feedback strategies
• Foot controls
• Already used in music where hands might be busy
• Cars
• Foot mouse was twice as slow as hand mouse
• Could specify modes

24
Novel Devices

• Eye-tracking
• Accuracy 1-2 degrees
• selections are by constant stare for 200-600 ms
• How do you distinguish w/ a selection and a gaze?
• Combine w/ manual input
• Multiple degree of freedom devices
• Logitech Spaceball and SpaceMouse
• Ascension Bird
• Polhemus Liberty and IsoTrack

25
Novel Devices

• Boom Chameleon
• Pros Natural, good spatial understanding
• Cons limited applications, hard to interact
  (very passive)
• DataGlove
• Pinch glove
• Gesture recognition
• American Sign Language, musical director
• Pros Natural
• Cons Size, hygiene, accuracy, durability

26
Novel Devices

• Haptic Feedback
• Why is resistance useful?
• SensAble Technologys Phantom
• Cons limited applications
• Sound and vibration are easier and can be a good
  approximation
• Rumble pack
• Two-Handed input
• Different hands have different precision
• Non-dominant hand selects fill, the other selects
  objects
• Ubiquitous Computing and Tangible User Interface
• Active Badges allows you to move about the house
  w/ your profile
• Which sensors could you use?
• Elderly, disabled
• Research Smart House
• Myron Kruger novel user participation in art
  (Lots of exhibit art at siggraph)

27
Novel Devices

• Paper/Whiteboards
• Video capture of annotations
• Record notes (special tracked pens Logitech
  digital pen)
• Handheld Devices
• PDA
• Universal remote
• Help disabled
• Read LCD screens
• Rooms in building
• Maps
• Interesting body-context-sensitive.
• Ex. hold PDA by ear phone call answer.

28
Novel Devices

• Miscellaneous
• Shapetape reports 3D shape.
• Tracks limbs
• Engineer for specific app (like a gun trigger
  connected to serial port)
• Pros good affordance
• Cons Limited general use, time

29
Speech and Auditory Interfaces

• Theres the dream
• Then theres reality
• Practical apps dont really require freeform
  discussions with a computer
• Goals
• Low cognitive load
• Low error rates
• Smaller goals
• Speech Store and Forward (voice mail)
• Speech Generation
• Currently not too bad, low cost, available

30
Speech and Auditory Interfaces

• Bandwidth is much lower than visual displays
• Ephemeral nature of speech (tone, etc.)
• Difficulty in parsing/searching (Box 9.2)
• Types
• Discrete-word recognition
• Continuous speech
• Voice information
• Speech generation
• Non-speech auditory
• If you want to do research here, lots of research
  in the audio, audio psychology, and DSP field you
  should understand

31
Discrete-Word Recognition

• Individual words spoken by a specific person
• Command and control
• 90-98 for 100-10000 word vocabularies
• Training
• Speaker speaks the vocabulary
• Speaker-independent
• Still requires
• Low noise operating environment
• Microphones
• Vocabulary choice
• Clear voice (language disabled are hampered,
  stressed)
• Reduce most questions to very distinct answers
  (yes/no)

32
Discrete-Word Recognition

• Helps
• Disabled
• Elderly
• Cognitive challenged
• User is visually distracted
• Mobility or space restrictions
• Apps
• Telephone-based info
• Study much slower for cursor movement than mouse
  or keyboard (Christian 00)
• Study choosing actions (such as drawing actions)
  improved performance by 21 (Pausch 91) and word
  processing (Karl 93)
• However acoustic memory requires high cognitive
  load (gt than hand/eye)
• Toys are successful (dolls, robots). Accuracy
  isnt as important
• Feedback is difficult

33
Continuous Speech Recognition

• Dictation
• Error rates and error repair are still poor
• Higher cognitive load, could lower overall
  quality
• Why is it hard?
• Recognize boundaries (normal speech blurs them)
• Context sensitivity
• How to wreck a nice beach
• Much training
• Specialized vocabularies (like medical or legal)
• Apps
• Dictate reports, notes, letters
• Communication skills practice (virtual patient)
• Automatic retrieval/transcription of audio
  content (like radio, CC)
• Security/user ID

34
Voice Information Systems

• Use human voice as a source of info
• Apps
• Tourist info
• Museum audio tours
• Voice menus (Interactive Voice Response IVR
  systems)
• Use speech recognition to also cut through menus
• If menus are too long, users get frustrated
• Cheaper than hiring 24 hr/day reps
• Voice mail systems
• Interface isnt the best
• Get email in your car
• Also helps with non-tech savvy like the elderly
• Potentially aides with
• Learning (engage more senses)
• Cognitive load (hypothesize each sense has a
  limited bandwidth)
• Think ER, or fighter jets

35
Speech Generation

• Play back speech (games)
• Combine text (navigation systems)
• Careful evaluation!
• Speech isnt always great
• Door is ajar now just a tone
• Use flash
• Supermarket scanners
• Often times a simple tone is better
• Why? Cognitive load
• Thus cockpits and control rooms need speech
• Competes w/ human-human communication

36
Speech Generation

• Ex Text-to-Speech (TTS)
• Latest TTS uses multiple syllabi to make
  generated speech sound better
• Robotic speech could be desirable to get
  attention
• All depends on app
• Thus dont assume one way is the best, you should
  user test
• Apps TTS for blind, JAWS
• Web-based voice apps VoiceXML and SALT (tagged
  web pages).
• Good for disabled, and also for mobile devices
• Use if
• Message is short
• Requires dynamic responses
• Events in time
• Good when visual displays arent that useful.
  When?
• Bad lighting, vibrations (say liftoff)

37
Non-speech Auditory Interface

• Audio tones that provide information
• Major Research Area
• Sonification converting information into audio
• Audiolization
• Auditory Interfaces
• Browsers produced a click when you clicked on a
  link
• Increases confidence
• Can do tasks without visual cognitive load
• Helps figure out when things are wrong
• Greatly helps visually impaired

38
Non-speech Auditory Interface

• Terms
• Auditory icons familiar sounds (record real
  world sound and play it in your app)
• Earcons new learned sounds (door ajar)
• Role in video games is huge
• Emotions, Tension, set mood
• To create 3D sound
• Need to do more than stereo
• Take into account Head-related transfer function
  (HRTF)
• Ear and head shape
• New musical instruments
• Theremin
• New ways to arrange music

39
Displays

• Primary Source of feedback
• Properties
• Physical Dimension
• Resolution
• Color Depth and correctness
• Brightness, contrast, glare
• Power
• Refresh rate
• Cost
• Reliability
• of users

40
Display Technology

• Monochrome displays (single color)
• Low cost
• Greater intensity range (medical)
• Color
• Raster Scan CRT
• LCD thin, bright
• Plasma very bright, thin
• LED large public displays
• Electronic Ink new product w/ tiny capsules of
  negative black particles and positive white
• Braille refreshable cells with dots that rise up

41
Large Displays

• Wall displays
• Informational
• Control rooms, military, flight control rooms,
  emergency response
• Provides
• System overview
• Increases situational awareness
• Effective team review
• Old Array of CRTs
• Interactive
• Require new interaction methods (freehand sketch,
  PDAs)
• Local and remote collaboration
• Art, engineering

42
Large Displays

• Multiple Desktop Displays
• Multiple CRTs or Flat panels for large desktops
• Cheap
• Familiar
• Spatial divide up tasks
• Comparison tasks are easier
• Too much info?
• HMD
• Eventually -gt Every surface a pixel

43
Mobile device displays

• Applications
• Personal
• Reprogrammable picture frames
• Digital family portrait (GaTech)
• Business
• PDAs, cellphones
• Medical
• Monitor patients
• Research Modality Translation Services (Trace
  Center University of Wisconsin)
• As you move about it auto converts data, info,
  etc. for you

44
Mobile device displays

• Actions on mobile devices
• Monitor information and alert (calendar)
• Gather then spread out information (phone)
• Participate in groups and relate to individual
  (networked devices)
• Locate services and identify objects (GPS car
  system)
• Capture and then share info (phone)

45
Mobile device displays

• Guidelines for design
• Bergman 00, Weiss, 02
• Industry led research and design case studies
  (Lindholm 03)
• Typically short in time usage (except handheld
  games)
• Optimize for repetitive tasks (rank functions by
  frequency)
• Research new ways to organize large amounts of
  info on a small screen
• Study Rapid Serial Visual Presentation (RSVP)
  presents text at a constant speed (33
  improvement Oquist 03)
• Searching and web browsing still very poor
  performance
• Promising Hierarchical representation (show full
  document and allow user to select where to zoom
  into)

46
Animation, Image, and Video

• Content quality has also greatly increased
• 3D rendering is near life-like
• Digital Photography is common
• Scanned documents
• Video compression
• Multimedia considerations for the disabled
• Printers
• 3D Printers create custom objects from 3D models