Interactive Systems Design

1
Interactive Systems Design

• MSc in Communications, Computing and
• Human-Centred Systems
• Dr. Chris Baber

2
Human-Centred Systems

• Dr Chris Baber
• Room N313 Tel. ext. 43965
• Email c.baber_at_bham.ac.uk

3
Aims

• Introduce some basic concepts from Cognitive
  Psychology
• Demonstrate relationships between these concepts
  and interactive systems design

4
Objectives

• After completing this module, students will be
  able to
• Critique user interfaces through the application
  of appropriate concepts from cognitive psychology
• Offer guidelines on how to improve a design

5
Assignment

• Select ANY mobile telephone user interface
• Critically evaluate the user interface in terms
  of
• layout
• use of colour
• Gestalt
• consistency
• support for navigation
• Explain how and why the user interface might
  present problems to users
• Consider the (non-computer) metaphor(s) that can
  be related to the design
• Explain what assumptions you believe that the
  designers have made in terms of the way that
  users interpret the user interface
• Produce a redesign of the user interface that you
  believe is superior to the current version and
  explain why your redesign is better

6
Marking Scheme

• Presentation of system image and screen-shots
  (images and description) - up to 10 marks
• Definition of a good user interface - up to 10
  marks
• Discussion of key topics from cognitive
  psychology - up to 30 marks
• Critique of selected user interface, in terms of
  2 and 3 - up to 25 marks
• Redesign (images and description) - up to 10
  marks
• Why is your redesign superior to the current
  design? - up to 10 marks
• Up to 5 additional marks for presentation of
  report

7
Useful References

• Ji, Y.G. et al., 2006, A usability checklist for
  the usability evaluation of mobile phone user
  interface, International Journal of
  Human-Computer Interaction, 20 (3), pp. 207-231
• Bocker, M. and Suwita, A., 1999, The evaluation
  of the Siemens C10 mobile phone usability
  testing beyond quick and dirty, Proceedings of
  the 43rd Annual Meeting of the Human Factors and
  Ergonomics Society, Santa Monica, CA HFES,
  pp.379-383
• Ziefle, M., 2002, Usability of menu structures
  and navigation keys in mobile phones a
  comparison of the ease of use in three different
  brands, Proceedings of the 6th International
  Scientific Conference on Work with Display Units,
  Berlin, pp.359-361
• Jokela, T. et al., 2006, Methods for Quantitative
  Usability requirements a case study on the
  development of the user interface of a mobile
  phone, Personal and Ubiquitous Computing, 10 (6),
  345-355

8
Timetable
9
1. Introduction
10
Design Principles

• Know your user
• Build on systems that are familiar to your user
• Display available functions
• Ensure Coherence and Consistency
• Visualise states and transitions
• Design shortcuts
• Design Help
• Tolerate user error
• Provide Context
• Manage Focus
• Display Grammar

11
Varieties of Technology

• Technology mediates human action on the world in
  ways that
• Augment
• Correct
• Extend
• Modify
• Disrupt
• human performance and ability

12
Augment / Correct / Extend

• Technologies augment human abilities
• SUBSTITUTION calculators
• REPRESENTATION shopping lists
• Technologies extend human abilities
• CORRECTION spectacles, hearing aids
• increasing RANGE telescope, telephone

13
Modify

• Technology changes
• Task sequence
• Required actions
• Information content
• Example
• Calculate 9 hours of work at 3.50 per hour.
  What if you add another hour?

14
Disrupt

• Changes ways of working
• Requires new ways of thinking or acting
• Interferes with activities
• MicroSoft PaperClip
• PDA in meetings
• Mobile phones in lectures

15
How different models of the user lead to
different assumptions on how to Minimise Human
Error
16
Instant Experts

• You get a new mobile telephone for Xmas, do you
• Read the manual from cover to cover and then
  switch on the phone?
• Switch on the phone and try to use it?
• Switch on the phone and read the manual when you
  have problems?

17
Using tools

• Physical appearance
• Knowledge of use
• Sequence of activity

18
Affordance

• See handle
• Reach out hand
• Grasp handle
• Turn handle
• Pull door

19
The Cooker Problem 1
Which control acts on which ring?
20
The Cooker Problem 2
Which control acts on which ring?
21
Paul Fitts Spatial Compatibility
22
Direction of motion Stereotypes
c
1 2 3 4 5 6 7
a
d
?
?
1 2 3 4 5 6 7
b
Clockwise increase Clockwise right Clockwise
away from control Clockwise increase on scale
23
Clockwise to Increase?
1 2 3 4 5 6 7
24
Conclusions

• We have learned routines for how to use many
  sorts of technology
• We apply these routines automatically
• When the routines succeed, they are reinforced
• When the routines fail, we think about what were
  doing

25
Expectations and Affordances

• People know how technology works
• Out of box experience
• Reading instructions
• Prior knowledge and expectations

26
Automaticity

• Norman and Shallice (1980)
• Fully automatic processing controlled by SCHEMATA
• Partially automatic processing controlled by
  either Contention Scheduling
• Supervisory Attentional System (SAS)

27
Supervisory Attentional System Model
Supervisory Attentional System
Control schema
Trigger database
Perceptual System
Effector System
Contention scheduling
28
Contention Scheduling

• Gear changing when driving involves many routine
  activities but is performed automatically
  without conscious awareness
• When routines clash, relative importance is used
  to determine which to perform Contention
  Scheduling
• e.g., right foot on brake or clutch

29
SAS activation

• Driving on roundabouts in France
• Inhibit look right Activate look left
• SAS to over-ride habitual actions
• SAS active when
• Danger, Choice of response, Novelty etc.

30
Know your user

• When designing any form of user interface, you
  should ask
• What am I trying to help the reader (or user) of
  this display do?
• How does the information I am presenting to them
  help with their thinking or their activity?

31
Seven Stage Action ModelNorman, 1990
GOAL OF PERSON
32
Routes from Intention to Action

• Route 1Automaticity
• Apply known routine
• Compatibility
• Affordance
• Route 2 Formulate plan
• Problem solving
• Situated actions

33
Problem Solving

• A problem is something that doesnt solve easily
• A problem doesnt solve easily because
• you dont have the necessary knowledge or,
• you have misrepresented part of the problem
• If at first you dont succeed, try something else
• Tackle one part of the problem and other parts
  may fall into place

34
Problem Solving

• Representation affects strategy
• More than one possible solution
• Solution limited by boundary conditions
• Active involvement and testing

35
Problem

• SEND
• MORE
• MONEY

36
Functional Fixedness

• Strategy developed in one version of the problem
• Strategy might be inefficient
• XIV X vs. XXXX X
• Convert numerals or just see 4

37
Interacting with Products as Problem Solving

• User has a goal to achieve using the product
• The product offers functions to the user
• The user must select the best function to
  progress towards the goal
• The interaction can be thought of as movement
  through a space of possible actions towards a
  goal
• Selection of the best action will be influenced
  by available functions and their interpretation

38
Situated Actions

• Plans imply a specified sequence of actions
• plans are representations of action
• but some actions are opportunistic, i.e.,
  supported by the current context-of-use
• Suchman (1992) terms these situated actions
• plans are resources for action

39
2. Fundamental of graphical user interface design
40
Information Design can Influence Thinking
John Snow, 1854m Mapping death from cholera, in
London, showed clusters around water pump (and
not near brewery)
41
Attributes affecting utility
Legibility Font design Contrast Illumination Lumin
ance
Conspicuity Graphic design Spatial
coding Temporal coding Shape coding Colour coding
Readability Words Syntax
Display
Mediating User attributes
Psychological process Extraction of
meaning Comprehension
42
Gestalt Principles
43
Density and Clutter

• Density is related to available screen space
• E.g 80 x 24 line display 1920 character spaces
• The proportion of filled spaces Density
• Density averages 25 but rarely exceeds 50

44
Levels of Density
70 density 50 density 30
density
Shneiderman, 1992
45
Reducing Density
Grouping / tabulating Reduce number of words
Reduce number of characters
46
Conclusions

• Understanding basic Gestalt principles helps
  manage focus
• Some objects on a display are more conspicuous
  than others
• Use highlighting sparingly
• Some objects form perceptual groups
• Use this to help design screen layout and to
  minimise risk of confusion

47
3. Link and Layout Analysis

• Link Analysis
• Sequence in which objects are used, or attended
  to, during a task
• Data recorded from eye-movement or from
  observation of product use
• Layout Analysis
• Define relationship between objects in terms of
  functional grouping and sequence of use

48
3. Link and Layout Analysis
49
Eye-Movement Heat Maps
50
Reading

• Saccades and Fixations
• Anticipation and Inferences
• Interpretation

51
Saccades and Fixations
52
Reading from Computer Screens

• Reading Speed
• Computer 20 - 30 slower than paper
• BUT only when
• Refresh rate lt 60Hz
• Resolution lt 640x480 pixels
• Differences essentially related to character
  discrimination
• data-driven perception (see slide 27)

53
Link Analysis Worked Examplescan of initial
layout
B
Cassette door
Radio display
SCAN SEEK
CD
BAND ASPM ME-SCAN
ST DX/LO
A
54
Link Analysis Worked Examplerevised layout
B
BAND ASPM ME-SCAN
ST DX/LO
Cassette door
SCAN SEEK
CD
Radio display
A
55
Layout Analysis Worked Exampleinitial layout
56
Layout Analysis Worked Examplefunctional
grouping
RADIO
CASSETTE
RADIO
57
Layout Analysis Worked Exampleredesign within
functional groupings
Radio display
BAND ASPM ME-SCAN
ST DX/LO
A
SCAN SEEK
CD
58
Context

• When presented with ambiguous stimuli our prior
  knowledge of the world helps us to make sense of
  it

59
Context
60
Interpretation

• Knowledge of what you are looking at can aid
  in interpretation
• JA CKAN DJI
• LLW ENTU PTH
• EHI LLT OFE
• TCH APA ILO
• FWA TER
• Organisation of information is also useful

Gestalt also influences interpretation
61
Concepts

• How do you know a chair is a chair?

A chair has four legsdoes it? A chair has a
seatdoes it?
62
Mental models

• Craik
• Internal representation of external reality
• Every good monitor needs a model of the world it
  is monitoring
• Johnson-Laird
• Strategies of knowledge assimilation

63
Mental Models

• Van Dijk and Kintsch (1983)
• Text processed to extract propositions, which are
  held in working memory
• When sufficient propositions in WM, then linking
  performed
• Relevance of propositions to linking proportional
  to recall
• Linking reveals gist

64
Inferences

• Comprehension typically requires our active
  involvement in order to supply information which
  is not explicit in the text
• 1. Mary heard the ice-cream van coming
• 2. She remembered her pocket money
• 3. She rushed into the house.

65
Inference and Recall

• Thorndyke (1976) recall of sentences from Mary
  story
• 85 correct sentence
• 58 correct inference
• sentence not presented
• 6 incorrect inference

66
Scripts, Schema and Frames

• Schema chunks of knowledge
• Slots for information fixed, default, optional
• Scripts action sequences
• Generalised event schema (Nelson, 1986)
• Frames knowledge about the properties of things
• DESIGN CAN HELP CALL APPROPRIATE CONCEPTS TO
  MIND, e.g., through the use of metaphor

67
VisiCalc
Xerox Star
68
WIMP

• Windows
• Icons
• Menus
• Pointing devices

69
Metaphor - Pros

• WOZNY (1989)
• Metaphors allow users to compare knowledge of the
  world with what is happening in the system, and
  use analogy in initial reasoning with the system
• MARCUS (1998)
• A familiar concept in an unfamiliar environment
  can add appeal and when users are new to a topic
  or application domain, seeing familiar references
  can reduce tension, stress, boredom, confusion,
  anxiety, and alienation, while increasing their
  self-assurance, calm, interest, engagement and
  dependence upon the user interface.

70
Designing with Metaphors
71
Smilowitz, Do metaphors make web browsers easier
to use?http//www.baddesigns.com/mswebcnf.ht
m
72
Integral metaphor (library) Standard
browser interface (no metaphor)
73
Integral metaphor (travel) Composite
metaphor (library travel)
74
Metaphor - Cons

• Those who think that one should use metaphors in
  design are destined to produce crappy designs.
  (DON NORMAN)
• PIRHONEN (2003)
• A metaphor cannot cover the whole domain of the
  referent
• And so limits functionality of the design
  (HARRISON ET AL. 1998)
• Once a person has learned the application, the
  metaphor becomes redundant
• HUDSON (2002)
• Metaphor can lead to inappropriate assumptions on
  error recovery
• COOPER (1995)
• Searching for the magic metaphor can be one of
  the biggest mistakes an interface designer can
  make
• A Metaphor could offer a tiny boost to
  learnability at first but eventually limits
  functionality
• There are not enough metaphors to go around
• Metaphors do not scale well

75
4. Use of Colour and Highlighting
76
Contrast Effects
This text is quite easy to read because the
contrast between figure and ground is high
This text is more difficult to read because the
contrast between figure and ground is lower
This text is very hard to read because the
contrast between figure and ground is minimal
77
Galitz, W.O., 1996, The Essential Guide to User
Interface Design

• Potential problems
• screen backgrounds being more attention grabbing
  than screen data
• overuse of colour as a code (the colour itself
  meaning something to the screen viewer), which
  forces the user to interpret a colours meaning
  before the message it is communicating can be
  reacted to.

78
Galitz, W.O., 1996, The Essential Guide to User
Interface Design

• Use colour to assist in formatting a screen
• relate or tie elements into groupings
• break apart separate groupings of information
• associate information that is widely separated on
  the screen
• highlight or call attention to important
  information by setting it off from other
  information

79
Tufte, E., 1989, Visual Design of the User
Interface

• The border of an active window should be light in
  value (to avoid clutter with other windows), yet
  deeply saturated (to provide a conspicuous
  signal)
• Yellow is the only colour jointly satisfying both
  these conditions, and therefore proves valuable
  for bordering windows
• A good way to avoid colorjunk is to use colours
  found in nature, particularly towards the lighter
  side, such as blues, greys, yellows
• Natures colour are familiar and have a widely
  accepted harmony

80
Apple Computer Inc., 1992, Macintosh Human
Interface Guidelines

• Use of Colour in Windows
• Distinguishes the active window from other
  windows and enhances the appearance of user
  controls on the window frame.
• Scroll bars and title bars are gray
• User controls are coloured to make them more
  apparent
• Inactive window borders are gray to make them
  recede into background and active windows black
  border stands out.
• User can change colours from colour control panel
• User chosen highlight colours will be used in
  windows and dialog boxes.
• Limit the number of Colours
• To maintain consistency with operating system,
  use as few colours as possible.
• Fewer colours results in less visual clutter on
  the screen
• Colours on Gray
• Colours look best against a background of neutral
  gray
• Colours will stand out more if the background and
  surrounding areas are gray black, or white
• Beware of Blue
• Light Blue is the colour most difficult to
  distinguish. Avoided for text, thin lines, and
  small shapes.
• If you want things to be unobtrusive, thought,
  light blue is the perfect colour.

81
Factors that Affect Contrast and Visibility
Wickens et al., 1998, An Introduction to Human
Factors Engineering
82
Data-driven perception

• Activation of neural structures of sensory
  system by pattern of stimulation from environment

83
Theory-driven perception

• Perception driven by memories and expectations
  about incoming information.

84
Exercise
i. X on outside front corner ii. X on
inside back corner
x
85
Visual Illusions
http//www.genesishci.com/illusions2.htm
Rabbit or duck?
Old Woman or Young girl?
86
KEYPOINT

• Perception limits are set by sensory / neural
  mechanisms but beyond these limits, perception
  can be cognitively controlled
• Sensory experiences interpreted in a CONTEXT and
  derive from a variety of sources

87
Conclusions

• Humans constantly seek patterns in the world
  around them
• and use these patterns to impose meaning on what
  they see or hear.
• Good design encourages this search for pattern
• bad design makes people see patterns when none
  are intended.

88
5. Commands and Menus
89
Commands and Icons

• GREP V fileagtfileb

A B
90
Command Languages

• Advantages
• Fast
• Efficient
• Precise
• Concise
• Flexible
• User initiated

• Disadvantages
• Training
• Regular use
• High memory load
• Poor error handling

91
Commands Languages

• Command languages rely on recall
• Draper (1985)
• UNIX users poor recall on written test, but
  excellent recall when using computer
• Black and Moran (1982)
• Learn 8 commands. Performance best for
  infrequent, discriminating words, e.g., insert,
  delete. Random words also good.

92
Knowledge in the HeadKnowledge in the World

• Menus relieve users of the need to remember
  command names, but not of the need to know what
  functions can be performed by some commands
• Mayes et al., 1988
• Knowledge held by users and recalled
• Knowledge prompted by objects in world

93
Command Languages Guidelines

• Kidd (1982)
• Choose memorable, nonconfusable command word
• Use consistent command formats
• Keep command strings short
• Provide help
• Use natural syntax
• Place optional / least used commands at end of
  list
• Ensure useful defaults
• If errors frequent, then revert to computer
  initiated

94
Menus

• Advantages
• Minimal typing
• Low memory load
• Defined structure

• Disadvantages
• Speed
• Screen space
• Poor for data entry
• Computer initiated

95
Menus

• Single menu
• Binary options
• Do you want instructions? Y / N
• Linear sequence
• Series of interdependent menus
• Sequence of forms for printing
• Tree structure
• Semantic network
• 8 items per menu x 2 levels for time and error

96
Tree structure

• Menu Trees
• reduce error rate when compared with random
• Search faster over trees than alphabetic or
  random
• Semantic networks

97
Menu Guidelines

• Task semantics
• Meaningful groupings
• Short cuts
• Items brief
• Consistent terminology
• Headings relevant
• Limit to 8 items

98
Icons

• Concrete vs. Abstract
• Reference and interpretation
• Verbal labels can speed response

99
Icon design

• Marcus (1992)
• Lexical pixel shape, colour, blinking
• Synatatics lines, pattern, size, shape
• Semantics concrete, abstract, part
• Pragmatics legibility, utility, identification,
  recognition
• Dynamics highlighting, selection

100
Icon Guidelines

• Represent object in familiar manner
• Limit number of different icons
• Make icon conspicuous
• Ensure selection made clear
• Ensure icons belong to families

101
Working Memory Experiments
102
Baddeleys (1986) Model of Working Memory
Central executive
Visual Cache
Inner scribe
Phonological store
Auditory word presentation
Visual word presentation
Articulatory control process
103
Slave Systems

• Articulatory loop
• Memory Activation
• Rehearsal capacity
• Word length effect and Rehearsal speed
• Visual cache
• Visual patterns
• Complexity of pattern, number of elements etc
• Inner scribe
• Sequences of movement
• Complexity of movement

104
Recognition vs Recall

• Recall
• Retrieve from memory
• Remembering print command
• Recognise
• To bring back into awareness through prompt
• Recognising print icon
• Recognising print item in menu

105
Recognition vs Recall

• The boxer chewed the meat
• Recall who chewed the meat?
• Recognition did the boxer chew the meat?
• Recognition easier than recall
• More information retrieved under recognition than
  recall

106
Long Term Memory as a Semantic Network
Has Skin Can move Eats Breathes
ANIMAL
Can fly Has Wings Has feathers
BIRD
Has fins Can swim Has gills
FISH
Is Yellow Can sing
CANARY
107
Levels and Reaction time
A canary can sing
A canary can fly
A canary has gills
A canary has skin
Collins Quillian, 1969
A canary is a canary
A canary is a bird
A canary is a fish
A canary is an animal
108
Canaries

• Different times to verify the statements
• A canary is a bird
• A canary can fly
• A canary can sing
• Time proportional to movement through network

109
Spreading Activation

• When searching semantic network, activate paths
  for search
• The active a node, the more easily information
  can be obtained from it
• Semantic Priming Effects

110
Forgetting

• Encoding failure
• Failure of consolidation
• Storage failure
• Disruption by new or existing information
• Associative interference
• Two responses associated with same stimulus
• Retrieval failure

111
6. Navigation in User Interfaces
112
Navigating Menus
113
Hypertext

• Bush (1945)
• Nelson (1963)
• Hyperlinks between objects in document
• Navigation
• Following links through the document

114
Navigation Aids
115
Searching the World Wide Web
http//www.firstmonday.org/issues/issue5_2/choo/f
ig1
116
Choice of Aid x Search Style
Hammond and Allinson (1989)
117
Information Foraging

• Priolli, P. ,2007, Information Foraging Theory
• information scent left on path of search by
  users
• More users on path means more scent
• Users attempt to predict what information will
  obtained from following a given path

118
6. Interaction Devices Entering Data
119
Generic Forms of Interaction

• Select Object
• Drag Object
• Change Orientation of Object
• Enter Data

120
Conversation

• Kevin Want to see that film?
• Brian the uh (500 ms) with the bloke from
  that other um that prison thing, Shaw
• Kevin The Shawshank Redemption
• Tim something married that bird with the
  eyes
• Brian Robbins
• (laughs) Yeah. 'The Green whatsit?
• Kevin OK. Theres one at uh ... (looks at
  watch - 1.2 s) ...
• 20 to?

121
Human Communication

• Units of analysis
• Proposition
• Lack of proper grammar
• Adjacency Pairs
• Question-answer
• Turn-taking
• Over-lapping
• Extralinguistic and Paralinguistic cues
• Back channels
• Gestures

122
Turn Taking

• When it is your turn?
• Subtle cues from partner
• When they are finishing
• When they want to speak
• Unsubtle cues from machine
• Speak after beep
• Respond from list

123
Asymmetrical Conversations

• Telephone
• Short phrases
• Task oriented
• Problem of silence
• Expert / naïve
• 90 of doctors turns are questions
• Negotiation of knowledge level

124
Is speaking to machines natural?

• Turn taking
• Input / Feedback
• Turn taking cues
• Asymmetrical partners
• Domain / World Knowledge
• Goals
• Language abilities
• Interpretation of tokens

125
Is speaking to machines natural?

• Politeness
• You dont say please to a machine
• Social Norms
• What happens when machine violates norms?
• Should synthesised speech sound human?
• What are the norms?

126
Back channelling from machines

• MicroSoft Office Assistant
• Process Control training simulation
• Nodding ATM / TVM

127
Breakdown and Repair

• Redundancy
• Shared knowledge / anticipation
• Focus signals
• Given/new
• Deixis
• Comprehension signals
• Communication signals

128
Eye contact and gaze

• to convey interest
• signal comprehension and communication
• provide sense of engagement
• establish social presence

129
Video tunnel
mirror
camera
monitor
half- silvered mirror
130
7. Interaction Devices Selecting Objects
131
Deixis

• Pointing to things using
• words and gestures
• Please can I have that.
• Please can I have that tart.
• Please can I have that fruit tart.
• Please can I have that strawberry tart.

132
Deixis in HCI

• Point
• Move cursor onto object
• Indicate object (highlighting)
• Select
• Click button
• Manipulate
• Click button

133
Select Object

• Step One Indicate Object
• Enter Objects Name or I.D.
• Type
• Write
• Speak
• Position Selector on Object
• Direct touchscreen, stylus, pen
• Indirect cursor (using mouse, trackball,
  joystick)
• Step Two Confirm Selection
• Press key to confirm
• Press button to confirm
• Tap object to confirm

134
Device Models

• Buxtons 3-state device model

State 2
State 1
State 0
135
Application
Button up
Pen off
State 2
State 1
State 0
Pen on
Button down
drag
select
Out of range
136
Different pointing devices
137
8. Multimodal Human Computer Interaction
138
Divided Attention

• Dual tasks require people to divide attention
• Limited attentional resource that is shared
  between tasks
• Depends on tasks, e.g., talk and drive, sight
  read music and shadow speech

139
Allport et al. (1972)

• Participants presented with essay, either visual
  and auditory
• Recognition test far worse with auditory

Errors in Recognition ()
140
Task Similarity

• Interference when use same stimulus modality
  visual or auditory
• Interference when use same stage of processing
  input central output
• Interference when use same memory codes verbal
  or visual
• Interference when use same response codes
  spoken or manual

141
Practice and Expertise

• Highly practised dual-task performance, e.g., 6
  weeks read take dictation
• Expert pianists can sight-read shadow expert
  typists can touch-type shadow
• Performance strategies
• Reduced demand
• Reduced resource load

142
Contrasting Definitions

• Technological
• Computers can present information using different
  display modes and can receive information from
  different devices
• Human
• Humans can receive information via different
  senses and perform actions using different
  effectors

143
Modalities

• Modality
• Refers to human sensory modalities
• Sight
• Hearing
• Touch
• Smell , Taste
• and response modalities
• Speech
• Manual
• Multimodal system
• Supports HCI using more than one sensory and
  response modality

144
Types of Multimodal System
145
Exclusive

• Separate tasks, each with their own goals
• Performance on tasks interact
• Example
• Surveillance system with camera control
• Enter data control camera
• Data entry delayed by camera control

146
Alternate

• Separate tasks with shared goals
• Performance on tasks interact
• Example
• Surveillance and target identification
• Control camera and zoom-in on targets
• Identify and report targets

147
Concurrent

• Separate tasks, each with their own goals
• Tasks do not affect each other
• Example
• Target identification with vehicle control
• Identify targets manoeuvre vehicle

148
Synergistic

• Separate tasks with shared goal
• Tasks do not affect each other
• Example
• Mark distance on map
• How far from here to here?
• Pen speech

149
Put That There
Put that red triangle

• Spoken commands
• Manual gestures
• Bolt (1980)

there
150
IBM OpenSpace
151
QuickSet
152
Pointing and Speaking

• People do not interact multimodally with
  multimodal interfaces
• Speech and pointing is not dominant
• Around 20 of conversation, and 14 of multimodal
  HCI

153
Sequencing and Timing

• Not all actions are simultaneous
• 99 of pen speech has pen actions first

154
Languages of Multimodal HCI

• Speech place a boat dock on the east, no, the
  west end of Reward Lake
• Pen draws rectangle add dock

155
Future Multimodal HCI

• Development of future multimodal systems
  depends on knowledge of the natural integration
  patterns that typify peoples use of combined
  input modes.
• The design of multimodal systems that blend
  modes synergistically depends on
• properties of different modes
• the information they convey
• how multimodal input is synchronized

156
9. Future Human Computer Interaction
157
Weisers Scales of Interaction

• Inch, foot, yard
• Tab, pad, board

158
Inch Pad

• ParcTAB
• PDA
• Individual
• many
• networkable

159
Foot pad

• Notebooks
• Tablet
• E-paper
• Individual
• several

160
Yard board

• Smartboards
• Plasma displays
• Control mimics
• Public
• sharable

161
Forms of interaction

• Implicit interaction devices
• Sensors
• Awareness
• Recognition technologies
• Natural
• Explicit interaction devices
• Keyboard, pointing

162
WIMP considered fatal?

• Brad Rhodes
• The user has screen real-estate to burn
• too much information on screen
• The user has Fine motor control
• Eye-hand coordination
• Digital information is a primary task

• http//www.hitl.washington.edu/people/grof/VRAIS98
  /Rhodes.html

163
Wearable WIMP

• Clear mapping between user action and goal
• Windows
• Multiple views for multiple functions
• Icons
• Presentation of graphical signs and symbols for
  prompts and commands
• Menus
• Presentation of restricted option set
• Pointing devices
• Object selection and manipulation

164
Windows on Wearables

• Unclutter world
• Minimal information views
• Design display to augment world
• Emergent features
• Design display to support pattern recognition
• Integrate display
• Design display to interact with world

165
WIMP considered fatal?

• Brad Rhodes
• The user has screen real-estate to burn
• too much information on screen
• The user has Fine motor control
• Eye-hand coordination
• Digital information is a primary task

• http//www.hitl.washington.edu/people/grof/VRAIS98
  /Rhodes.html

166
Wearable WIMP

• Clear mapping between user action and goal
• Windows
• Multiple views for multiple functions
• Icons
• Presentation of graphical signs and symbols for
  prompts and commands
• Menus
• Presentation of restricted option set
• Pointing devices
• Object selection and manipulation

167
Windows on Wearables

• Unclutter world
• Minimal information views
• Design display to augment world
• Emergent features
• Design display to support pattern recognition
• Integrate display
• Design display to interact with world

168
Icons on Wearables

• Reduce text
• Reduce information load

169
Text vs. Graphics
Mean Reaction Times Graphical Textual HMD
1.45s 1.85s VDU 1.0s 1.5s
Baber et al., 1999, Mobile Networks and
Applications 4
170
Menus on Wearables

• Clarify options
• Restrict activity
• Access and enable through world activity

171
Emergent Properties
Baber et al., 1999, Interact99
172
Limited Options
Bristow, WECAPC
173
Integrate with world
Baber et al., 2000, Interact00
174
Pointing on Wearables

• Selection on display
• Selection in the world

175
Selection in the World
Baber et al., 2000, Interact00
176
Selection from Display

• Carnegie-Mellon
• Rotary dial selector
• Accelerometer for simple gestures
• Speech recognition

Bass et al., 1997, CHI97
177
Augmenting the World
178
Digital Desk
Real paper on desk is scanned by a camera,
computer images are projected onto the paper
179
Graspable Objects

• BUILD-IT
• Move blocks on table
• Change layout of room on displays

Fjeld 1999
180
GeoSPACE

• Hiroshi Ishii MIT
• Physical objects to manipulate map display

181
Rekimotos InfoBoard

• Pick and place objects using pen
• Move objects between laptops or virtual storage

182
Data Storage

• Data containers from Philips

183
Data tiles

• Magnetic tiles linked to data
• Projection labels tiles

184
Rekimotos Data Tiles
185
Haptic VR

• Provide touch to virtual objects using
  force-feedback devices

186
Haptic Augmentation

• Use real objects to control virtual
  representations

187
Handheld AR

• PDA used to provide views on world
• Views change with PDA movement

188
Information Appliances
189
Domestic Appliances

• Microprocessors in cars, dishwashers, washing
  machines, televisions etc.
• Domestic appliances as embedded systems

190
Norman (1999)

• Information Appliance
• An appliance specializing in information
  knowledge, facts, graphics, images, video, or
  sound. An information appliance is designed to
  perform a specific activity, such as music,
  photography, or writing. A distinguishing feature
  of information appliances is the ability to share
  information among themselves.

D.A. Norman, 1999, The Invisible Computer, MIT
Press, p. 53
191
Pervasive / Ubiquitous

• Invisible Computer
• Embedded Systems
• Networks

192
Information Appliances

• Digital camera
• Capture image
• Immediate (shared) viewing of image
• Download image onto computer
• Edit image
• Email image

193
Three Axioms

• Design Axiom 1 Simplicity
• The complexity of the appliance is that of the
  task not the tool. The technology is invisible.
• Design Axiom 2 Versatility
• Appliances are designed to allow and encourage
  novel, creative interaction
• Design Axiom 3 Pleasurability (fun)
• Products should be pleasurable, fun, enjoyable. A
  joy to use, a joy to own.

D.A. Norman, 1999, The Invisible Computer, MIT
Press, p. 67
194
Current Trends

• Device marriages
• digital camera telephone
• PDA telephone MP3 digital camera
• Problems
• Visibility
• Which device is currently on
• Accessibility
• How to switch devices / modes?
• Functionality
• What functions are actually used?

195
11. Distributed Cognition
196
Using an Abacus for Addition
Heaven beads 5 each

• 6 2 8

Earth beads 1 each
8 2 10
197
Shopping Lists

• Construct list
• Using the process of writing the list to support
  decision making
• Remember to consult list
• Check the list during shopping
• Use the list to help navigate the store
• Reading and interpret list
• Make sure everything is bought

198
Shopping Lists

• Memory aid
• Do we only buy whats on the list?
• Do we buy items not on the list?
• Additional tasks
• Does writing the list create a new task?
• Modified task
• Does using the list change the way we shop?

199
Things that make us smart

• Cognitive Artefacts used to assist everyday
  activity
• Shopping List
• Knotted handkerchief
• Calculator
• Diary

200
Using Lists

• Construct list
• Pre-computation Hutchins
• Remember to consult list
• Checklists in inspection
• Reading and interpret list
• Following checklists accurately

201
Cognitive Artefacts

• Distribute actions across time
• Pre-computation
• Distribute actions across actors
• Distributed cognition
• Change actions required

202
Artefacts Representation

• Surface representation
• Display and maintenance of symbols on a visible
  surface
• Internal representation
• Storage and organisation of symbols
• External representation

203
Mapping

• Naturalness
• Related to directness of mapping
• Related to expertise / familiarity
• Appropriateness
• Information should be appropriate to the task
  (neither more nor less)

204
Distributed Cognition

• Share knowledge and information across people,
  things, actions
• Shopping lists
• Ed Hutchins on calculating ships speed

E. Hutchins, 1990, The technology of team
navigation, In Galegher et al. Intellectual
Teamwork, LEA
205
Calculating Ships Speed

• DRT, (RD/T) using pencil and paper
• DRT, using calculator
• 3-minute rule
• 3-minutes 1/20 of an hour, and 100yds in 1/20
  of a nautical mile
• ? 1500 yds in 3 minutes 15nmph

206
Calculating Ships Speed
4. Nautical Slide Rule
207
Calculating Ships Speed

• Knowledge-in-the-head 1
• 2000 yds nautical mile
• 60 minutes in hour
• DRT
• Transposition of equations
• Knowledge-in-the-head 2
• 3-minute rule schema for shortcut

208
Calculating Ships Speed

• Knowledge-in-the-World
• Nautical slide rule
• Replace calculation with manipulation
• Colleagues
• Draw upon experience of others
• Teamwork
• Plotter
• Bearing taker
• Bearing timer-recorder

209
12. Collaborative Working
210
Traditional HCI

• One User
• One Computer
• One Location
• One Task?

211
Limitations of this view

• Task focus on computer AND on other artefacts,
  e.g., paper, telephone, filing cabinet etc.
• Other artefacts imply more than one location
• Office work often collaborative
• SO computer only part of the work domain and is
  not designed to fit with other parts

212
Personal Computers in the Office

• 1970s development of the desktop metaphor
• Objects on screen analogous to objects in world,
• e.g., folders, files etc.
• Manipulation of objects on screen analogous to
  manipulation of objects in world,
• e.g., open folders, put files into folders
• BUT no link between virtual and real objects

213
Changing HCI

• Many Users
• Communication and Collaboration
• Multiple Access (WWW)
• Many Computing Devices
• Embedded Systems
• PDAs, cell-phones and beyond
• Many Locations
• Physical location (mobility)
• Virtual locations (WWW)
• Many Tasks

214
Computer Mediated Communication (CMC)
215
CMC

• Synchronous
• Speech telephone, videophone, videoconference
• Text SMS, chatrooms, instant messaging
• Asynchronous
• Speech voicemail, answerphone
• Text bulletin boards, newsgroups, computer
  conferencing
• Mixed with other tasks
• Electronic Meeting Room

216
Shared Representations

• Shared Calendars
• Access appointments diary to view or edit
  (depending on privileges)
• Shared Awareness
• Shared Activities

217
Shared Awareness

• Portholes Xerox 1992
• Low resolution images of people in their offices
• Overhearing / overseeing

218
Shared Activities

• Work on same text
• Add comments
• Restructure
• Work on same drawing
• Modify
• Suggest alternatives

219
Collaborative Virtual Environments (CVEs)

• Distributed VR systems
• Individuals collaborate in virtual landscapes to
  share information and work together

220
CVEs

• Meeting places
• Multiple representations
• Shared context
• Shared knowledge, environment, tasks, objects
• Awareness of others
• Communication

221
Reading Virtual Control Room
Virtual Display Boards
Augmented Displays
To Virtual Plant
Avatars
Virtual Monitors
222
Interaction in Virtual Space

• With other actors
• With virtual objects
• With real objects?

MagicBook project
223
Collaborating in real space

• Technology to bring the virtual world to the real
  world
• Shared representations on familiar surfaces
• Surfaces to support interaction

224
RoomWare
DynaWall
GMD-IPSI, Wiege, Wilkhahn
ConnecTable
InteracTable
CommChair
Source Carroll, p. 569
225
Shared Awareness

• Portholes Xerox 1992
• Low resolution images of people in their offices
• Overhearing / overseeing

226
Exercise .1

• 1. Apple 2. North 3. Charlie 4. Nickel 5. Red
• 6. Banana 7. South 8. Fred 9. Cent 10. Green
• 11. Pear 12. East 13. George 14. Dime 15. Yellow
• 16. Grape 17. Wayne 18. Dave 19. Penny 20. Blue
• Make a note of the word numbers and enter into
  table
• How many people wrote West or Orange?
• What pattern do you notice in the words?

227
Exercise .2

• 1. Latch 2. Bream 3. Hot 4. Mayor 5. Jab
• 6. Busk 7. Else 8. Wage 9. Clog 10. Jowl
• 11. Chap 12. Big 13. Smug 14. Duck 15. Trout
• 16. Blot 17. Reek 18. Tape 19. List 20. Skirt
• Make a note of the word numbers and enter into
  table
• What pattern do you notice in the words?
• Did you do as well as with the first list?

228
Exercise .3

• 1. Time 2. House 3. Mit 4. Stab 5. Solve
• 6. Draft 7. Say 8. Off 9. Boil 10. Court
• 11. Greet 12. Slot 13. Hand 14. Dirt 15. Clot
• 16. Stale 17. Out 18. Dumped 19. Stone 20. Dice
• Make a note of the word numbers and enter into
  table
• Did you do as well as with the other lists?