Evaluating the User Interface

1
Evaluating the User Interface

2
Factors to consider

• user characteristics
• type of activities to evaluate
• evaluation environment
• nature of artifact to be evaluated

3
Types of evaluations

• formative evaluation - guides design by providing
  early feedback
• summative evaluation - takes place after produce
  developed, makes judgment on finished product

4
Why evaluate?

• understand the real world
• - requirements gathering
• - prototype checking
• compare designs
• design specifications also
• engineer toward a target
• metric-based targets
• check conformance to a standard
• screen legibility, reading level

5
When to evaluate

• Early evaluations
• predict usability of product or feature
• test understanding of user's requirements
• test ideas quickly and informally (design process)

6
When to evaluate?

• Later evaluations
• identify difficulties, fine tune
• upgrade, improve

7

• What kinds of evaluation were done in the Olympic
  Messaging System and the British air-traffic
  control system?

8
Forte-Travelodge Example

• conducted at IBM Usability Evaluation Center in
  London
• Aims
• identify and eliminate outstanding problems with
  system before going live
• avoid business difficulties during implementation
• ensure that system was easy to use by
  inexperienced staff
• develop improved training material and
  documentation

9
Aspects to evaluate

• system navigation -- how quickly bookings
  completed while answering phone
• screen design -- ease of use, clarity, efficiency
• effectiveness of onscreen help panels, error
  messages
• complexity of keyboard for non-computer literate
  operators
• effectiveness, practicality of training program
• documentation clarity, ease of use

10
The evaluation

• 15 common scenarios developed, enacted by
  cross-section of reception staff
• 8 half-day sessions, several scenarios per
  session, each lasting 2-3 hours
• all were FT personnel (reception, management,
  designers, programmers)
• "evaluating system, not staff"
• remote controlled video cameras
• debriefing sessions

11
Results

• reception and mgmt familiar with system
• 62 usability failures identified

12
Priorities for improvements

• speed of navigation through system
• titles/ screen format problems
• operators unable to find key points in
  documentation
• need for redesigned telephone headsets
• uncomfortable furniture

13
Claimed long-term results

• productivity, occupancy rates increased
• faster bookings, improved match w/ guest
  requirements
• staff turnover low, morale high
• expanded quickly to meet demand 14,500/wk -gt
  27,000/wk
• "reasonable" operating costs/ training times

14
General Methods for UI Evaluation

• Observing and Monitoring usage
• Collecting user's opinions
• Experiments and Benchmarking
• Interpretive Evaluation
• Predictive Evaluation
• Pilot studies

15
Observing and Monitoring Usage

• informal
• lab setting
• participative
• ethnographic
• observer makes notes
• keystroke/interaction logging video

16
Direct Observation

• cheap easy
• "see what you want to see
• single pass / lack of independent permanent
  record
• effect of observation (Hawthorne effect)
• Example EuroATM --
• problems involved in direct observation?

17
Direct Observation

• good for informal
• specific recording technique/shorthand/checklist
  can helpto standardize

18
Indirect observation - video recording

• independent permanent record
• less intrusive (set up and leave in place for
  several days)
• multiple views keyboard, screen, user -gt
  synchronization problems
• how long to run study, on whom to focus??

19
Analysis of video data

• time-consuming in-depth analysis can take 51
• task-based analysis how did users approach task,
  where do difficulties lie, what can be done to
  remedy?
• performance-based analysis frequency of correct
  task completion, task timing, use of commands,
  frequency of errors, etc.

20
Automated support

• European ESPRIT project -gt MUSiC suite of tools
• U. of Toronto - VANNA -gt Mac, powerbook, 8mm
  Sony VHS recorder real-time and post-mortem
  annotation, playback, searching, etc.

21
Verbal protocols

• user's spoken observations plans, feelings,
  knowledge of interface objects -gt think aloud
  protocol
• pairs of users
• prompting
• post-event protocols

22
Software Logging

• time-stamped keypresses
• interaction logging
• may be combined with audio/video
• replay, annotation, automated analysis
• expensive

23
Collecting user's opinions Interviews

• Goal flexible data gathering .. elicit info
• Structured interviews predetermined questions
  asked in a set way
• Flexible interviews set topics, but no set
  sequence, interviewer
• can follow up on interviewee's replies
• less formal -gt good for requirements gathering

24
Interviews

• need to make interviewee comfortable, willing to
  talk about problems, non-threatening environment
• semi-structured set of questions that can be
  used to move interviewee toward goal
• prompted interviewing
• "card sorting
• "twenty questions" (all T/F)

25
Collecting user's opinions Surveys

• Goal preparation of unambiguous, unbiased
  questions
• pilot study important
• open questions respondent supplies her own
  answer
• closed questions respondent asked to select
  response from set of replies

26
Surveys

• 3-point scale
• yes / no / don't know
• multi-point scale
• 6 5 4 3 2 1
• very of no
• useful use
• Likert scale - given a statement
• strongly agree slightly neutral slightly
  disagree strongly agree agree disagree
  disagree

27
Surveys

• semantic differential - given bi-polar adjectives
• extremely quite slightly neutrual slightly quite
  extremely
• easy difficult
• clear confusing
• fun boring
• ranked order
• Rank the following in order of usefulness(1-4),
  (1 is most useful)
• Paste
• Duplicate
• Group
• Clear

28
Surveys analysis

• means, std deviations
• statistical packages (SPSS - Statistics Package
  for Social Sciences)

29
Surveys Goals

• clear, unambiguous, unbiased
• increase chances of completion return
• short
• send copy of report
• conferences -gt small favor (pen, mouse pad, etc.)

30
Experiments and Benchmarking

• Modified scientific method
• measurement important
• not possible to control all variables except
  independent variable

31
Examples

• Depth of menus
• icon design
• text selection schemes
• 6 selection schemes
• 4 subjects per group, 2 with mouse experience, 2
  without
• trained in use of mouse, S/W
• taught the selection scheme
• 10 tasks performed
• selection times, errors measured

32
Variables

• Independent variable variable the experimenter
  manipulates (stimulus)
• Dependent variable output or response that is
  measured

33
Experimental Design

• Independent subject design
• randomly allocated to group
• Matched subject design
• matched in pairs and randomly assigned to groups
• Repeated measures design
• all subjects in both groups -- ordering must be
  randomly assigned
• Single subject design
• one subject studied in depth

34
Problems

• Learning during study
• statistical validation of small groups
• experimental effects -- change in the dependent
  variable that is caused by a change in the
  independent variable learning between tasks,
  fatigue, boredom

35
Critical review of exp. procedure

• User preparation
• adequate instruction, sufficient practice
• Impact of variables
• how change in independent variable affects user
• Structure of the tasks
• sufficiently complex? Did users understand goal
  of task?
• Time taken
• fatigue, boredom ?

36
Critical review of exp. results

• Size of effect
• absolute size of differences and practical
  implications
• Alternative interpretations
• other, uncontrolled, variables at work?
• consistency between dependent variables
• time v. error rate
• task completion v. learning
• generalization of results

37
Usability Engineering

• Define usability goals through metrics
• Set planned levels of usability that need to be
  achieved
• Analyze impact of design choices
• Incorporate user-derived feedback in product
  design
• Iterate design-eval-design until goals achieve

38
Metrics spec

• See chart on p. 651

39
Also

• Benchmark tests Carefully constructed standard
  tests
• Attitude metrics surveys and interviews
• Tradeoffs impact analysis

40
Interpretive Evaluation

• Contextual Inquiry
• Cooperative evaluation
• Participative evaluation
• Ethnography

41
Contextual Inquiry

• Form of elicitation that can be used in
  evaluation
• users work in natural environment or at home
• no metrics, but video clips and interpretations
  are sent to design team

42
Of Interest

• Structure and language used in the work
• individual and group actions and intentions
• culture affecting the work
• explicit and implicit aspects

43
Evaluators should

• Get as close to the work as possible
• uncover hidden work practices
• create interpretations with customers
• let customers expand scope of discussion

44
Cooperative Evaluation

• A technique to improve user interface
  specification by detecting the possible usability
  problems in an early prototype or partial
  simulation
• users help select tasks, issues
• representative tasks selected
• think aloud protocols used
• debrief, summarize, report

45
Participative Evaluation

• Part of participative design
• focus groups work w/designers
• designers coordinate, prepare prototypes
• early prototypes used to explore social/political
  issues that might arise in workplace

46
Predictive Evaluation