Lecture 3: UCD 0: Evaluating Existing Technologies

1
Lecture 3UCD 0 Evaluating Existing Technologies
Human-Computer Interaction Autumn
2006-07 Department of Informatics, University of
Sussex
Lecturer Dr Greg Hooper Convener Dr Geraldine
Fitzpatrick
2
This lecture will...

• Introduce you to ways to evaluate technologies
  that are already out there...
• Testing with experts
• Testing with users

3
Why evaluate existing technologies?

• As part of UCD
• Understanding issues/experiences with existing
  technologies can help inform the design of new
  technologies or the re-design of the current
  technology
• For you personally
• To develop a battery of analytic techniques that
  help you decide whether a technology is good or
  bad
• To develop critical faculties of use in your own
  design activities
• To enable you to suggest design changes grounded
  in structured analysis

4
Overview REDO

• Usability goals (recap)
• Usability principles
• User experience goals
• Testing with experts
• Heuristic evaluation
• Testing with users
• User testing methods
• Interface design guidelines/approaches
• Other methods and techniques

5
Recap usability goals user experience goals
6
Usability goals (recap)

• effectiveness, efficiency, safety, utility,
  learnability, memorability
• a technology can be evaluated by applying these
  goals how far does it reach those goals?
• requires specific scenarios of use to be effective

7
Usability Principles (recap)

• Don Norman visibility, feedback, constraints
  (physical, logical, cultural) mapping
  consistency affordance (also known as design
  principles)
• Dix et al Learnability
• predictability, synthesizability,
  familiarity, generalizability, consistency
• Flexibility
• dialog initiative, multi-threading, task
  migratability, substitutivity,
  customizability
• Robustness
• observability, recoverability,
  responsiveness, task conformance
• again, application needs specific scenarios of
  use for evaluation to be effective

8
User Experience Goals

• How does it feel to interact with a technology?
• satisfying
• enjoyable
• fun
• entertaining
• helpful
• motivating
• aesthetically pleasing
• supportive of creativity
• rewarding
• emotionally fulfilling
• Rogers et al (2002) Interaction Design. pp18-20

9
How do usability goals relate to user experience
goals?
fun
emotionally fulfilling
satisfying
effectiveness
usability goals
memorability
rewarding
enjoyable
efficiency
supportive of creativity
entertaining
learnability
safety
utility
aesthetically pleasing
helpful
motivating
Rogers et al (2002) Interaction Design. p19
10
Useful/usable/desirable

• the useful/usable components are about usability
  goals the desirable component is about user
  experience goals
• the useful/usable/desirable rubric is useful for
  remembering that these two kinds of goals should
  both be reached

useful
usable
desirable
11
Evaluation with experts heuristic
evaluationcognitive walkthrough
12
Heuristic Evaluation (HE)

• Developed by Jakob Nielsen
• (see Rogers et al 26-7 Dix et al 394-6)
• Heuristic is a rule of thumb or form of guidance
• (Note overlap with design principles)

13
HE Nielsens Heuristics

• 1 visibility of system status
• 2 match between system and real world
• 3 user control and freedom
• 4 consistency and standards
• 5 error prevention
• 6 recognition rather than recall
• 7 flexibility and efficiency of use
• 8 aesthetic and minimalist design
• 9 help users to recognize, diagnose and recover
  from errors
• 10 help and documentation

14
HE discount usability method

• Discount because
• Cheap
• No special labs or equipment
• Can be performed on working UI or sketches
• Fast easy to use
• On order of 1 day to apply
• Can be taught in a number of hours
• Helps find usability problems in a UI design
• Small set (3-5) of expert evaluators examine UI
• Each individually checks for compliance with
  heuristics/usability principles
• Different evaluators will find different problems
• Evaluators only communicate afterwards

15
HE(1) Visibility of system status

• Are users kept informed about whats going on? Is
  appropriate feedback provided within reasonable
  time about a users actions?
• what about this?

16
HE(2) Match between system and real world

• Is the language used based on users
  understandings and experience?

17
HE(3) User control and freedom

• Are there ways for users to escape from unwanted
  situations?
• e.g. accidentally opening an application in
  Windows
• e.g. resizing your DVD image by mistake and no
  longer being able to see the subtitles

18
HE(4) Consistency and standards

• Users should not have to wonder whether words,
  situations or actions mean something different in
  different situations
• e.g. /- buttons on remote
• e.g. file not found search for files or
  folders

19
These are labels with a raised appearance. Is it
any surprise that people try and click on them?
From Saul Greenberg
20
HE(5) Error prevention

• Make it difficult to make errors. Even better
  than a good error message is a design that means
  it doesnt happen in the first place
• e.g. enter date DD/MM/YY
• 01/11 2004
• please enter date in correct format
• repair

/ /
21
HE(6) Recognition rather than recall

• Make objects, actions and options visible. The
  user should not have to remember information from
  one part of the interaction to another.
  Instructions for the use of the system should be
  visible or easily retrievable whenever
  appropriate

22
HE(7) Flexibility and efficiency of use

• Allow users to tailor frequent actions. Have
  accelerators (i.e. shortcuts) been provided that
  allow experienced users to carry out actions more
  quickly?
• e.g. (menu) select-edit-copy/select- edit-paste
  
• CTRLC, CTRLV

23
HE(8) Aesthetic and minimalist design

• Clearly organised information, avoiding
  irrelevant information attractive design

24
HE(9) Recovery from errors

• Are error messages clear and useful? Do they
  suggest a solution?

25
HE(10) Help and documentation

• Is help information available that can be easily
  searched and easily followed?

26
HE example problem report descriptions

• Eg Typography uses mix of upper/lower case
  formats and fonts
• Violates Consistency and standards (HE4)
• Slows users down
• Probably wouldnt be found in user testing
• Fix pick a single format for entire interface
• Reports also often include
• Severity ratings
• Frequency
• impact

27
Cognitive Walkthrough

• Alternative expert/practitioner evaluation
  method
• Involves walking through a task with the system
  as if you were the user and noting problems
• (Could also do Guidelines Walkthrough)
• Steps
• Identify typical users and tasks and set up clear
  sequence of actions to complete task
• Evaluator walks through the action sequence for
  each task
• As the walk through is done, record problems and
  issues and design changes to be made

28
Brainstorm

• The advantages of testing with experts are to do
  with time, effort and so on
• What do you think are the disadvantages of
  testing with experts?

29
Evaluation with users
30
Evaluation with users

• Bigger investment of time and effort
• Select users as close as possible to real users
• If real users are scarce try surrogates
• Keep close to the real tasks
• May need to shorten some for time
• Remind users you are not testing them - they are
  testing the interface
• Issues of informed consent, how to recruit
  participants etc
• (will be covered in a later lecture)

31
Where, what to test

• User testing Where?
• In lab
• Adv more controlled disadv not authentic
  user setting
• In field natural setting
• Adv see the interface used in real setting
  Disadv cant control
• User testing What?
• The interface you are going to re-design
• Competitor or other similar applications

32
Evaluation Data

• Objective measure or model performance on
  typical tasks
• throughput measure of productivity (e.g. number
  of commands used, menus navigated, windows
  opened, forms filled in, spreadsheets completed)
• execution time time taken to perform operations,
  e.g. time to select an option, time to recover
  from an error
• accuracy measure of precision, e.g. pie vs list
  menu selection
• error number, when, where, severity, frequency
  of errors
• Subjective measure users views
• opinions and preferences via rating scales,
  questionnaires, interviews
• users questions, problems, reactions during the
  process
• testers impressions, etc

33
Evaluation Methods

• User testing
• Think aloud (Quick Dirty, user test)
• Task-based scenarios (user test)
• Post-task walkthrough (field, user test)
• Eye tracking (user test) etc
• ltthe following will be covered in later lecturesgt
• Observation (field, also in QD, user test)
• Questionnaires, surveys
• Focus groups
• Interviews (field, QD, user test)

34
User Testing (1)

• Controlled investigation of a specific aspect of
  an interface
• Manipulation of one or two independent variables
• e.g. user experience, usability goal
• Measure dependent variables
• time taken, errors made
• Postulate hypotheses about the different
  interfaces
• e.g. pie menus aremore efficient than pull-downs

35
User Testing (2)

• Devise set of tasks that will enable hypotheses
  to be tested
• e.g. get users to select options from different
  menus
• Collect quantitative and/or qualitative data from
  subjects
• e.g. measure time it takes to do tasks and the
  errors made
• Satisfaction
• Analyse data
• Quantitative using means and statistical tests,
  to see if one type of menu takes longer to
  perform tasks, or more errors are committed e.g.
  means are higher for performing tasks using
  pull-down compared with pie menu
• Qualitative looking for patterns of response etc

36
Think-Aloud Method

• Participant performs typical tasks or (selected
  tasks next slide)
• Participant asked to think-aloud
• Literally verbalising thinking while using the
  interface
• What questions and confusions they have
• How they are making decisions, etc
• Tester records session
• Avoids interfering as much as possible
• only when test would end otherwise
• Explain to subject that you wont answer
  questions until end
• as you want to see how they work it through
• Observes body language and facial expressions as
  well
• Occasionally will prompt for clarification
• eg what are you looking for what do you mean
  by that what did you expect to happen

Adapted from Joe Konstan
37
Task-based Scenarios

• Participants can be asked to perform set tasks
• that you know will test specific aspects of the
  interface or specific task scenarios
• Example
• Finding the Lessons Plans - You need to
  incorporate censorship and art into your
  curriculum next month. You have heard that the
  Blanton Museum has some TEKS aligned activities
  for students. How would you find this resource?
• ltfrom http//www.utexas.edu/learn/usability/
  test.docgt

38
Post-task Walkthrough

• Researcher plays back video or transcript with
  participant
• Can be immediately after or delayed
• Participant invited to comment, explain, clarify
  etc
• Useful for finding more about why
• eg why they selected undo menu and not
  something else
• Can you think of circumstances when this would be
  useful?

39
Eye-Tracking Testing

• Technology to support monitoring where subjects
  are looking and for how long
• Challenge easy to direct results
• avoid thinking out loud
• careful presentation of tasks
• careful design to avoid distractions
• ltfind images, video???gt

From Joe Konstan
40
Eye-Tracking Testing

• Static eye tracker
• Head mounted eye tracker

From http//www.noldus.com/site/doc200604006
41
Alternative Methods

• Natural testing conditions
• gather performance data
• video-prompted review
• Two-person tests
• forces thinking aloud through interaction
• Field studies instead of user tests
• consider deployment, logging

From Joe Konstan
42
Gathering Data

• How to get it?
• Log high-level actions
• Log low-level actions
• Log problems
• Work products
• What to get?
• Detailed and statistical usage data
• Example cases

43
Examples

• Consider a Word Processor
• many alternative solutions for commands
• toolbars, menus, keyboard shortcuts
• relative frequencies of commands
• co-occurrence of commands with undo
• document statistics

From Joe Konstan
44
Examples

• Consider a Website
• maps of link traversal rates
• traffic maps
• hidden co-occurrence
• web usage mining
• errors
• apparent rates of back from destinations
• time on page

From Joe Konstan
45
Guidelines for User Testing

• Plan ahead of time
• what data to record
• what instructions to deliver
• what to do if user falls off prototype
• when to provide help, and what help
• Know your objectives
• but never lose sight of the user
• Focus on one type of test at a time
• cant time and use think-aloud
• Always have a pilot study

From Joe Konstan
46
DECIDE evaluation framework

• A framework to help remember
• Determine the goals of the evaluation.
• Explore the questions to be answered.
• Choose the evaluation paradigm and techniques to
  answer the questions.
• Identify the practical issues.
• Decide how to deal with the ethical issues
• Evaluate, interpret and present the data.
• Basili et al, 1994, In Rogers et al, 248-56

47
Final Note

• Here weve talked about evaluating existing
  technologies as might happen at the very
  beginning of a UCD process (hence step 0)
• Many of the same techniques and methods can be
  used as part of iterative evaluation in step 4 of
  UCD e.g., using mock-ups and prototypes instead
  of a fully interactive system
• Key
• Understand the principles of methods and
  adv/disadv
• Make choice of method and adapt as appropriate to
  stage of project and types of user data required

48
Summary

• There are many ways to evaluate what is already
  out there
• Evaluation with experts
• heuristic evaluation, cognitive walkthrough
• Evaluation with users
• user testing
• methods for gathering user data (think aloud
  etc.)
• Issues
• What are the advantages and disadvantages of
  different methods?
• What is right for a particular interaction
  scenario?
• How do they complement each other?
• When are people talking about similar things?
• How could you come up with evaluations that
  appropriately mix/synthesize different approaches
  and methods?

49
Read

• Rogers et al
• Ch 10, 21.2, 12.3, 13.4, 13.5, 14.2, 14.3
• Dix et al
• Ch 9
• Some useful web resources
• The Usability Methods Toolbox http//jthom.best.vw
  h.net/usability/
• Usability Net http//www.usabilitynet.org/home.htm