Empirical Methods: User Testing I

1
Empirical Methods User Testing I

• Web Interaction Design
• Lecture 6

2
Overview

• Empirical Methods user testing I
• Plan and prepare for a usability test.
• Understand the key factors in conducting an
  effective user test.
• Empirical Methods user testing II
• analysis and reporting, field studies

3
Test Planning

• Planning is the key to a successful user test
• Defining the goals/concerns that inform the test
• Decide who your participants should be
• Recruit participants
• Create task set/scenarios
• Decide how you will measure usability
• Prepare test materials
• Prepare environment
• De-brief team
• Pilot the test

4
Test Goals

• Systems are generally too complex to test all
  user groups and concerns.
• Decide upon what the test goals and or concerns
  will be.
• Concerns must be specific and measurable

5
Identifying Goals and Concerns

• Client specified
• Heuristic Evaluation
• Previous tests of the product or similar products

6
Test Participants

• Primary goal in test participant selection
• Sample user group should be representative of the
  larger population of users as a whole
• User profiles and personas
• Should ideally be done during design but are
  often neglected
• Develop in collaboration with product designers,
  marketing department
• User Characteristics
• Characteristics that all users share
• Individual differences

7
User Characteristics

• Work experience
• General and specific computer experience
• Product experience
• Age and Gender
• Personality Factors
• Cognitive abilities

8
Subgroups

• Subgroups are composed of users who share
  specific characteristics.
• E.g. novice vs. expert frequently referred to in
  the literature
• Define and quantify characteristics
• Subjective measures via questionnaire
• Objective measures via tests or biographical
  assessment

9
How many users?

• Nielsen and Landauer (1993)
• 5 users is enough to find 84 of usability
  problems
• Most often cutback in Discount approaches is test
  user numbers
• There is evidence to counter this model
• Spool and Schroeder (2001) 35 of problems found
  by the first 5 users, serious problems not
  discovered until users 13 and 15
• Molich et al (2004) highest problem discovery
  rate was 48.
• Woolrych and Cockton (2001) argue that it is
  based on flawed assumptions about individual
  differences in problem discovery

10
Implications of the 5 user debate

• What factors contribute to user X finding severe
  usability problems that have not been found by
  users A through W?
• Individual differences
• Task set
• Cant determine apriori how many participants you
  need to discover the majority of problems
• Getting a representative sample of the right
  users is key

11
Recruitment

• Build a test timetable with some slack for no
  shows
• Make the test sound interesting and non
  threatening
• Paying users may affect performance
• Be clear about time commitments
• User screening to get an appropriate sample

12
Tasks and scenarios

• Tasks which tap into potential problems
• Use heuristic evaluation results
• Tasks based on concerns and experience
• Talk to developers
• Draw on experience from other tests
• Tasks that mirror real world use of the software
• Frequent tasks
• Tasks that may be under time pressure
• Critical path tasks

13
Task Resources

• How long will each task take?
• What is an acceptable time to complete the task?
• When making time estimates consider the
  characteristics of your users
• What resources do you need to run the task in
  question?
• Instructions and procedures
• Order the set of tasks.
• Key ones early not all users will complete
• Natural order e.g. create a message before
  editing a message

14
Task Scenarios

• Used to tell participants what they have to do.
  They describe tasks in a way that is less
  artificial.
• They should be
• Brief
• In language that the user can understand
• Unambiguous

15
Scenario examples

• A friend is thinking of taking a degree course in
  computing, you decide to help them but requesting
  a prospectus
• You are back at work after a short break, check
  your email to see how many new messages you have.
  If there is a message from Rupert then reply to
  it.

16
Performance Measures

• Performance measures
• Action counts, errors, speed
• Can be recorded by background software e.g. web
  logging programmes and through observation
• Subjective measures
• Perceptions, opinions, judgements
• Likert scales based on user ratings

17
Measurement techniques

• Screen capture
• Video of user
• Web logging
• Verbal protocols
• Questionnaires e.g. WAMI and SUS
• Observation

18
Test Materials

• Consent Forms
• Pre-test questionnaires
• Post-test questionnaires
• Tasks and scenarios
• Checklist for testing

19
Environment test team

• Test out all necessary equipment.
• Prepare for disaster
• Assign roles
• Briefer
• Test administrator
• Camera operator

20
Usability Laboratory
21
Pilot study

• Reveals problems with method
• Reveals problems with materials
• Conduct the pilot exactly as you would the full
  test
• Use a representative participant
• Change the test accordingly

22
Summary

• Usability test planning involves the following
  issues
• Deciding why to test (Goals and Concerns)
• Deciding on who to test (selection of user group)
• Deciding on the way the test will run
  (Identifying an appropriate task set)
• It also involves thinking about where the test
  should take place and when in the development
  process testing should take place

23
Specific Reading

• Barnum, C. et al. (2003). The "Magic Number 5"
  Is It Enough for Web Testing. CHI, Ft.
  Lauderdale, Florida USA.
• Molich, R., M. R. Ede, et al. (2004).
  Comparative Usability Evaluation. Behaviour and
  Information Technology 23(1) 65-74.
• Spool, J. and W. Schroeder (2001). Testing Web
  Sites Five Users is Nowhere Near Enough. CHI
  '01, Seattle, ACM.
• Woolrych, A. and G. Cockton (2001). Why and When
  Five Test Users Aren't Enough. IHM-HCI 2001.