Got Data, Now What? Analyzing Usability Study Results

1
Got Data, Now What?Analyzing Usability Study
Results

• Lynn Silipigni Connaway
• June 26, 2005
• Presented at the ALA 2005 Annual Conference
• Chicago, IL
• LAMA/MAES
• Using Measurement Data for Library Planning and
  Assessment Committee

2
Usability Testing Why?

• Probably the best reason to test for usability
  is to eliminate those interminable arguments
  about the right way to do something. With
  human-factors input and testing, however, you can
  replace opinion with data.  Real data tend to
  make arguments evaporate and meeting schedules
  shrink. (Fowler, 1998, Appendix, p. 283)

3
Usability Testing Definition

• Degree to which a user can successfully learn and
  use a product to achieve a goal
• Research methodology
• Evaluation
• Experimental design
• Observation and analysis of user behavior while
  users use a product or product prototype to
  achieve a goal (Dumas and Reddish, 1993, p.22)
• User-centered design process involving user
  from initial design to product upgrade (Norlin
  and Winters, 2002)
• Approach is to be a servant to the users of a
  system NOT to be subservient to technology
  (Gluck, 1998)
• Goal is to identify usability problems and make
  recommendations for fixing and improving the
  design (Rubin, 1994)

4
Usability Testing Background

• Relatively new methodology (Norlin and Winters,
  2002)
• Origins in aircraft design
• Traced back to marketing
• Development of a product
• Popular in 1980s with widespread access to
  computers
• Initiation of human computer interface usability
  studies
• Evolved from human ethnographic observation,
  ergonomics, and cognitive psychology
• Qualitative and quantitative data

5
Usability Testing Purpose

• Evaluation tool
• Identify problem areas
• Determine the fit of the design to the intended
  users (Norlin and Winters, 2002, p. 5)

6
Usability Testing Suitable Questions

• What is the best layout for a web page?
• How can you optimize reading from PDAs and small
  screen interfaces?
• Which online fonts are the best?
• What makes an e-commerce site difficult to use?
• Can individual personality or cognitive skills
  predict Internet use behavior?
• How can library collection holdings and library
  data be represented geographically?

7
Usability Testing Principles

• Keep the end user in mind
• Achieve superiority through simplicity
• Improve performance through design
• Refine and iterate (Norlin and Winters, 2002,
  p.10)

8
Usability Testing Web Design Criteria

• Links must be consistent and predictable
• Group like things on the same page 
• Be consistent with language
• Most important information should be on the first
  screen
• Provide keywords for quick reading/scanning
• Do not use animation or sounds
• Make links look like links
• Distinguish text from graphics
• Avoid jargon (Spool, 1999)

9
Usability Testing Web Design Criteria

• Ten Usability Heuristics (Nielsen)
• Visibility of system status
• Match between system and the real world
• User control and freedom
• Consistency and standards
• Error prevention
• Recognition rather than recall
• Flexibility and efficiency of use
• Aesthetic and minimalist design
• Help users recognize, diagnose, and recover from
  errors
• Help and documentation

10
Usability Testing Web Design Criteria

• Goals for user-centered design
• Enable users to
• Achieve their particular goals and meet their
  needs
• Move quickly and with few errors
• Create a site that users like
• More likely to perform well on a product that
  provides satisfaction

11
Usability Testing Methodology

• Artificial environment (laboratory)
• Maintain more control
• May provide more specific data on a particular
  feature
• Natural environment
• Better holistic representation of real people
  doing real work

12
Usability Testing Methodology

• Four types of usability tests (Rubin, 1994, p.
  31-46)
• Exploratory test early product development
• Assessment test most typical, either early or
  midway in the product development
• Validation test verification of products
  usability
• Comparison test compare two or more designs
  can be used with other three types of tests

13
Usability Testing Methodology

• Develop problem statements, objectives, and/or
  hypotheses
• Identify and select participants who represent
  target population
• May or may not be randomly selected
• Select test monitor/administrator
• Empathetic
• Impartial
• Good communicator
• Good memory
• Able to follow test structure
• Able to react spontaneously to situations that
  cannot be anticipated
• Allow user time for task
• Dont rescue the user
• Continue with the plan if mistakes occur

14
Usability Testing Methodology

• Design test materials
• Screening questionnaire
• Provides user profile
• Ascertains pretest attitudes and background
  information
• Provides information about participants previous
  knowledge and experience
• Orientation script
• Describes the test to participants
• Aids in understanding the participants
  performance
• Data logger materials
• Data collection instrument for categorizing
  participants actions
• Can note time to match with videotape recording

15
Usability Testing Methodology

• Design test materials
• Non-disclosure and tape consent forms for legal
  protection
• Task list
• List of actions participants will execute
• Desired end results
• Motives for performing task
• Actual observations monitor will record
• State of system

16
Usability Testing Methodology

• Design test materials
• Posttest questionnaire
• All participants asked the same questions
• Gather qualitative information and precision
  measurements
• Debriefing guide
• Structure and protocols for ending the session
• Participants explain things not apparent in
  actions
• Motive
• Rationale
• Points of confusion

17
Usability Testing Methodology

• Test materials and equipment
• Conduct the test
• Represent the actual work environment
• Users are asked to think aloud
• Observe users while using or reviewing the
  product
• Probe
• Controlled and extensive questioning
• Collect quantitative and qualitative data and
  measures
• Record comments or questions about the product
• Observe and document users behaviors

18
Usability Testing Methodology

• Debrief
• Analyze the data
• Diagnose and recommend corrections
• Categorize and identify problems with the product
• Identify solutions
• Qualitative analysis
• Textual notes from debriefing
• Read responses
• Summarize findings

19
Usability Testing Methodology

• Analyze the data
• Quantitative analysis
• Questionnaires
• Screening
• Posttest
• Triangulation to validate findings
• Data from questionnaires, observations, screen
  tracking software, comments, and open-ended
  questions

20
Usability Testing Interpret Data

• Interpret the data
• Five factors for benchmarking the usability of an
  interface (Shneiderman and Plaisant, 2004)
• Time to learn
• Speed of performance
• Rate of errors
• Retention over time
• Subjective satisfaction

21
Usability Testing Interpret Data

• Interpret the data
• Prioritize severity of problems
• Severity ratings (Zimmerman and Akerelrea, 2004)
• Time required to complete task
• Number of users who encountered problem
• Negative impact on users perception of the
  product
• Difficult if 70 of users cannot perform task
• Error criticality Severity Probability of
  Occurrence (Rubin, 1994)

22
Usability Testing Interpret Data

• Usable Web site (Rubin, 1994)
• Usefulness
• Establish whether it does what the user needs it
  to do
• Effectiveness
• Ease of use to achieve the desired task
• Learnability
• Ease of learning application and moving from
  being a novice to a skilled user
• User satisfaction
• Users attitude about the sitehow enjoyable it
  is to use

23
Usability Testing Report Results

• Executive summary
• Report
• Describe methodology
• Who, what, when, where, and how
• Describe how tests were conducted
• Profile users and describe sampling
• Detail data collection methods
• Succinctly explain the analysis
• Provide screen captures
• Include tables and graphs
• Provide examples
• Identify strengths and weaknesses
• Recommend improvements

24
Usability Testing Making the Data Work

• Read report
• Determine what worked and what did not work
• Redesign product/system based upon findings
• May be necessary to conduct another usability
  test

25
Usability Testing Limitations

• Two major limitations (Wheat)
• Reliability
• Testing of users who are not atypical users
• Individual variation within the test population
• Validity
• Test tasks, scenarios of the search processes,
  and testing environment are not accurate
• Results not generalizable to the entire user
  population
• Testing is always artificial (Rubin, 1994, p.27)

26
OCLC WorldMapTM

• Research prototype
• Test geographical representation of WorldCat
  holdings
• By country and date of publication
• For library collection assessment and comparison
• Complement the AAU/ARL Global Resources Network
  project
• Geographically represent library statistical data
  from UNESCO, ARL, Bowker, and others
• Number of libraries by type
• Expenditures by library type
• Number of volumes and titles
• Number of librarians
• Number of users

27
(No Transcript)
28
Usability Testing OCLC WorldMapTM

• Review sample handouts
• Screening questionnaire
• Task list
• Posttest questionnaire
• Executive summary

29
(No Transcript)
30
(No Transcript)
31
(No Transcript)
32
(No Transcript)
33
(No Transcript)
34
(No Transcript)
35
(No Transcript)
36
(No Transcript)
37
(No Transcript)
38
(No Transcript)
39
Usability Testing OCLC WorldMapTM

• Conducted informal usability tests
• Currently redesigning the interface
• Conduct second group of formal usability tests
• Make revisions prior to making publicly available

40
Questions and Discussion

• connawal_at_oclc.org