User Testing of WebVoyage

1
User Testing of WebVoyage

• Mary J. Jarvis
• Head, Reference Services
• Bennett Claire Ponsford
• Government Documents Librarian

West Texas AM University
http//www.wtamu.edu/library/usability/
2
Overview

• Why we decided to test
• Overview of usability testing
• Process we followed
• Results
• Changes we made
• What well do different next time

3
(No Transcript)
4
(No Transcript)
5
Problem

• Perception that people had difficulty using the
  online catalog
• Debate within library over what (if any) changes
  to make
• Upgrading to WebVoyage 2001 over the summer of
  2002

6
West Texas AM University

• Member of the Texas AM University System
• 6,800 students
• 2,300 WTOnline students
• 20 of online students taking only WTOnline
  classes

7
Cornette Library

• 1,750,000 volumes and 5,500 current subscriptions
• Web Task Force
• 8 librarians and staff, representing all parts of
  the library
• Makes recommendations about the overall web site
  and implementation of WebVoyage

8
Background

• Summer 2001 testing of WebVoyage
• Preparatory to upgrading to 2000.1
• Watched a class of primarily lower division
  students search the catalog to answer 8 questions
• Students had the best results with keyword
  searching but kept trying to do author, title, or
  subject searches
• Moved keyword search to top of the list of Quick
  Searches (simple search)

9
(No Transcript)
10
(No Transcript)
11
(No Transcript)
12
(No Transcript)
13
(No Transcript)
14
Measuring Usability

• Purpose To see how everyday, untrained people
  use a product
• Can involve card sorting, focus groups, analysis
  of usage logs, cognitive walkthrough, or
  heuristic evaluation
• We focused on a general web-based survey and
  task-based usability testing

15
Surveys

• Strengths
• Finds out what users like, dislike, etc.
• Easy to do
• Not too time consuming
• Weaknesses
• Does not necessarily catch usability problems
• Excess user politeness
• Low user expectation

16
Task Based Usability Testing

• Formal Usability Testing
• Qualitative rather than quantitative
• 6-8 volunteers can catch 80 of usability
  problems according to Jakob Nielsen
• Process
• Select a few volunteers
• Have them perform some typical tasks
• Have them talk about why they make the choices
  they do
• Record what they do and say
• Analyze the results

17
Task Based Testing

• Strengths
• Focuses on real life users
• Finds out how people really use the catalog
• Finds out why people use the catalog as they do
• Weaknesses
• Time consuming
• Can be dependant on facilitators personality
• Results could be manipulated
• Analysis open to interpretation

18
Our Process

• Determine what to test
• Develop tasks and script
• Recruit facilitators and recorders
• Recruit volunteers
• Test volunteers
• Survey volunteers
• Survey general student body

19
Step 1 What to Test?

• Traditional library search as default
• Concern over dumbing down the catalog
• More point-of-use help suggested
• Google approach as default
• Problem of point-of-use help for off-campus
  patrons
• Concern over driving off patrons to other
  information sources

20
Decision

• Switched Guided Keyword Search to default
  search for testing purposes
• Changed name of Simple Search to Advanced
  Search
• Kept the rest the same

21
(No Transcript)
22
(No Transcript)
23
Step 2 Determine Tasks

• Objectives
• To see if volunteers can search using standard
  library options
• To see if volunteers can search by keyword
• To see if volunteers can determine whether they
  have found the right title
• To see if volunteers can interpret holdings
  information
• To see where volunteers look for journals

24
Tasks

• 2 looking for a specific book by author or title
• 2 looking for specific journals by title and then
  interpreting holdings
• 1 looking for a book of poems by an author
• 2 looking for books by subject (1 more
  complicated than the other)
• 1 looking for a specific book with limited
  information (authors first name and subject)

25
Step 3 Recruit Facilitator and Recorder

• Facilitator
• Explains process, gets demographic details, makes
  the volunteer comfortable
• Keeps the volunteer talking about what theyre
  thinking as they work on tasks
• Emphasizes need for their honest opinion
• Recorder
• Makes sure the video equipment is working
• Writes down volunteers steps and comments
• Recruited from Web Task Force and Reference Dept.

26
Step 4 Recruit Volunteers

• Had already asked for volunteers for general web
  site usability testing with less than moderate
  response
• Asked 2 faculty members to assign the task-based
  testing as extra credit
• 5 responses from 1 class
• No responses from another
• Another faculty member did not give extra credit,
  but encouraged his students to take the general
  survey
• Recruited 1 library staff family member

27
Step 5 Testing

• Set up in librarys Lecture Room so we could
  videotape the display
• Facilitator makes the volunteer comfortable and
  encourages him/her through the tasks
• Recorder takes notes
• At the end, the facilitator and recorder leave
  the volunteer to answer the wrap-up survey
• Reward the volunteer

28
Results Demographics

• Grade level
• 1 high school, 3 seniors, 1 graduate student
• Age range
• 1 below 17, 3 17-24 age range, 1 25-35 age range
• Gender
• 2 male, 3 female
• Major
• 1 Biology, 3 Education, 1 undecided
• Library experience
• 3 average, 2 above average or higher
• Computer experience
• 1 average, 4 above average or higher

29
Results Finding a Known Item

• All 5 could find a simple title, 4 on the first
  try
• 4 could find a journal by title, but had
  difficulty interpreting holdings information
• 3 eventually found a complicated title, but only
  1 found it on the first try
• Only 1 successfully found the collection of poems
• None was completely successful determining
  whether we owned the online journal

30
Results Searching by Subject

• 3 successfully found something for the more
  straightforward question
• Did not usually find everything
• Were satisfied with finding just one or two
  titles
• No one successfully found anything for the
  complex question
• Most would not have looked in the catalog for the
  answer anyway

31
Problems Keyword Search

• Usually did not change the default as a phrase
  on the first try
• Worked OK for title searches
• Did not work for author or subject searches
• Use of multiple rows
• 1 used all three rows
• 1 used two rows
• 3 used only the first row

32
Problems Advanced Search

• Some volunteers were intimidated by Advanced
  Search and did not use it
• When they did use it, most did not follow the
  hints, at least on the first try
• Tried to use author or title, but did not
  understand the rules

33
Problem Choosing Keywords

• Most simply did not understand the concept of
  keyword searching
• None used synonyms
• All used only words from the task description
• Only 1 reduced the search to essentials
• Only 1 truncated their search
• Previous experience with Google or not sure

34
Problems Expectations

• Most did not expect to find journals,
  particularly online journals, in the catalog
• Some were intimidated by the library in general
  and expected the online catalog to be difficult
  to use
• Most would not have used the catalog for more
  complicated questions

35
Step 6 Volunteer Survey
36
Volunteer Survey Results

• Overall, I found the easy to use.
• 3 agree, 1 neutral, 1 disagree
• I can usually find what I'm looking for quickly
  .
• 2 agree, 3 disagree
• The wording on the links and graphics are easy
  to understand
• 2 agree, 2 neutral, 1 disagree

37
Volunteer Survey Results

• If I am searching for a journal, I need it to be
  laid out for me I do not want to spend all of
  my time searching for how to find something in
  the library. I will just click on google.
• 1 commented that she frustrated easily and was
  overwhelmed from the beginning
• Help pages
• 1 felt the help pages were too general wanted
  something more like Mr. Clippy
• 1 wanted very specific pop-up, point-of-use help
• 3 thought a tutorial would be useful

38
Step 7 General Use Survey
39
General Use Survey Results

• Only received 17 replies
• Overall, I found catalog easy to use.
• 16 agree or strongly agree, 1 neutral
• I can usually find what I'm looking for on the
  first try.
• 10 agree or strongly agree, 4 neutral, 3 disagree
• The wording on the links and graphics used on the
  online catalog are easy to understand.
• 13 agree or strongly agree, 3 neutral, 1 disagree

40
General Use Survey Results

• When you search a database on the web, such as an
  online catalog, which of the following do you
  prefer to START your search?
• 10 preferred a simple, Google-like interface
• 7 preferred a guided search
• Didnt ask specifically about a tutorial, but no
  one suggested it as a way to improve the catalog

41
Changes Overall Look and Feel

• Added header links similar to those on the main
  library pages
• No one had understood the buffalo with glasses
  anyway
• Standard library logo
• Standard navigation links
• Different layout
• Navigation buttons
• Made the Help button bright red
• Eliminated greyed-out inactive buttons

42
(No Transcript)
43
Changes Catalog Home

• Cleaner layout
• More description, especially of the resources
  available in the catalog
• Separate links to keyword and advanced search
  options
• Changed Patron Information to Your Account

44
(No Transcript)
45
Changes Keyword Search

• Made keyword the default search
• Changed Simple Search to Advanced Search
• Changed connector terminology
• any of these - match any one word
• all of these - match all words
• as a phrase - match exact phrase
• Changed default connector
• as a phrase - match all words
• Changed search field terminology
• Keyword anywhere - Anywhere

46
Changes Keyword Search

• 3 rows changed to 1
• Expanded Keyword Search Hints
• Instructions to leave out common words
• Moved up truncation information
• Established keyword field weighting
• Gives more weight to title and subject fields for
  relevance sorting

47
(No Transcript)
48
Changes Advanced Search

• Set Author as the default but left Keyword with
  Relevance (FT) as an option
• Hints in Search By changed to examples
• lastname, firstname - ex. Austin, Jane
• Terminology more in line with rest of web site
• Internet resources - Recommended web sites
• Internet journals - Online journals
• Titles that can be checked out - Books
• Expanded Advanced Search Hints

49
(No Transcript)
50
Changes Record Display

• Added brief record and made it the default
• Customized journal title display
• Changed column headers
• Changed beginning publication date to beginning
  and ending dates

51
(No Transcript)
52
(No Transcript)
53
Changes Error Messages

• Reworded system error message and linked to
  search tips
• Included help on choosing keywords in search tips
• Reworded no hits error message and linked to
  broadening your search help
• Reworded Your search has been truncated error
  message and linked to narrowing your search help

54
(No Transcript)
55
(No Transcript)
56
(No Transcript)
57
(No Transcript)
58
(No Transcript)
59
(No Transcript)
60
(No Transcript)
61
Other Changes

• Added Quick Search to librarys home page
• Linked to Advanced Search from more search
  options beneath the Quick Search box
• Improved the librarys help screens to help
  patrons more easily determine which username and
  password to use in which circumstances

62
(No Transcript)
63
Initial Response

• Went live late August 2002
• Follow-up testing of just 2 new student hires
• Improvement with subject searches
• Will do more testing later
• Some staff complaints about only having one row
  for the keyword search

64
What Well Do Differently

• Tasks and script
• Ask faculty for actual assignments from which to
  develop tasks
• Start from the librarys home page
• Include more subject/keyword tasks
• Do a better job of pre-testing the tasks

65
What Well Do Differently

• Recruiting volunteers
• Recruit more faculty to offer participation as
  extra credit and/or reward volunteers better
• Test during the regular school year
• Recruit some faculty and staff volunteers as well
• Test more students
• Test a greater variety of students

66
What Well Do Differently

• Get more library staff involved as facilitators
  and recorders
• Ease the work load
• See for themselves where students have problems
• Reduce the chance of unconscious bias
• Capture sound and video better

67
What Well Focus On Next

• Determine how well the current catalog is working
• Investigate how patrons formulate searches
• Determine the best way to direct patrons to the
  most appropriate search
• Determine how to better integrate WebVoyage into
  web site as a whole

68
What Well Focus On Next

• Determine how to best introduce new features
• ILL module
• Simultaneous searching
• Username and password issues
• MyOPAC

69
Conclusions

• Extremely useful to see WebVoyage through
  non-librarian eyes
• Patrons do not understand keyword searching
• Patrons understand author and title, but not the
  rules libraries use to search for them
• Patrons do not differentiate between web site,
  WebVoyage, etc. transition needs to be smooth
  between them

70
Bibliography Books

• Designing Web usability / Jakob Nielsen.
  Indianapolis, Ind. New Riders, 2000.
• Don't make me think! a common sense approach to
  Web usability / Steve Krug. Indianapolis, Ind.
  Que, 2000.
• Homepage usability 50 websites deconstructed /
  Jakob Nielsen Marie Tahir. Indianapolis, Ind.
  New Riders, 2002.
• Usability assessment of library-related web sites
  methods and case studies / edited by Nicole
  Campbell. Chicago, Ill. LITA, 2001.

71
Bibliography Web Sites

• The Alertbox Current Issues in Web Usability
• http//www.useit.com/alertbox/index.html
• Research-Based Web Design Usability Guidelines
• http//usability.gov/guidelines/
• Usable Web
• http//usableweb.com/

72
More Information

• Cornette Librarys Usability Testing
• http//www.wtamu.edu/library/usability/
• Mary Jarvis
• mjarvis_at_mail.wtamu.edu
• Bennett Ponsford
• bponsford_at_mail.wtamu.edu