Low-Fidelity User Testing

1
Low-Fidelity User Testing

• IS 485, Professor Matt Thatcher

2
Agenda

• Administrivia
• High-fidelity vs. low-fidelity prototyping
• Hand out CD of user testing from past project

3
What is Fidelity?

• High-fidelity (Hi-fi)
• prototypes look like their final product
• computer-based prototypes (e.g., Palm Pilot
  prototype)
• limited functionality so users can actually
  interact with it
• more polished and aesthetically pleasing
• Low-fidelity (Low-fi)
• artists rendition with many details missing
• paper-based prototypes
• quick and inexpensive, provide valuable insights,
  but dont really demonstrate functionality
• think about a fashion designer or an architect

4
Why Use Low-Fi Prototypes?

• Traditional methods take too long
• sketches --gt hi-fi prototype --gt evaluate --gt
  iterate
• Can simulate the prototypes
• sketches --gt evaluate --gt iterate
• sketches act as prototype
• test prototypes in days vs. weeks
• Kindergarten implementation skills
• allows non-programmers to participate

5
Disadvantages of Hi-Fi Prototypes

• Perceptions of the tester/reviewer?
• formal representation indicates finished nature
• users focus on appearance (color, fonts, etc.)
• Time?
• encourages precision
• specifying details takes more time and results in
  fewer design iterations
• Creativity?
• we tend to fall in love with our software
  solutions and our program code

6
Stages of Low-Fi Prototyping

• Initial rough sketches of UI design ideas
• evaluate and iterate (usually w/i design team)
• Storyboards
• explicitly conveys the design model
• evaluate and iterate (often w/i design team)
• Chauffeured prototype
• evaluate (with actual users) and iterate
• When do you stop?
• when you converge on a design model that is
  stable enough to commit to software

7
Evaluating Low-Fi Prototypeswith Real Users

• Constructing chauffeured low-fis
• the materials
• faking the interaction (wizard of oz technique)
• Conducting a low-fi test with real users
• preparing test materials
• conducting the test (various roles)
• debriefing test users
• generating a report
• Evaluate the results and iterate based on user
  feedback

8
The Basic Materials(Chauffeured Prototype)

• Large, heavy, white paper (11 x 17)
• 5 x 8 index cards
• Post-it notes of various size
• Tape, glue stick, correction tape
• Pens and markers (many colors and sizes)
• Overhead transparencies
• Scissors, X-acto knives, etc.
• And more

9
Constructing the Model

• Set a deadline
• dont think too long - build it!!
• its about getting ideas on paper
• Draw a window frame on large paper
• Put different screen regions on cards
• anything that moves, changes, appears/disappears
• Ready responses for any user action
• e.g., have those pull-down menus already made
• Use photocopiers to make many versions

10
Some Examples
11
(No Transcript)
12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
Briefing Room Interactive (BRI)
17
Clarissas Embroidery Website (continued on next
page)
18
(No Transcript)
19
A small part of the prototype for the SALT Center
(UofA)
20
Low-Fi Computer Screen(pretty cool)
21
Scenario materials
Test participants view (with a low-fi calculator
and scan-o-matic)
Computers view
And a printer, too!!!!
22
Constructing the Model With UI Tools / Builders
(but do not code)
23
UI Builder Tools

• Pen and paper
• Visio or Smartdraw
• Software drawing tools
• Microsoft Acess, Paint, PowerPoint, Word, Excel,
  etc.
• If you construct the model with tools
• print out the background screens and moving
  pieces
• the prototype should still be paper-based even
  if you use software to make it look clean
• hand write most of the words

24
Video

• Paper Prototyping A How-To Video
• Nielsen Norman Group.
• Get some ideas and take some notes

25
Testing the Model

• Select your users
• look at user profile for target users
• use a questionnaire to get the people you need
• dont use friends or family
• get at least 3 users to evaluate the system
• Prepare scenarios that are
• typical of the product during actual use
• make prototype support these (small, yet broad)
• use 3 scenarios of varying difficulty in the
  evaluation
• Practice to avoid bugs

26
Conducting a Chauffeured Test

• Four testers (minimum)
• greeter - puts users at ease gets data
• facilitator - only team member who speaks
• gives instructions to the users
• hands the scenarios to be completed to the users
• encourages thoughts, opinions (think aloud
  protocol)
• computer - knows application logic controls it
• always simulates the response, w/o explanation
• observers - take notes recommendations
• Typical session is 1 hour
• preparation, the test, debriefing

27
Conducting a Chauffeured Test

• Greet
• get consent forms filled
• reads greeters script (or orientation script)
• provides brief background of what test is about,
  describes what a low-fi prototype is and its
  importance, assures confidentiality, puts user
  at ease (voluntary, quit at any time, testing UI,
  etc.)
• Test
• facilitator reads demo script hands written
  scenarios to user
• must be clear detailed
• chauffeur the prototypes
• involves user watching while another person
  drives the system
• facilitator keeps getting output from
  participant
• Think Aloud Protocol - what are you thinking
  right now?
• observe -gt no a-ha, laugh, gape, etc.
• possible to make changes to prototype in
  real-time if you want

28
Conducting a Chauffeured Test

• Debrief
• fill out post-evaluation questionnaire
• discuss prepared list of debriefing topics
• ask questions about parts you saw problems
• gather impressions
• give thanks

29
Video

• Paper Prototyping A How-To Video
• Nielsen Norman Group.
• Get some ideas and take some notes

30
Evaluating Results

• Sort prioritize observations
• what was important?
• were there lots of problems in the same area?
• assign a severity rating (0-4 scale) to each
  problem identified or uncovered during the
  testing
• Create a written report on findings
• gives agenda for meeting on design changes
• Make changes iterate
• note that you can make some changes as you test

31
IBMs Prototype Design Labs (Participatory
Design Sessions)
We start with low fidelity prototypes using a
pencil and paper toolkit with paper window
templates as well as colored sticky notes.
Although our prototyping tools are can create
these prototypes as quickly as pencil and paper
can, we find that users are more willing to
propose changes when the prototypes are low
fidelity ones. We then use higher fidelity
prototypes typically using IBM VisualAge. Rather
than discussing what users would want in team
meetings, we simply ask users directly.
(from IBMs Ease of Use Website)
32
Microsoft Corporation

• Prior to specifying the product, teams will
  perform contextual inquiry with usability
  engineers. Early user interface evaluations may
  take the form of paper prototypesMuch early
  testing relies heavily on paper prototypes, which
  are used to iteratively lock in a design or a set
  of features and characters on the screen.
• (from the article, Organizing Usability Work to
  Fit the Full Product Range)

33
Clothes-Shopper(An Example)

• Problem
• contemporary approach to shopping is often slow
  and painful
• people often have problems shopping for clothing
  through catalogs (on-line or otherwise) because
  they are unable to visualize how they will look
  in the clothing
• Design solution
• design a UI that allow shoppers to quickly and
  easily visualize how various combinations of
  clothing will look on them
• Method
• use low-fi prototyping to develop a good, general
  understanding of the strengths / weaknesses of
  the initial UI design w/o spending a large amount
  of time on developing sftwr

34
Three Scenarios to Be Evaluated

• Moderate scenario
• Joe from Berkeley wants to buy a new shirt and
  sports coat to be more fashionable in school. He
  does not have a car today and decides to give the
  Clothes Shopper a try. Help him navigate through
  the system, buy the things he wants and leave.
  Although there is no budget constraint, he wants
  a white plain shirt and a cool brown jacket.
• Moderate - difficult scenario
• Sally is a regular user of the Clothes Shopper
  system. She wants to buy a white gown for an
  upcoming prom but only has 150 as her budget.
  She already has an existing profile and does not
  wish to reenter the same information. Help Sally
  buy the gown and get her on a happy date!
• Beginner scenario
• Joanna is a new user to the Clothes Shopping
  system. She wants to go on the tour first and
  become familiarized with the system before doing
  any shopping. After the tour she wants to window
  shop some blue shorts, but doesn't feel like
  buying anything today. She prefers to try on some
  shorts casually and leave the mall.

35
Procedure

• Greet
• Conduct the test
• facilitator, computer, observers
• Debrief
• Evaluate and analyze
• discuss what went well, what went poorly
• comment on users reactions and comments
• compile into master copy and make suggestions for
  next iteration

36
(No Transcript)
37
(No Transcript)
38
(No Transcript)
39
Some Raw Notes from Observers (1/3)

• Participant 2 Marlon Urias Scenario 1 (Joe
  from Berkeley)
• Moved mouse around at first
• "Is Tours clickable?"
• Confused at mall entrance
• "If you want to buy clothes, do you tour or go
  shopping?"
• Easily finishes login screen
• Claims he doesnt know four of the measurements
  asked for, one of them "maybe" he knows
• Thinks its "good" the male measurement screen
  didnt block the profile setup screen, but was
  bottom part of that screen
• Would be "helpful" if you knew where to get pants
  and shirts. Guesses its the Navigational
  Assistant. Clicks on torso.
• Spent a lot of time at Navigational Assistant,
  confused. Pop-up notes should show up as you move
  around the Navigational Assistant, so you know
  where to click for clothes

40
Some Raw Notes from Observers (2/3)

• After he picks shirts, realizes he needs to
  choose color, assuming program knows he is
  choosing color for shirt.
• Wanted to see what results you would get so far
  before you finished choosing your preferences,
  "Would you click on the red curtain?" Since there
  are graphics, he hopes/assumes you would see
  results there. "Wish I could see it" (wanted to
  see Search Results updated automatically after
  changing one preference)
• Clicked on left page flipper in Search Results
  window looking for clothes not there yet, before
  he hit Search button
• Finally realized hes supposed to hit Search
  button, says the term should change to "Show
  Product" or "See Product" or "See Progress"
• Goes back to change size after seeing Search
  Results
• He hopes preferences are kept track of, it wasnt
  clear.
• "Does preference change update the Search
  Results?" To him, Search button doesnt equal
  update Search Results
• Wishes he could buy the clothes before he sees it
  on Dressing Room model by clicking on Search
  Results thumbnails

41
Some Raw Notes from Observers (3/3)

• Wants to see a big "Buy" button to buy clothes in
  Search Results window without opening Dressing
  Room
• Wanted to delete things from Search Results
  window with right-click
• "Can you decide to buy clothes from the
  thumbnail?"
• Wonders what happens if model wears multiple
  shirts
• Thinks Select All Clothes button chooses
  everything in Search Results
• Confused how to make Dressing Room model wear
  clothes Would rather have right-click menus than
  "make assumptions"
• Doesn't like Search button, "What's the
  difference between search and narrow down
  options?" (which is done in preferences)
• Says that go and previous buttons are annoying.
• Again there is problems with the page tabs, "are
  the numbers sizes or choices?" Recommended to
  avoid page tabs because not intuitive.
• Had confusion with Select All, "does it buy it?"
  "Returns it to the model."
• After opening the Shopping Cart, has no problem
  buying the items. "happy, very intuitive shopping
  cart window."

42
Summary Results (1/2)(or Merged List of Critical
Incidents)

• User knew how to enter system by clicking on
  diagram
• one user had question about what Tour meant
• Users got through logon screen pretty easily
• comments about how much information the user was
  able or willing to reveal (e.g., body weight,
  exact dimensions of clothes, etc.)
• Big confusion at the main menu
• did not know what to do to begin shopping
  (clicked on a lot of areas)
• finally noticed Navigation Assistant after much
  searching and experimentation and learned that
  they needed to click on it to begin
• when selected garment that they wanted, many
  clicked on preferences (colors) thinking they
  would be presented with choices
• specifying search parameters was not intuitive
  and natural in choosing their garment
• began to play with search parameters and figured
  out how to use it to get results

43
Summary Results (2/2)

• Big confusion at the main menu (continued)
• once the user got to the search results, two
  users were often confused on how to get the
  garment of their choice on the model
• they just clicked on the garment and expected it
  to appear on the model
• the design model was to have the user drag the
  garment to the model
• after getting the garments on the model the users
  expected to see a checkout or buy option that
  would send them to a checkout screen
• the designers intended for the users to
  double-click on the shopping cart to do this
  function
• after the users figured out how to get to the
  checkout, they bought the garments and exited the
  system pretty easily
• Final note was that once the user learned how the
  system worked, they were able to quickly run
  through the system to perform the other scenarios

44
Discussion and Recommendations(What to Do Next?)
(1/3)

• Change Begin Tour to Begin Tutorial
• Login screen was too wordy -- the number of
  buttons will be reduced to 3
• New User, Use existing profile, Change Current
  Profile
• Profile setup was fine except, for women, bra and
  chest size were considered redundant (eliminate
  one)
• Navigational Assistant was not intuitive -- maybe
  add text or have menus automatically pop up
  instead of requiring a click
• also need to add dresses and suit to torso
  section of assistant since users asked where
  these would be
• gray out preferences menu initially so user knows
  to click on the navigational assistant first

45
Discussion and Recommendations(What to Do Next?)
(2/3)

• Change preferences menu
• use words Show Clothes instead of Search (not
  very intuitive)
• Change some things about the Search Results
  section
• user did not know whether to click or drag the
  thumbnails to dress the dressing room model.
  They decided to just require a click to minimize
  confusion
• change title from Search Results to Clothes
  Gallery
• Dressing room changes?
• users liked the red curtain that concealed the
  model until a search was performed
• liked the Shopping Cart and Trash Can options
• were very confused by the Select All Clothes
  option
• designers just dropped the idea to reduce
  confusion

46
Discussion and Recommendations(What to Do Next?)
(3/3)

• Dressing room changes?
• so users can just select clothing on the model
  and put it in the shopping cart or trash can as
  desired
• add a Checkout button so the user can go ahead
  and buy what s/he has in his / her shopping cart
  (many users asked if such an option existed)
• A user also requested if it were possible to have
  a counter showing how many pieces of clothing
  were shown in the Search Results window (soon to
  be called the Clothes Gallery)
• this function will be added

47
High-Fi Revisions
48
(No Transcript)
49
(No Transcript)
50
Hints for Low-Fi User Testing(from Waving Magic
Wands)

• Interaction techniques to optimize lo-fi testing
• establish a paper world
• e.g., low-fi computer screen, printer, keyboard,
  etc.
• provide a pointing toy
• use input and output interaction
• prototype all relevant elements
• let test participants design (on the spot)
• facilitate fun!

51
Milestones 2 and 3

• Milestones 2
• look at initial sketches from Milestone 1,
  discuss with team, and converge on a design model
• develop low-fi storyboards to support scenarios
• Milestone 3
• develop a low-fi chauffeured prototype for user
  testing
• identify and recruit 3 evaluators (preferably
  real users)
• prepare and conduct a chauffeured user test
• discuss the results (what test users said and
  did, or -)
• discuss likely changes to design for the next
  design iteration

52
Milestone 3

• Results
• raw data
• what each observer writes down during user
  testing, video taping, audio taping, etc.
• merged list of critical incidents (and severity
  ratings)
• get rid of redundant observations, make
  clarifications, etc.
• textual summary of results
• Discussion and recommendations
• specific recommendations to address potential
  usability problems uncovered during testing

53
Low-Fidelity Prototypes

• Paper-based prototypes
• focus on high-level aspects of UI
• see if the design model matches the user model
• not concerned about details (e.g., color, fonts,
  alignment)
• Low-fis help answer certain questions
• are the various UI screens organized well?
• does the layout of each screen make sense?
• is the navigation natural and intuitive?
• is the wording/terminology user-centered?
• are important tasks missing?
• are the metaphors/mappings effective and
  appropriate?

54
Disadvantages of Low-Fis

• Paper mock-ups don't go far enough
• not realistic interaction with the interface
• how would you show a drag operation?
• how will users interact with I/O devices to be
  used
• doesnt provide realistic response time (how
  measure performance)
• has no functionality
• evaluators may scoff at low-tech prototypes
• little feedback about details (appearance like
  colors, fonts, alignments, etc.) of the design
• dont know if design is technically feasible

55
Summary

• Use prototypes to involve the user
• Advantages of low-fi prototypes
• cheap and quick to develop and evaluate
• requires few technical skills
• leads to fast iterations (and more iterations)
• users and designers less likely to see design as
  fixed

56
E-Groceries Case Study