Running User Experiments

1
Running User Experiments

• Humeyra Topcu-AltintasDavid GirsaultFeyi Agagu

2
Outline

• Recruiting Participants
• Selecting and Organizing Tasks to Test
• Creating Task Scenarios
• Deciding How to Measure Usability
• Prepare Test Materials
• Prepare Testing Environment
• Prepare Test Team

3
Recruiting Participants
4
Finding appropriate participants

• Problems about wrong participant
• More experienced
• Less experienced
• Where should we go?
• The source, ex new version of existing os -gt
  customer list
• bank teller system -gt banks
• Should we use company employees?
• Tighter security
• Lower costs
• Recruiting is easier
• But, they may not represent the actual users

5
Methods to find a participant

• Working with temporary agencies
• Advertising
• Networking
• Working with professional associations

6
Building a database of qualified participants

• For people who might be appropriate for the types
  of usability test
• Select appropriate people from db to recruit for
  each test

7
Reaching and screening potential participants

• Sending out letters and questionnaires
• Sending initial letter first, questionnaires to
  people who express interest
• By telephone

8

• Introductory letter (to get interest)
• Type of product
• The environment
• How long the test will last
• How much we will pay
• Screening Questionnaire
• Ask only what we need to know
• Ask specific questions
• Ask questions in an order that allows the
  recruiter to stop

9
(No Transcript)
10
(No Transcript)
11
(No Transcript)
12
Deciding who should recruit participants

• Recruiter should be someone who
• understands usability testing
• is enthusiastic about the test

13
Planning how many calls it will take to find a
participant

• More specialized skills -gt More calls
• More stringent criteria -gt More calls

14
Knowing when to recruit and schedule participants

• A few weeks before the test. Because
• people may not know what their work schedule
  will be
• they may forget

15
Knowing what to say when you are recruiting

• If it is the first contact, start the recruiting
  with the subjects of the introductory letter
• If the questionnaire has been already filled out,
  verify the information
• If not, find out quickly if the person
  appropriate or not
• Dont give detailed answers
• Dont assume people show up w/o reminders

16
Arranging for payment or other incentives

• If working with a temporary agency
• It will charge for each participant
• Plus 33 (in general) for themselves
• How much should we pay to the participants?
• The one that motivates people to come
• Not have to be money, maybe gift certificates,
  travel to the test side, samples of the product,
  samples of other products etc.

17
Making sure that participants show up

• 10 no show is common
• What can we do about the people who forget
• Reschedule them
• What can we do to make sure participants come?
• To make the usability test sound interesting,
  nonthreatening
• To confirm the appointment and a day or two
  before the schedule time

18
Selecting and Organizing Tasks to Test
19
Selecting tasks

• Not possible to test every task users can do with
  a product
• What tasks should we sample?
• Tasks that probe potential usability problems
• Tasks suggested from our concerns
• Tasks derived from other criteria
• Tasks that users will do with the product

20

• Set a password for the mail account
• Create a personal mailing list
• Move messages between folders
• Sorts items in folder
• Read any new mail message
• Read a selected mail message
• Create and send a mail message
• Edit a message
• File a mail message
• Create a folder
• Forward a message
• Delete a mail message
• Delete a folder
• Find a mail message
• Send a CC of a message
• Reply to a mail message
• Attach a file to a message
• Create a distribution list

21
Determining the resources you need for each task

• How long it will take to do the task
• What hardware, software, procedures, and other
  resources will we need to run

22
Creating the final list of tasks to test

• After completing task list with their
  requirements, eliminate some.
• While eliminating, we should consider
• The objective of the task, whether we can achieve
  more than one obj. with it
• Whether a task uses expensive resources is worth
  testing
• Whether time consuming tasks are more important
  than two or three tasks

23
Creating Task Scenarios
24
Creating Task Scenarios

• List of tasks for test ? presentation to the
  participants
• Scenario embedding tasks in a short story
• How to create good scenarios?

25
What is a Scenario?

• Best defined by example
• Youve just bought a new combination telephone
  and answering machine. The box is on the table.
  Take the product out of the box and set it up so
  that you can make and receive calls.
• Makes tasks more realistic
• Gives goal, not steps

26
What Makes a Good Scenario?

• Short no time waste on reading
• In the users words easy to understand
• Unambiguous task must be clear enough
• Enough information same data as in a real
  situation
• Directly linked to tasks scenario tests a set of
  tasks

27
Do you Always Give Participants Written Scenarios?

• Written scenarios ? most of the time
• RPG style scenarios ? useful in some cases
  (customers, supervisors, colleagues)
• Written version always necessary

28
How do you Divide up the Tasks and Scenarios for
Participants?

• Most important criteria task measurement
• Time
• Number of errors
• Etc.
• Separating tasks ? separate measurements
• Better precision

29
How do you Make Participants Stop Between
Scenarios?

• Just tell them!
• Only provide scenarios one by one
• For written scenarios, ask the user to stop when
  finished at the end of the instructions

30
Deciding how to measure usability
31
Deciding how to Measure Usability

• Goal Planning the observations and measurements
  for a usability test
• Quantitative goals
• vague ideas ? measurable data
• Reasons for planning
• Focus on important points during test
• Quantitative measures

32
Understanding What You Can Measure

• Performance measures (quantitative)
• Ex time to finish a task, number of wrong menu
  choices
• Counting keystrokes is difficult
• Subjective measures
• Quantitative or qualitative
• Ex user rating the product, collect spontaneous
  comments

33
Matching Measures to your Goals and Concerns

• Performance measures ? quantitative goals
• Ex time requirement ? time measurement
• Commercial software available for collecting data
• Taking notes could be as good
• Measuring positive aspects useful
• Good aspects of the product
• Rewarding for product developers

34
Matching Measures to the Products Stage of
Development

• Performance measures adapted to stage of
  development
• Example 1 avoid time measurements with
  prototypes
• Example 2 avoid measuring usability for
  partially implemented interfaces

35
Setting Quantitative Criteria for Each Measure
and Each Task

• Must set quantitative criteria for usability
• Focus on users
• Use previous data to set goals
• Common sense
• 3 categories of results excellent, acceptable,
  unacceptable ? value ranges
• Possible criteria modification

36
Setting Quantitative Criteria (continued)

• Different criteria for each task
• Measures can change with tasks
• Small amount of time to get used to the
  environment
• System response time ? focus on users

37
Preparing Test Materials
38
The Concept of minimal Risk

• Minimal risk is defined as the probability of
  harm anticipated in the test are not greater than
  what participant encounters in everyday life.
• If there is risk involved
• Eliminate the risk
• Follow the federal policies for such tests
• Dont do the test

39
The Legal Form

• Informed Consent Information
• Explain procedure you will follow
• Explain the purpose of the test
• Explain any risks
• Explain opportunity to ask questions

40
The Legal Form

• Informed consent - Comprehension
• Convey information clearly and completely
• Allow questions so participants can make
  informed decision
• If you rush through It seems unimportant

41
The Legal Form

• Informed consent - Voluntariness
• The participant must be free from coercion
• When participants wish to withdraw remain calm
  and neutral.

42
The Legal Form

• When obtaining consent
• Develop a rapport
• Neutral but friendly tone
• If video tapping have the camera on
• Body language must not show pressure
• Be Professional
• Never put participant at risk

43
The Legal Form

• Obtaining consent - Legal issues
• Non-disclosure Participants cant discuss about
  product.
• Waivers You need participants permission to use
  tapes or questionnaire.
• Participants rights right to be informed, right
  to withdraw.

44
Questionnaires

• You need to anticipate question
• You need to write questions down
• So you can ask same question
• So you will not forget to ask the question
• Put questions in specific and appropriate form

45
Questionnaires

• There are several points where you would want to
  ask participants questions. They are
• Pretest
• Post-task
• Post-test

46
Questionnaires

• Pretest Questionnaires
• Gather background information
• Verify qualifications of participants
• Questions should be one page or less
• How long have you worked with this product?

47
Questionnaires

• Post-task Questionnaires
• To obtain immediate reaction or judgments about
  each task
• Measure changes in perception over time
• Did you use the manual to complete the task?

48
Questionnaires

• Post-test Questionnaire
• To gather judgments and rating after test
• To clarify any ambiguities that may have occurred
• General questions and specific questions
• What did you like least about the product?

49
Training Scripts

• To ensure all participants have the same skill
  level
• To provide participants with training that others
  do not have.

50
Task Scenarios

• Package in such a way that it will allow u to
  observe and record data
• There must be a clear beginning and a clear
  ending.

51
Preparing Testing Environment
52
Preparing Testing Environment

• Arranging equipment and the props
• Preparing the products
• Creating sample data
• Planning for disaster
• Preparing manuals
• Adjusting the video and audio controls
• Preparing data logging materials or software
• Labeling tapes and disks

53
Preparing Test Team
54
Preparing the Test Team

• Number of people to run test
• Three people should be enough!
• Two usability specialists
• One member technical staff
• Some go for five but one should make sure they
  are not tripping over each other

55
Preparing the Test Team

• Can other people participate in the test?
• Observers can view in case they have any doubts
  about the test.
• Should not switch roles during tests?
• No, people should not switch roles because it
  might make the results irregular

56
Preparing the Test Team

• Who should be on the test team?
• Testing is done by usability professional
• Testing is done by product developers
• Testing is done by usability professionals and
  product developers

57
Usability specialists as testers

• Advantages
• They know how to plan and conduct tests
• They know what to look for
• They know how to analyze data
• They become experts at using lab equip.

• Disadvantages
• Not experts in each product
• Not experts in how users for each product do
  their jobs

58
Product developers as testers

• Advantages
• They see users using their products
• They are forced to think about usability
• They already know the product
• Dont have to hire people

• Disadvantage
• Developers must learn new tool
• Developers are not trained observers
• Testing takes time away from work
• The cost may be higher

59
Preparing the Test Team

• What roles are there for test team members?
• Test Administrator
• Briefer
• Camera operator
• Data recorder
• Help desk operator
• Product expert
• Narrator

60
Preparing the Test Team

• There should be checklists for each role
• Test Administrator - Greet Guest observers
• Test Administrator Make sure someone is
  responsible for setting up for next session.
• Briefer - Check participants name
• Briefer Let participants see cameras
• Product expert Bring the product backup

61
THE END

• Questions??? Science Library!