How to collect data

1
Chapter 23

• How to collect data

2
How to collect data

• This chapter is about the tools/techniques used
  to collect data
• Hang on, lets review what are we collecting?
  Whats being measured?

3
What are we collecting?

• Classify the type of info we want

4
What are we collecting?

• Classify the type of info we want
• Subjective
• Satisfaction qualitative / quantitative (e.g.,
  Likert)

5
What are we collecting?

• Classify the type of info we want
• Subjective
• Satisfaction qualitative / quantitative (e.g.,
  Likert)
• Objective Performance measures
• Efficiency speed
• Effectiveness accuarcy (error rates)

6
What are we collecting?

• Classify the type of info we want
• Subjective
• Satisfaction qualitative / quantitative (e.g.,
  Likert)
• Objective Performance measures
• Efficiency speed
• Effectiveness accuarcy (error rates)
• Objective Process measures
• Eye movements, brain waves, physiological data
  (heart rate, skin response, etc.)

7
Timing and logging

• Performance measures
• e.g., time to task completion
• Start / stop times stopwatch
• Specific events (e.g., time Help was clicked)
  time stamp (best if code is retooled to provide
  this)
• Maybe use logging software?

8
Logging software

• Can be expensive
• Ovo Logger appears to be freedoes it come with
  source? (doesnt look like it)
• Does it provide mechanism for aligning notes or
  comments? (You may want to add comments at
  specific points in time)
• Does it interfere with running program? (e.g.,
  ClearView samples at 15 fps, therefore slows PC
  app down a bit)

9
Talk Aloud

• Subjective Process measures
• User says what s/hes thinking as s/hes doing
• Talk-aloud protocols
• good gives glimpes of what users thinking
• bad may influence performance (probably slow
  user down reduce errors), may be distracting
• Alternative record (log) session, ask questions
  in review what did you do here?
• use cognitive walkthrough questions

10
Note taking

• Take good notes
• Bring paper and pens

Dont use subjects name should be anonymous
Task no. Date Subject No. Evaluator Name Start Time End Time Task no. Date Subject No. Evaluator Name Start Time End Time Task no. Date Subject No. Evaluator Name Start Time End Time
Action Users remarks Observers comments



11
Debrief

• Following session, you may want/need to
• ask user more questions
• explain in more detail what was going on
  (participants may be curious as to what was
  really being tested)
• some users may blame themselves for problems
  during test---this may sound silly but you need
  to be sensitive to this if in doubt blame the
  machine, software, etc.

12
Questionnaires

• May be difficult to design
• did this program meet your expectations?
• some expect crappy program, so yes
• others expect good program, also yes
• badly formed question
• Supplement questionnaires with interviews
• Use pre-designed questionnaires
• SUMI Software Usability Measurement
  Inventory
• WAMMIWebsite Analysis and MeasureMent Inventory

13
SUMI

• Questions such as
• Ill never learn all the features of this
  software
• The instructions are helpful
• I sometimes dont know what to do next
• I would recommend this software to my Mom
• Of course you may want to adapt the questions to
  your study

14
Recording technologies

• Video and audio recording good idea
• But what kind of data is it recording?
• satisfaction (maybe)
• efficiency (only if you look at length of video)
• effectiveness (if you can tell an error is made)
• Doesnt seem to offer any data recording, so
  whats the point?
• process measures perhaps
• dont forget asking for permission to record