Analysis and interpretation of user observation evaluation data

1
Chapter 25

• Analysis and interpretation of user observation
  evaluation data

2
Analysis and interpretation

• Analysis just the facts, maam
• usually just the statistical number crunching
• (if you have quantifiable data, that is)
• observation data can be quasi-scientific,
  meaning a different set of analysis tools than
  traditional techniques like ANOVA
• Interpretation opinions
• opportunity to opine about why you think data
  turned out the way it did

3
Collating the data

• Collating the data
• a rather weak section of the book--all theyre
  saying is
• bring pen paper
• dont forget to number your notes with subject
  ids
• maybe type them upgosh!
• there can be a whole lot more to this

4
Collating the data

• Collating the data (other considerations)
• re-tool your app to record quantitative data,
  e.g., time to completion (computers make good egg
  timers)
• automatically store the data in (gasp!) database
  records---can use .csv instead, then read into
  SQL (really helps in teasing out data later)
• see if you can use a stats package to help with
  the analysis (e.g., the R language -- pretty
  powerful will plot stuff too, fairly steep
  learning curve though)
• maybe combine everything into a Makefile script
  (soooo convenient, especially for your advisor )

5
Summarizing the data

• Time spent to recover from an error end time
  - start time
• Uh, thats it? Wait, theres more
• we have not included details or methods for the
  statistical analysis of evaluation data.
• Are they kidding? Thats where all the powerful
  insights are! Its next to impossible to publish
  papers without that crucial p lt 0.01

6
Finding usability defects

• Usability defect usability problem in the UI (I
  think we get that by now)
• Review the data to find defects
• Make a table
• So far pretty lame chapter

7
Quantitative Data

• Quantitative data are more eye catching for the
  readersand are often regarded as more objective
  than qualitative data.
• Thats just an awful thing to say (Theyre
  confusing two issues and mixing in marketing
  notions such as pseudoscientific jargon--4 out
  of 5 doctors recommends)
• Quantitative data allows numerical / statistical
  analysis with which you can make convincing
  arguments
• Quantitative data can still be subjective, e.g.,
  after quantifying a questionnaire

8
Stats

• Be particularly careful to avoid statistical
  language such as significant result
• if you dont know what that means )
• Good call here --- never B.S. your way out of a
  poorly designed studyif the data doesnt support
  your hypothesis, tough
• Summarizing quantitative data
• graphs, charts, yadda yadda---crunch the numbers
  first, then worry about the graphics
• descriptive stats mean, median, etc.
• inferential stats statistical significance---that
  s where its at---your claim to a scientific
  study
• you still need to know what that means though

9
Descriptive Stats

• many readers of usability reports either do not
  have statistical training or re too busy to
  challenge your figures.
• Ok, some readers are ignorant
• (5 6 55) / 3 22
• 2 experts, 1 novice -- not enough data to say
  anything substantial but the point is made cant
  just blindly report stats, have to reason about
  the results

10
ANOVA

• A very popular approach based on assumption of
  normal distribution
• Performs mean (and variance) comparisons
• Consider (5 6 55) / 3 22
• Suppose we had a bunch of experts and a bunch of
  novices (say 10 each)
• Do ANOVA to see if the groups differ

11
Simple ANOVA example (using Excels data analysis)
F(1,18) 2452.74, p lt 0.01
Experts performed task significantly faster than
novices (F(1,18) 2452.74, p lt 0.01).
Experts are probably faster than novices due to
their considerable experience.
12
Qualitative Data

• Often adds insights into quantitative results
• Look for trends in user responses, particularly
  in their statements, e.g., this is difficult,
  things like that
• You may have to pay attention to body language,
  facial expressions, etc.

13
Coding scheme?

• How to group comments?
• by chronological order?
• by severity of defect?
• by UI component?
• your certainty of the cause of the comment?
• difficulty of fixing the defect?
• Perhaps a coding scheme is needed?
• any method of assigning a group, number, or label
  to an item of data
• SayI can use this A usability coding scheme
  for eye movement metrics, see CHI 2006 workshop

14
Data (Results) Interpretation

• Summarize data, assign severity rankings
• Recommend changes (similar table)

Task Scenario No. 1 Evaluator John Session 2/25 Start 930am End 1020am
Observation Comments Cause (if any) Severity
User didnt select the right menu item User wasnt sure which menu the item was in The menu name is inappropriate High
15
Write it up

• Academic paper
• abstract
• introduction
• background
• methodology
• results
• discussion
• conclusions future work