Empirical Evaluation

1
Empirical Evaluation
Designing user studies
2
Agenda

• Questions
• Exam 2 next Monday
• Project Part 3
• Experimental design
• Wednesday understanding HCI statistics
• Jason Brotherton

3
Project Part 3

• Deciding on final design

4
Project Part 3

• Creating evaluation plan

5
Working on Intuition

• Some techniques use this
• cognitive walkthrough
• heuristic evaluation
• formative exercises rely on this somewhat
• We cannot always predict impact of systems on
  users before the system exists.

6
Why empirical studies?

• Summative Quantitative
• Requires resources
• Wont other methods be more effective and less
  expensive?

7
Experimental design

• Need a system and subjects
• Controlled experiment to prove some relationship
  between users and system.

8
Experimental factors

• Subjects
• Variables
• controlled / independent
• measured / dependent
• Hypothesis

9
Experimental condition

• Determined by levels of independent variables.
• Number of experimental conditions is important.

10
How many subjects?

• Book advice
• at least 10
• Other advice
• 6 subjects per experimental condition

11
Experimental purpose

• Reveal influence of independent variables on the
  dependent variables.
• Eliminate possibility of chance relationship.

12
The null hypothesis

• The independent variables do not influence the
  dependent variables.
• Statistical analysis aims to prove more than
  chance relationship.

13
Confidence statements

• Goal of an experiment is to get gt95 confidence
  in significance of result
• that is, null hypothesis disproved
• OR, there is an influence
• ORR, only 1 in 20 chance that difference occurred
  due to random chance

14
Subjects Experimental Conditions

• Between-groups (randomized)
• Each subject assigned to single experimental
  condition
• Within-groups
• Each subject tested on all conditions

15
Comparison

• Between-groups
• no learning effects
• more subjects needed need careful subject
  selection
• Within-groups
• fewer subjects
• potential learning, so mix ordering of conditions

16
Analyzing data

• Relationship between analysis and experimental
  design
• Look at data
• Save data

17
Results possible

• Is there a difference?
• How big is the difference?
• Accuracy of estimate

18
Advice

• Data quality
• avoid error bias
• Data reliability
• multiple observers, rules for measurement
• Determine central tendency
• mean, trimming, median, variance

19
Confounding factors

• Controlling all the variables