Experiments

1
Experiments

• Experimental Design
• Experimental Analysis

2
Agenda

• Experimental design
• Variables
• Methods
• Results

3
Experiments

• Utilize classical scientific method of
  hypothesis, experiment and analysis
• Key is methodology

4
Experimental Methodology

• Variables - facets or attributes of study that
  can vary
• We want to control all variables but the ones
  were testing

subject experience gender
interface 1 vs. interface 2 lighting
intelligence location color vs. b/w
etc.
5
Control

• Two methods of achieving it
• Dont allow it to vary
• Make subjects/attributes as representative of
  population as possible
• In both mean and range
• Often, second method is all you can do

6
Types of Variables

• Participants are a random variable
• In experiment, we have independent and
  dependent variables
• Independent - What youre studying, what you
  intentionally vary (eg, interface feature)
• Dependent - What the study produces and you
  tabulate, measure or examine (eg, time, number of
  errors)

7
Example

• Do people complete operations faster with a
  black-and-white display or a color one?
• Independent - color or b/w
• Dependent - time it takes to complete

8
Experimental Designs

• 1. Within Subjects
• Every participant provides a score for all levels
  or conditions

Color
B/W P1 12 secs. 17
secs. P2 19 secs. 15
secs. P3 13 secs. 21
secs. ...
9
Experimental Designs

• 2. Between Subjects
• Each participant provides results for only one
  condition

Color B/W P1 12 secs.
P2 17 secs. P7 19 secs. P5
15 secs. P3 13 secs. P8 21 secs. ...
10
Which to Use?

• What are the advantages and disadvantages of the
  two techniques?

11
Within Advantages

• Within subjects gives you more relative
  information - Each person is their own control
• Need bigger number of participants in between
  subjects to average it out better

12
Between Advantages

• Within subjects tests are much more liable to
  ordering effects
• Participant may learn from first condition
• Fatigue may make second performance worse
• Balance!
• Half go first in one condition, half go first in
  other

13
Experimental Results

• How does one know if an experiments results mean
  anything or confirm any beliefs?
• Example 20 people participated,
• 11 preferred interface 1,
• 9 preferred interface 2
• What do you conclude?

14
Statistics

• Central Limit Theorem
• The Sum of N random variables with the same
  distribution is Normally Distributed
• The Bell curve

15
Normal Distribution

• Occurs in many natural processes
• Grades
• IQ Scores
• Abilities of various kinds
• Typically observed in UI experiments

16
Normal Distribution

• Can use this knowledge to compute how probable
  some particular set of observations is
• Eg. If group A takes 10 seconds and group B takes
  15 on average
• Are A B different?

17
Hypothesis Testing

• In experiment, we set up a null hypothesis to
  check
• Basically, it says that what occurred was simply
  because of chance
• For example, any participant has an equal chance
  of preferring interface 1 over interface 2

18
Hypothesis Testing

• If probability result happened by chance is low,
  then your results are said to be significant
• Statistical measures of significance levels
• p lt 0.05 often used
• Less than 5 probability it occurred by chance

19
Example
Experiment 1 Group 1 Group 2 Mean 7
Mean 10 1,10,10 3,6,21
Experiment 2 Group 1 Group 2 Mean 7
Mean 10 6,7,8 8,11,11
20
Errors

• Errors do occur
• Types
• Type I/False positive - You conclude there is a
  difference, when in fact there isnt
• Type II/False negative - You conclude there is no
  different when there is

21
Presentation Techniques
Middle 50
Time In Secs
low
high
Mean
0
20
Age