Observational techniques and empirical evaluation

1
Observational techniques and empirical evaluation

• Involving users in evaluation

2
Agenda

• Questions?
• Project Part 3 schedule
• Observational evaluation techniques
• General methods
• Think aloud
• Overview of empirical evaluation
• Designing empirical studies
• Predictors
• Hypotheses
• Pitfalls
• Physiological response

3
Tentative Part 3 schedule

• 7/7 Show me your evaluation plans
• 7/9 Submit IRB protocols
• 7/12 or 7/14 in-class evaluations with HCI
  experts
• 7/19 Project Part 3 writeups due
• 7/19, 7/21, 7/23? Project presentations

4
Im sorry

• There probably will not be time for project part
  3 revisions
• Of course, getting feedback before the deadline
  is a wise idea
• But I will be out of town and mostly unavailable
  from July 13th- July 18th
• But.

5
Remember Ed?

• Ed gets lonely at his office hours
• Dont forget, he grades your papers too
• Ed knows a lot of stuff.

6
Recap

• What other evaluation techniques have we
  considered?

7
Observational techniques

• Watching users as they perform
• Summative or formative
• depends on the purpose of exercise
• Qualitative or quantitative
• depends on recording and analysis

8
Recording

• Paper and pencil
• Audio/video
• Computer logging
• User journals

9
Paper and pencil

• Inexpensive
• Flexible
• Limited in speed
• Requires attention
• Can be improved with predefined coding schemes

10
High fidelity, richness of dataOverview

• Empirical evaluation
• Primarily quantitative (statistical)
• Lab-based
• Scientifically rigorous

11
Computer logging

• Varies in difficulty
• Often cheap and unobtrusive
• Limited in scope

12
User journals

• Interpreted records both good and bad
• Amount of data depends on user
• Best for infrequent tasks

13
Support for recording

• Effective coding schemes
• Automated tools

14
Post-task walkthroughs

• Discussion with subject after observation
• Added richness and interpretations
• Warning post hoc interpretation

15
Specific techniques

• Think aloud
• encourage verbalization during interaction
• Cooperative evaluation
• designer/user discussion

16
Objective

• Formative
• need operational prototype
• Qualitative
• process or how-to information
• compare users (verbalized) mental model to
  designers intended model

17
Example Think aloud

• Use www.cnn.com to find out who is playing in
  tomorrows Euro Cup soccer match

18
Example What was learned

• What was the users model?
• What was the designers intended model?
• Any discrepancies?

19
Benefits of Think Aloud

• Process is less constrained gets at users
  perspective
• User is encouraged to criticize the system
• Evaluator can clarify at problem spots

20
Cooperative evaluation

• Similar to Think Aloud, but evaluator acts as
  collaborator, not just as a subject
• Different kind of feedback encouraged by designer

21
Empirical Methods

• Controlled laboratory experiment to support a
  claim or hypothesis

22
Why use empirical eval?

• According to the logical positivist point of
  view, the experiment is the best way to conduct
  research (Whitley, 1996).
• Why?
• Rigorous
• Causal relationships
• If x then y
• AND
• If NOT x then NOT y

23
The Case for Experimentation Benefits
Limitations

• Consider an example
• Designing a website for older adults

24
Benefits

• High degree of control
• Precise measurement (related to control)
• Allows for causal judgments

High Perceptibility (WhizBang.com)
Low perceptibility (BoringStuff.com)
25
Limitations

• Artificiality
• Environment
• Methods
• Ethics

Just be comfortable and respond normally!
26
Designing an empirical study

• Question
• Experimental factors

27
Dont Forget Your Question

• What is your question?
• Review the literature
• Has you question already been answered?
• Is there a debate about your question?
• How have previous studies been designed to answer
  this question?

28
Experimental Factors

• Define the variables
• Hypotheses
• Method
• Participants

29
Define the Variables

• Independent variables Manipulated, Cause
• Dependent variables Measured, Effect
• The dependent variable(s) needs to be affected by
  the independent variable(s).
• Control group

30
Hypotheses

• Hypothesis A variation in the independent
  variable will cause a difference in the dependent
  variable.
• Null hypothesis There is no difference in the
  dependent variable by the independent variable.

31
Example
2 (age) x 4 (site design)
DV Time to complete the task
32
Experimental Method

• Between-groups
• Within-groups
• Mixed

33
Between-groups

• Each participant assigned to a different
  condition
• No learning effects
• -- Disadv. Need more participants
• -- Disadv Individual differences could bias the
  results

34
Within-groups

• Participant tested on all conditions
• Adv. Fewer participants
• -- Disadv. Potential learning effects

35
Mixed

• One variable between-groups and one variable
  within-groups

36
Participants

• Characteristics
• Number
• Consent
• Anonymity
• Risks
• Compensation
• Free will

37
Independent vs. Confound Variables

• Independent variables
• Confound variables
• Participant variables
• Procedural variables

38
Tips

• Experimental control, not as easy as it may seem
• Plan ahead
• Create a detailed protocol be organized
• Pilot studies

39
Evaluation through monitoring physiological
responses

• Eye tracking
• Heart activity blood pressure, volume, pulse
• Galvanic skin response
• Electrical activity in muscles
• Electrical activity in brain

40
Eye tracking
41
Physiological responses

• Can get at tacit responses
• Can somewhat measure interest, involvement,
  emotional response, stress

42
Upcoming

• Diary and pager studies
• Project part 3 sample prototypes