Usage Data Observations and User Attitudes

1
Usage DataObservations and User Attitudes

• HCI 30

2
Observations

• Data collection can be done in a number of ways
• Direct observation
• Take notes
• Video recording
• Audio recording
• Software logging

3
Observation tradeoffs

• Incomplete data record
• Taking notes by hand
• Also it is not available for review in a
  different light
• Too complete data record (awash in data)
• Video recording
• Audio recording
• Software logging

4
User attitude collection

• Questionnaire
• Interviews

5
Observation in general

• Not as easy as it seems
• You see what you want to see
• This is a human attention issue
• In a nutshell Human senses suck BUT our mental
  processes fill in the missing detail based on
  what we THINK is going on
• The missing part of the visual field

6
Direct Observation

• Cheap
• Provides only a hand written record
• Not available for review and re-evaluation
• What you are recording by hand is based on what
  you think is important at the time
• Can be obtrusive you gotta stand there which
  can have unintended effects on the tasks being
  observed Hawthorne Effect

7
Hand-taken notes

• ALWAYS! Once your observation is complete and you
  leave the site ALWAYS! Take 30 minutes to review
  your notes
• Do not wait until the next day do it within an
  hour of leaving the site
• To
• Clarify your handwriting
• Include additional comments that come to mind
• Make additional notations on drawings made

8
More on direct observation

• Kids and other strange beasts
• Madness usually ensues when there are two or more
  in a confined space (like school)
• Being singled out for a study can have a
  powerful negative effect can equate to
  punishment

9
Finally Direct Observation

• Can be extremely useful in developing an idea
  about the total picture of the work environment
• Checklists can be useful support but only if you
  know what you want to start with
• Using data logging tools like video and audio
  tape can help BUT its administration can (will)
  distract from the experience

10
Indirect observation

• Video capture tied to salient events like
  keystrokes
• Puts distance between you and your subjects
• Multiple cameras to capture multiple aspects of a
  task WHOA! Data swamp!

11
Video Administrative Issues

• Time for setup and breakdown
• How do you label the recordings?
• Who starts and stops the equipment?

12
Video Data Analysis

• Task-based
• How did the users attack the assigned task?
• Were there major difficulties encountered?
• Performance-based
• Frequency of correct task completion
• Frequency of errors (requires error
  classification)
• Task timing
• Timing between actions (perhaps a measure of
  cognitive load?)

13
Audio data

• Verbal protocol An audio record
• Can contain users observations
• Can contain observers commentary
• Can provide an insight into the way the user
  views the system
• What names they use for certain elements
• Reactions to errors
• User understandings of error messages

14
Think aloud protocol

• User says out loud what they are thinking
• Can be helpful in understanding how the user
  understands the task and the interface
• Can increase the strain on the user it becomes a
  divided attention task
• Punctuated by silences during times of confusion
  just when it gets interesting!
• May or may not be reliable

15
Think aloud protocol

• Overcoming the silence arrange for two users to
  work with and talk to each other about what is
  going on.
• Pairing an expert with a novice can help
• This dialog becomes the protocol of interest

16
Post event protocol

• After it is recorded, the user views the tape and
  provides commentary
• Can contain information that is recalled during
  the commentary that was not remembered or used in
  the actual event
• Hindsight can produce a rationalization that did
  not exist at the time of the experiment

17
Briefly Software logging

• Hooks put into the software at interesting points
• Can produce time-stamped salient events
• Interaction logging allows real time playback
• Can be combined with video and audio

18
User opinion collection

• Structured interviews
• Have a predetermined list of questions to be
  asked
• Provide stronger statistical support
• Not as flexible
• Flexible interviews
• Have set topics but not predetermined questions
• More useful in early stages of design

19
User opinion collection

• Semi structured interview
• Have a set of questions to draw on (in your head)
• Select from the set based on the direction
• Prompted interview
• please expand on x for me
• and you mean by x that
• tell me about your childhood ThomasDolby

20
Interview issues

• Have a rough plan
• Consider how to develop rapport with user
• The more structured the interview, the easier it
  is on the interviewer
• The less structure, the broader the scope of
  issues that can be addressed, but data analysis
  is more diffficult
• A checklist of important questions is helpful

21
Alternative interview techniques

• Twenty questions
• Asking questions that only have yes or no as
  their answer can be useful
• Interview that is prompted with artifacts
• Actual artifacts from the design
• Cards that represent important issues and are
  arranged by or selected by the user

22
Questionnaires

• Like a highly structured interview
• The questions should be well thought out and
  unambiguous
• Should be preceded by a pilot study
• Can reach a large audience
• Two styles
• Closed question (multiple choice)
• Open question (free response)

23
Questionnaires

• Simple (yes, no, dont know)
• Multi-point rating scale (useful to of no
  use)
• Likert scale (a variant of multi-point)
• Semantic differential
• easy-difficult, clear-confusing, fun-boring
  (bi-polar adjectives)
• extremely,quite,slightly,neutral,slightly,quite,ex
  tremely
• Ranked order of listed items

24
Questionnaires, Finally

• Pre and Post experience questionnaire
• Looks at how user attitudes change after the
  artifact is introduced

25
The End!