What Did You Learn Last Week

1
What Did You Learn Last Week?

• Describe two alternative ways of eliciting
  particiants' thought processes as they work.
• Discuss the tradeoffs among the following data
  collection methods notetaking, audiotape,
  videotape, diary collection, interaction logs
• Give an example of a research question one might
  pose within the scope of a usability test.
• Suppose you are running a usability test. What
  are the key materials you'll need to prepare?
• How many participants should you recruit for a
  usability test?

2
Lecture 11 Analysis of User Tests/Experiments(P
reece 14.3, 14.4)
Open Source Usability
3
Lecture Overview

• Part I User Testing Part II (Analysis)
• Part II User Testing Case Study
• Part III Discuss Final Presentations
• Part IV Discuss Usability Lab Orientation and
  Design Docs

4
Part I User Testing II (Analyzing the Results)
5
How to do User TestingAnalyzing the Results

• In the last lecture, we discussed
• Preparing for a usability study
• Running a usability study
• In this lecture, we assume that we have a
  recording of the session, and we focus on
  analyzing the recording

6
How to do User TestingAnalyzing the Results
(cont.)

• Best to analyze visual and verbal record as a
  team, so that episodes and interpretations can be
  freely discussed
• Compile a log of critical incidents with the
  following information
• Location on videotape
• Task participant was performing
• Stated intentions of participant
• Actions performed by participant
• Participants interpretation of results
• Analysis of cause of problem
• Recommended design solution

7
How to do User TestingAnalyzing the Results
(cont.)

• Analyze recording in two passes
• First pass Identify critical incidents
• Second pass Further analyze critical incidents
• Alternatively, you can identify critical
  incidents live, and analyze them post-hoc (this
  saves time, but requires an experienced observer)
• For reporting purposes, make a highlights tape
  or transcribe certain critical incidents
• Challenge Diagnosing cause of problems
• Often difficult to ascribe intent to
  participants, since they may not state it
  explicitly
• Need domain knowledge, good recording (transcript
  is not enough), and ability to put yourself in
  participants shoes
• Analyzing within a group may help diagnose
  problem and generate possible solutions

8
How to do User TestingAnalyzing the Results
(cont.)

• Severity ratings (Dumas and Redish, 1993)
• Severity 1 Prevent users from completing tasks
• Severity 2 Create significant delay and
  frustration
• Severity 3 Have a minor effect on usability (an
  annoyance)
• Improvement Not really a problem, but the task
  can be made even easier
• Scope ratings (Dumas Redish, 1993)
• Scope 1 Will affect almost all users
• Scope 2 Will affect many users
• Scope 3 Will affect few users

9
How to do User TestingAnalyzing the Results
(cont.)

• Other quantitative measures
• Number of participants who encounter a particular
  problem
• Number of participants who complete a task
  successfully
• Time to complete a task
• Number and type of errors per task
• Number of errors per unit of time
• Number of navigations to online help or manuals
• Usability engineering orientation
• Current level of performance
• Minimum acceptable level of performance
• Target level of performance

10
Part II Case Study(See pp. 8-14 of Douglas
Supplement)
11
Case StudyBackground

• Example drawn from the three year development
  cycle of the Cardio-Vascular Construction Kit
  (CVCK)
• Target users Introductory college biology
  students in collaborative biology laboratories
• Goal Teach students concepts of pressure and
  flow within context of cardio-vascular systems

12
Case StudyInterface Under Test
Ventri-cle
Valved pipe
Compass Tool
T-pipe
Straight pipe
Elbow
Muscle
Cardio-vascular system under construction
13
Case StudyEpisode Under Analysis

• Video clip illustrating a series of problems
• We'll fill in a content log as we watch
• Each time you see a critical incident, raise your
  hand
• Transcript illustrates similar problems
• See Douglas supplement

14
ExampleSummary of Analysis

• Three problems identified
• Failure to recognize rotate icon
• Failure to select component (elbow) before using
  the rotate icon
• Failure to position cursor inside small target
• Proposed solution
• New icon
• Trap that tells user that component must be
  selected before rotate can be applied
• Follow-up testing indicates that solution works!

15
Part III Discuss Final Group Project
Presentations
16
Final Group Presentations

• You will give an in-class presentation of your
  final group project prototype during the final
  week of class
• Presentation format (20 min. maximum)
• Summarize software genre, user population, and
  early data gathering results (3-4 min.)
• Live demo of your high fidelity prototype (walk
  through core task scenarios) (6-7 min.)
• Present usability study, key results (video clips
  highly required), and proposed design changes
  (6-7 min.)
• 3-4 minutes Open question and answer session

17
Final Group Presentations (cont.)

• Presentation requirements
• Powerpoint presentation slides
• Live demo of software
• "Highlights reel" of usability problems and
  successes
• See assessment form for specifics
• Above implies that you must use a laptop computer
  for the presentation
• You must have your Powerpoint presentation,
  software, and highlights reel preloaded and ready
  to present

18
Final Group Presentations (cont.)

• Presentation schedule (10 presentations total)
• Tuesday, April 22
• 845 Requirements Management (Devin, Benjamin)
• 910 Rowing Management (Anna, Alvaro)
• 935 Recipe Management (Archana, Kavya,
  Vandhana)
• 1000 SBL Online (Anu, Pawan, Kyle)
• 1025 WSU Campus Map (Geetika, Arun)
• Thursday, April 24
• 845 Poker (Brian, Marvin)
• 910 Rowing Management (Anna, Alvaro)
• 935 Board Game (Devin, Shannon)
• 1010 WOZ (Adam, Shaikot)
• 1025 SmartHome (Sean, Joshua)

19
Part IV Discuss Design Docs and Usability Lab
Orientation