Needs Analysis: Design through Discovery

1
Needs AnalysisDesign through Discovery

• Philip Kortum, Ph.D.
• Robert Bushey, Ph.D.
• SBC Laboratories
• Human Factors

2
Todays Agenda

• Who we are (5 min)
• Basic needs analysis techniques (60 min)
• Virtual Lab tour (10 Min)
• Break (10 min)
• Needs analysis in high-data environments (70 min)
• Conclusions (10 min)

3
Who is SBC?

• Short Answer
• Were the phone company
• Long Answer
• 2nd largest telecommunications providers the US
• Wireless
• Long Distance
• Data
• Residential/Business phone service
• Over 60 million access lines
• 41 Billion in Revenue ( 27 on the Fortune 500)

4
Who is SBC Laboratories?
SBC Laboratories is the Applied Research and
Development arm of SBC. About 200
Advanced-degreed Scientists and Engineers work at
SBCs laboratories in Austin and the Bay area in
California.
Broadband Architecture, Infrastructure, Services
Intelligent Networks
Information Technology
Internet
Wireless Systems
5
SBC Laboratories Human Factors Engineering
6
Where todays talk fits in this class
From Mayhew, 1999
7
What is Needs Analysis?
Needs analysis is the method of uncovering user
requirements through direct or indirect
interaction with that user
8
Why do we perform needs analysis?

• We are not the user
• Users may not know what they really need
• User is unable to articulate the need
• True needs are often masked
• Good match between needs and implementation leads
  to superior efficiency and usability

9
How is Needs Analysis different than Task
Analysis?

• Needs Analysis trying to uncover the underlying
  motivations
• Task Analysis trying to uncover the procedural
  steps

10
How Is Needs Analysis Different Than Task
Analysis?

• Task Analysis
• Setting up a VCR
• Programming a VCR
• Needs Analysis
• The ways a person uses a VCR and the reasons
  behind that use
• What they record
• What they do with what they record
• Time-shifting
• Remote control use use

11
How Is Needs Analysis Different Than Marketing?

• They both deal with the user
• They both try to determine what the user needs
• Some of the techniques are similar
• Some of the information collected is the same

12
How Is Needs Analysis Different Than Marketing?

• Marketing
• Demographics
• Where people live, shop
• How much they make
• Physical attributes
• Purchasing behavior patterns
• What people will buy
• Price points

• Needs Analysis
• User behavior patterns
• What people do
• What users know
• Users environment
• Users mental models

13
Needs vs. Marketing ExampleA device that records
off of the television

• Marketing
• How much money do the users have to spend?
• What other similar devices do they have?
• What kinds of technology do they own?
• How much do they use these technologies

• Needs Analysis
• Why would they want to use an image recorder?
• How do they use current devices?
• When do they use such devices?
• What other technology interactions are there?

14
A Simple (if unfair) Characterization
Marketing deals with wants
Needs deals with needs
15
A word about Semantics.

• Many techniques are similar and are called
  different things by those who use them (even
  within the usability community)
• Good practitioners almost always use a mix of
  techniques, and often call these mixtures by a
  new name.

16
Needs Analysis
Can be very focused and limited in scope
Doesnt have to big sky
17
Basic Types of Needs Analysis

• Questionnaires
• Interviews
• Observational Research
• Hybrid methodologies

18
Questionnaires

• A set of questions for obtaining statistically
  useful or personal information from users
  
  Webster

19
Questionnaire Advantages

• Easy to administer
• Inexpensive
• Can be administered by untrained staff
• Can be administered remotely
• Can have a relatively large number of questions
  and data samples
• Can more easily target highly specific users
• Can branch to reliably capture pertinent
  information
• Data can be simple to code

20
Questionnaire Disadvantages

• Difficult to construct a good test instrument
• Must know a priori exactly what you want to ask
• No follow-up questions
• Question intent may be open to interpretation
• Can result in unknown bad results
• Open ended responses highly variable
• Open ended questions are difficult to code
• Validation/verification difficult if study is not
  closely controlled

21
Questionnaire Disadvantages Fixes

• Difficult to construct a good test instrument
• Follow scientifically designed questionnaire
  guides (Kirakowsi, Gillham, USARI)
• Test for validity, reliability, repeatability
• Must know a priori exactly what you want to ask
• Conducted pilot tests to try out the questions
• Conduct interviews to help build a reasonable
  first pass
• Use domain experts to help capture detail
  questions

22
Questionnaire Disadvantages Fixes

• Question intent may be open to interpretation
• Conduct pilot tests to check for weaknesses
• Use experts and novices to examine interpretation
  differences
• Use unambiguous, quantifiable, anchored language

23
Ambiguous Question Example
How bad was your last car accident?
Really bad bad not really bad
not bad at all
2 respondents, who both dented the front bumper
slightly, no injury
Respondent 1 Really Bad
Respondent 2 Not bad at all
Im only 16, its my first accident and it was
my dads new sports car, and I wasnt on the
insurance for the car. Im going to be grounded
for life!
Well, compared to that 23 car roll-over
collision I had with that semi last month, this
was nothing. Not to mention that I got the
collision waiver on the rental!
Class How could we fix this?
24
Questionnaire Disadvantages Fixes

• Open ended responses highly variable
• Provide hints, examples to guide the responses
• Use multi-part questions that are more precise
• Open ended questions are difficult to code
• Use more choices, or more questions, to remove
  need for open ended questions
• Focus on variables of interest and use other to
  capture low probability events that are not
  pertinent
• Devise keyword codebooks to help code open ended
  data

25
Questionnaire Disadvantages Fixes

• Validation/verification difficult if study is not
  closely controlled
• Key code questionnaires to match user and
  instrument
• Lock-out multiple take attempts
• Ask qualifying questions as part of a pretest, or
  branch

26
Interviews

• A meeting at which information
• is obtained from a person
• 
  Webster

27
Interviews

• Generally classified into two types
• Structured
• Unstructured

28
Interview Advantages

• Small numbers of interviews can be performed
  quickly and inexpensively (although not always)
• Can be conducted remotely
• Can be conducted in groups or individually
• Can be conducted with little notice, if needed
• Can change the interview for different levels of
  users

29
Interview Advantages

• Open for opportunistic data discovery
• Get non-verbal clues to help guide the interview
• Good technique for fishing expeditions
• Useful for gathering preliminary information to
  guide later needs analysis

30
Interview Disadvantages

• Hard to conduct a good interview
• Easy to lead the respondent, with both verbal and
  non-verbal cues
• Easy to unknowingly ask loaded questions
• How does it feel to be ugly?
• Harder still to conduct a good group interview
• herd mentality, strong leader
• Hard to stay up and focused for a large number
  of interviews

31
Interview Disadvantages

• Data can be difficult to code and quantify
• Can be large differences between interviewers
• Expensive and time consuming to conduct large
  numbers of interviews
• Usually a low upper limit to the number of
  questions

32
Interview Disadvantages

• Interview Example

33
Interview Disadvantages
What was said
What the data suggests
Question
Reps make 1 offer per call
Almost no reps make 1 offer, some do 0 some do 2
How many offers do reps make per call?
order takers are the least desirable
Order takers are good - they drive
accessibility and are among top performers
What kind of rep is the least desirable?
Taking lots of calls is what makes customers happy
No supporting data
What makes the customer happy?

• Interview data may not match reality

34
Interview Disadvantages Fixes

• Hard to conduct a good interviews
• Use scientifically developed methods (i.e.
  Weisss Learning from Strangers)
• Pre-develop a core set of questions
• Manage group dynamics
• Use hypothetical questions to elicit more
  detailed answers
• Hard to stay up and focused for a large number
  of interviews
• Limit the number of interviews
• Use tag teams, alternate sessions
• Use well trained people persons

35
Interview Disadvantages Fixes

• Data can be difficult to code and quantify
• use core set of questions
• use data code books to help quantify responses
• use affinity diagrams to aid in coding
• Can be large differences between interviewers
• use single interviewer
• use visible or hidden teams
• conduct pilot training to level the interviewers

36
Interview Disadvantages Fixes

• Expensive and time consuming to conduct large
  numbers of interviews
• Usually a low upper limit to the number of
  questions
• target exact population
• supplement with other techniques
• use remote techniques
• use core question set, based on pre-test
• Interview data may not match reality
• Always verify data

37
Observational Research

• An act of recognizing and
• noting a fact or occurrence
  Webster

38
Observational Research

• Observation doesnt have to occur in real time
• Observation doesnt have to be visually based
• audio recordings
• diaries
• telemetry (GPS, web logs, biometrics, equipment
  monitors) (e.g. car snooper)

39
Observational Advantages

• Good for understanding complex needs
• Results are not defined by the design of the
  method
• Capable of capturing unknown or undocumented
  behaviors
• Gather data on specific artifacts
• Can uncover inter-dependencies
• If documented appropriately, can review
  observations as many times as necessary
• Can parse expert/novice distinction

40
Observational Disadvantages

• Some phenomenon is too infrequent to catch
• Heisenberg Principle (i.e. driving tester)
• Can be more difficult to obtain a reviewable
  record
• Desired behavior may be interspersed with other
  unrelated activity
• Slow, expensive to collect data
• Can easily get off-target
• Data set can be overwhelming

41
Observational Disadvantages Fixes

• Some phenomena are too infrequent to catch
• use high sample rates to find low probability
  events
• Create low probability (disaster training)
• sample in areas known to have these events
• be lucky
• Heisenberg Principle (i.e. driving tester)
• observe for extended periods of time
• use remote measuring techniques
• use low/no interaction models
• clearly understand the cost-benefit of direct
  interaction
• work in the users environment

42
Observational Disadvantages Fixes

• Can be more difficult to obtain a reviewable
  record
• use alternate recording technologies
• work in teams
• Use advanced behavioral software (e.g. Noldus
  Observer)
• Desired behavior may be interspersed with other
  unrelated activity
• use specific artifact techniques
• use remote data collection techniques to time
  compress
• dont assume that unrelated activity is actually
  unrelated (e.g. using the restroom while waiting)

43
Observational Disadvantages Fixes

• Slow, expensive to collect data
• use remote or automated collection where possible
• understand required sample size
• carefully select person/situation/setting to be
  observed
• Can easily get off-target
• use pilot observations to help identify key
  behaviors
• be open to the fact that off-target behaviors may
  be pertinent

44
Observational Disadvantages Fixes

• Data set can be overwhelming
• use data reduction/consolidation techniques
• training samples
• affinity diagrams
• physical flow/sequence structures

45
Observational Research ExampleWatching Television
46
Observational Research ExampleWatching Television
Nathen, et al, 1985
47
Hybrid Methodologies

• Ethnography
• Contextual Inquiry
• Empathic Design
• Participatory learning
• actors
• rotational management assignments

48
What happens when needs isnt done?
49
What happens when needs isnt done?
50
Questions?