Adaptive interfaces for collecting survey data from users

1
Adaptive interfaces for collecting survey data
from users

• Michael Schober, PI
• New School University
• Frederick Conrad, Co-PI
• Bureau of Labor Statistics
• NSF/ITR Award IIS0081550

2
Project Objectives

• Determine how to design computer systems for
  collecting highest quality data from users
• class of applications on-line surveys, e-voting,
  on-line job applications, e-commerce purchases
• as opposed to systems that allow users to
  retrieve data from data bases
• What are the best ways to clarify words and
  phrases used in data collection interfaces?
• Using techniques from experimental psychology,
  determine what technology is worth pursuing

3
Understanding survey questions

• Many government surveys measure concepts with
  special meanings
• How many people live in this house?
• A person is considered to be living in a housing
  unit even if the person is not present at the
  time of the survey. Live-in servants or other
  employees, lodgers, and members of the household
  temporarily away from the unit on business or
  vacation are included in the count.
• Do not count any people who would normally
  consider this their (legal) address but who are
  living away on business, in the armed forces, or
  attending school (such as boarding school or
  college).
• Do not count overnight lodgers, guests and
  visitors. Do not count day employees who live
  elsewhere.

4
Understanding survey questions

• Without definition, hard to know, for example,
  whether to count child away at college
• complicated mapping
• versus straightforward mapping
• Interviewers generally cannot explain meaning
  because not standardized practice
• different respondents would answer on basis
  different question wording
• some with clarification and some without
• but when they do explain meaning, respondents
  comprehension and accuracy improve
• we know correct answer because responses based on
  fictional scenarios

5
Clarification in computerized questionnaires

• User-initiated clarification
• user requests definitions for words in the
  question
• desktop interface user clicks phrase for
  definition
• (simulated) speech interface (using Wizard of
  Oz technique) user asks for clarification
• System-initiated clarification
• system offers clarification when user seems
  uncertain
• desktop interface inactivity signals uncertainty
• (simulated) speech interface disfluencies, false
  starts, descriptions rather than answers also
  signal uncertainty

6
Clarification in computerized questionnaires
(contd)

• Desktop interface (Conrad Schober, 1998, 1999
  Schober, Conrad and Bloom, 1999, 2000)
• Varied users awareness of possible mismatch
  between their interpretation and systems
• told clarification either essential or available
• no clarification for another group
• Clarification either user-initiated or mixed
  initiative (both user- and system-initiated)

7
Definition for live in this house can be
obtained by clicking mouse on blue text
8
Computer offers to provide definition for live
in this house because user did not respond
before inactivity threshold
9
Definition presented for live in this house
10
(Schober, Conrad Fricker 1999)
11
Overall Response Accuracy
Type of clarification
12
Clarification in computerized questionnaires
(contd)

• Speech interface (Bloom Schober, in
  preparation)
• results comparable to desktop interface except
• users almost never requested clarification, and
  so
• mixed-initiative clarification increased accuracy
• users spoke less fluently than when system
  insensitive to their uncertainty (no
  clarification and clarification always)
• users most accurate when clarification always
  given but satisfaction ratings lowest
• users could not turn off long spoken definitions
• could not reject offer of clarification as in
  desktop study

13
Improving user-initiated clarification

• Can users be sensitized to potential differences
  in interpretation?
• One approach is simply to expose to definitions
  -- even if not directly relevant -- as part of
  question
• idea is to indicate that terms may not be used by
  survey sponsors in users everyday sense
• Users of desktop interface presented
• (1) original question wording
• (2) question parts of definitions that match
  (clarify) the confusion
• (3) question parts of definitions that do not
  match
• Hahn Lind, Schober Conrad, 2001 (supported by
  NSF grant IIS-00-81550 and SBR-97-30140)

14
Complicated scenario example

• The Gutierrez family owns the 3-bedroom house at
  4694 Marwood Drive. The family has four members
  Maria and Pablo Gutierrez, and their two children
  Linda and Marta. There is one bedroom for Maria
  and Pablo, one for Marta, and one for Linda.
  Linda is a college student. Although her legal
  address is still 4694 Marwood Drive, she stays at
  the college dorms all year, except for holidays
  and vacations.

15
(No Transcript)
16
(No Transcript)
17
(No Transcript)
18
Requests (clicks) for full definition
19
Response accuracy
20
Response time per question
21
Improving system-initiated clarification

• Lack of advantage for mixed initiative
  clarification in desktop interface due to
• clarification available users respond without
  clarification -- confidently, quickly and
  inaccurately
• clarification essential users request
  clarification before system can offer it
• Could help to system interprets different users
  signals of uncertainty differently

22
Improving system-initiated clarification

• Stereotypic user models
• desktop interface (work in progress, with Tania
  Coiner)
• different inactivity thresholds for different
  user groups
• longer threshold for older (gt 65) than younger (lt
  40) users
• age generally related speed of performing
  cognitive tasks
• 6 interfaces vary conditions for providing
  clarification
• (1) none -- used to set inactivity threshold
• (2) user-initiated
• (3) mixed initiative, generic user model
• (4) mixed-initiative, stereotypic user model
• (5) random
• (6) always

23
Improving system-initiated clarification (contd)

• Stereotypic user models (contd)
• speech interface (work in progress with Patrik
  Ehlen)
• in addition to age group inactivity thresholds,
  spoken indications of uncertainty
• disfluency, false starts, descriptions of
  circumstances
• 6 interfaces vary conditions for providing
  clarification
• (1) none -- used to set thresholds for various
  uncertainty cues
• (2) user-initiated
• (3) mixed initiative, generic user model
• (4) mixed-initiated, stereotypic user model
• (5) random
• (6) always
• older speakers shown to be less fluent than
  younger speakers

24
Improving system-initiated clarification (contd)

• Individual user models
• desktop interface inactivity threshold set for
  individual users
• based on
• each users performance on 2 or 3 practice items
  and
• statistical relationship between response latency
  for practice and survey items in first (desktop)
  experiment
• half users older and half younger to increase
  chances of differences in processing speed

25
Improving system-initiated clarification (contd)

• Individual user models
• speech interface in addition to individual
  inactivity thresholds, spoken indications of
  uncertainty also set for each user
• based on
• each users performance on 2 or 3 practice items
• statistical relationship between response latency
  for practice and survey items in previous
  experiment
• alternatives are more intrusive and increase user
  burden
• half users older and half younger to increase
  chances of differences in processing speed

26
Tailoring clarification

• Should survey systems determine what user needs
  to know or is it okay to present more than
  necessary?
• 4 mixed initiative, desktop interfaces
• no clarification
• full definition
• user formulated queries (Wizard of Oz)
• open ended clarification dialogue
• system-constrained queries (Wizard of Oz)
• clarification dialogue structured by menu choices
• Will measure accuracy and user satisfaction

27
Adopting the users perspective

• Thus far, users required to learn what survey
  designers mean
• What if system infers what users mean?
• 3 simulated mixed-initiative speech interfaces
• system gives definition user determines answer
• system asks user about each component of
  definition, determines answer and informs user
• same as second but system does not inform user of
  answer

28
Users may learn meaning across questionnaire

• Many surveys ask about a concept gt once
• Is it sensible to clarify concept repeatedly?
• 5 desktop interfaces
• No clarification
• Mixed initiative regardless of earlier
  presentation
• Definition once, user initiated
• Definition once, mixed initiative
• Constant reminder
• If users can maintain system perspective should
  prefer interface 3 to 4 and 5

29
Next steps

• Extend to other data collection systems
• e.g. e-voting has corresponding set of problems
  about clarifying issues and candidates
• Can system inform voters without biasing them?
• Begin developing and implementing techniques that
  look promising

30
(No Transcript)
31
Schober, Conrad Fricker, 1999
32
Obtaining clarification

• Any clarification improves response accuracy
• but mixed initiative (respondent- and
  interviewer-initiated) clarification helps more
  than only respondent-initiated
• makes less difference whether clarification is
  paraphrased or verbatim definition
• Respondents request clarification more when
  needed (complicated mappings) but not all the
  time that it is needed

33
Recognizing the need for clarification

• Respondents often dont realize they have
  interpreted words differently than intended

26
Only cigarettes you
finished
39
All puffs, whether or
Cigarettes you
not inhaled
finished or partly
52
Only puffs inhaled
smoked
Even one puff
61
smoke
cigarette
22

• despite multiple interpretations of ordinary
  words no one requested clarification when
  available

Suissbrick, Schober and Conrad (2000)
34
Recognizing the need for clarification (contd)

• Interviewers may not notice that respondents
  interpretation different than intended
• No clarification offered despite respondents
  multiple interpretations of tobacco terms
• Suessbrick, Schober Conrad, 2000
• Or may preempt evidence of conceptual differences
  
• Interviewers clarified wording before respondent
  spoke on 96 of clarification-giving occasions
• Conrad and Schober, 2000

35
(No Transcript)