Questionnaires

1
Questionnaires Interviewing
2
Transformations of Data
RAW/SENSE DATA
DATA 1
DATA 2
DATA 3published data and documents
memory
Actions/own experience
Temporary notes
Indexes and summaries
Statements and utterances
Diaries
notebooks
Preliminary analysis Coding, classifying,
simplifying, elaborating, interpreting, analysing
Index of indexes
Register of events
Maps, charts, tables, equations, etc. published
Witnessed events, in interviews or by observation
General documents
Survey returns
Survey schedules
Particular documents
Transcribed notes and memories
Tape recordings
photographs
Data dictionary and glossary
Special registers genealogies, tapes, photos, etc
videos
Other sources maps, official documents, research
reports
Other peoples documents and testimony
3
Types of Observational Data Acquisition

• Casual observation
• No/minimal deliberate preparation naturalistic
  research
• Systematic observation
• Time allocation traffic surveys scientific
  measurements
• Unstructured interviews
• Flexible content and flow
• Follow the flow interaction (conversation) and
  probing (spontaneous?)
• Driven by research questions and check lists
• Not restricted to check lists and check lists
  not always completed
• Life histories (time lines lists of significant
  events)
• Semi-structured interviews
• Flexible flow based on checklist of
  questions/topics
• Extensive probing for clarification and
  elaboration
• Resisting diversion from research questions
• Check lists answered in full
• Structured interviews
• Rigid content and flow
• Pre-worded questions administered in
  pre-determined flow
• Complete answering in all cases

4
Surveys
Interview
Self-completion questionnaire
Telephone
Face-to-face
Supervised
Postal
Internet
Paper pencil
Paper pencil
Direct data capture
E-mail
Web
Face-to-face
Bar code reader
Direct computer entry
Electronic recording (tape, video)
Direct data capture
Paper pencil
Embedded
Attached
Direct computer entry
Paper pencil
Electronic recording (hard disk, tape, video)
5
Use of Official/existing/others data

• Access and use (restrictions)
• Anonymisation
• Documentation of methods used
• objectively reflect reality?
• reflect organisational behaviour or actions of
  individuals in institutions?
• facilitate the /control regulation of the
  population?
• Censuses exacerbate race ..

6
Problems with Surveys

• Poor sampling (no/unsatisfactory frame, poor
  execution)
• Too long
• Too many questions
• Too diverse topics
• Poorly worded questions
• Language/jargon
• Leading
• Poorly laid out questionnaires
• Poorly trained/poor quality interviewers
• Poor management and supervision
• Personnel management
• Archiving
• Poor cooperation/rapport
• Poorly informed/poor choice of respondents
• Problems with data processing

7
Non-Sampling Errors - 1

• Errors in conception, logic, execution,
  reporting, data capture and manipulation,
  arithmetic, statistics, .
• Bias (systematic error)
• Sampling biases and errors
• Target and sampled populations
• Access and cooperation -
• Non-sampling biases
• Incompatible meanings (are they really your
  father/brother?)
• Pilot studies
• Hawthorne Effects
• Was the treatment actually applied?
• Pseudo randomisation (every random sample is not
  necessarily random)
• Simulation from ideal populations -
  computational statistics

8
Research and Data Process
topic
theories
concepts
Operationalisation
variables
Indicator of variable
Measurement theory
Raw data (2)
population
(Sample)
Survey/census responses
Coding, editing, inspection, and cleaning
sampling
Final data
analysis
communication
criticism
9
Non-sampling errors - 2

• Non-response and attrition
• Repeat (interpenetrating) samples (Central and
  State)
• partial panels (50 old 50 new households)
• Wording
• Strongly affects responses (see also leading
  questions)
• Cross-domain differences (space, social group,
  time)
• Use of standardised instrument has different
  biases in different contexts (consumption)
• Changing responses (timing of opinion polls,
  within and between year variations, time to
  evaluations)

10
Response errors

• Response errors
• Telescoping and Forgetting reference periods
• Too much and too little reported
• Self-edited and self-deceiving responses
• Acquiescence bias (agreeing with the
  interviewer/correct resonses, normal rather
  than particular)
• What people say (in particular circumstances)
• What people think
• What people do
• Position bias
• Age/number lumping selection from
  (ordered/numbered) list
• Triangulation
• Multiple sources (census, survey, and qualitative
  inquiry)
• Contested policy arenas

11
registered and non-registered information
reliance on memory -
12
Getting Process Right

• Relevance Accuracy
• Pre-tests notice the plural
• Rapport and cooperation
• Appropriate introduction and briefing
• Choice of respondent, anonymity, confidentiality
  
• Privacy
• Briefing
• Conduct
• Quick and efficient
• Taking refreshments
• Interruptions and returns
• Exit
• Affects response of others in cluster
• Affects cooperation with return checks

13
Improving participation(notes to be added)
Survey design
Respondent
Compliance Courtesy bias Helping/ altruism
change
Reciprocation Consistence Social
validation Authority Scarcity Liking
Method Respondent Environment interviewe
r
Contact Selection Length topic
Age Gender Income Health Education experience
Anger Happiness Sadness
Intrinsic factors Extrinsic features
Space/time
Appearance/age/sex Experience Expectations Mood
14
Choosing the right respondent

• Has the knowledge
• And is able to recall it
• Make the appropriate calculations
• Is willing to respond
• In principle
• Is able to respond (time)
• Has the authority
• In the appropriate way at the appropriate time

15
Better Questionnairs

• Length
• Complexity and diversity
• Flow
• Layout
• Size
• Space (for elaborate answers)
• Skips
• Computer aided questionnaires
• Speed of analysis
• Intrusion/suspicion
• Reliability and back-ups

16
Open, closed, leading, and other questions

• Leading , etc.
• Loaded or biased
• are women paid less than men for the same work?
• Loaded context
• privacy
• interviewer bias
• Double-(or more) barrelled
• Two or more questions
• Long
• General
• Technical
• (Double-)negatives

• Closed
• do you grow the following crops? list and
  check
• Specific predetermined answers
• Exclusive and exhaustive resonses
• Limits responses
• allow other categories
• describe/explain/elaborate
• provide opt-out (dont know/none..
• Standardises
• restricts diversity and exceptions
• Easier to analyse
• lots of data processing and statistical problems

• Open
• no pre-determined set of answers
• Allows a range of individual responses
• Liable to interviewer bias
• Harder to analyse
• non-standard responses
• data mining
• choosing cases without regard to broader set of
  responses