Prof Marcus Ormerod

1
Prof Marcus Ormerod

• Research Survey Methods

2
When to use a questionnaire

• The main research technique for data gathering in
  your study
• As a small part of some other research technique
  (triangulation, springboard)
• Usually cross-sectional rather than longitudinal

3
Ways to collect questionnaire data

• Face-to-face interviews
• Self-administered questionnaires
• By postage
• Dropped off and collected later
• Electronically
• Telephone interviews

4
Advantages of face-to-face interviews

• They can be used with groups of people who
  otherwise would not complete the questionnaire
• If a respondent does not understand a question
  the interviewer can clarify
• You can use techniques that cannot be used by
  post such as props, cue cards
• Can allow more time than a person would give for
  a postal questionnaire
• You give questions one at a time so they do not
  see what is coming up later
• You know who answers the questions

5
Disadvantages of face-to-face interviews

• Can be intrusive and interviewer can bias the
  answers
• Costly to undertake interviews in both time and
  money
• Interviewers can skew results by grabbing
  available respondents rather than the sample
  identified
• The amount of interviews that can be done is less
  than a postal questionnaire
• The time to conduct all the interviews can mean
  that events have changed dramatically

6
Advantages of self-administered questionnaires

• Allows a single researcher to survey a large
  sample of respondents at a low cost
• All respondents get the same questions
• Long list questions/ matrices allow more
  complexity than in an interview situation
• Response effects are minimised on sensitive
  questions. Can remain anonymous
• Can be delivered in a variety of mediums to suit
  the respondents needs
• People are generally familiar with the concept of
  self-administered questionnaires

7
Disadvantages of self-administered questionnaires

• You have no control over how people interpret
  questions
• You have to assume the person you targeted really
  did complete the answers
• Response rates can be very low (20-30) which
  means you need a larger sample to compensate
• Sampling strategies can be seriously flawed and
  will inevitably be flawed by a low return rate
• You cannot hide questions from a respondent until
  they have completed previous questions
• Certain groups of people are more likely to
  return it

8
Advantages of telephone interviews

• They are more impersonal than face-to-face
  interviews
• It is reasonably cheap to conduct, but can be
  time consuming
• You can talk to people who would otherwise be
  inaccessible through location

9
Disadvantages of telephone interviews

• Random sampling can be difficult
• Time tends to be short before someone hangs up
  about 20 minutes
• If using paid interviewers they can produce
  garbage data to meet quotas

10
Question wording

• Be unambiguous
• Use vocabulary that your respondents understand,
  but do not be condescending
• Respondents must know enough to respond to the
  question asked
• Make sure there is a clear purpose for every
  question and that the respondent understands the
  significance

11
Question wording

• Be careful with contingencies and filter
  questions
• Use clear scales that reflect the detail required
  and the medium used (telephone interviews usually
  3 point scale, self-administered questionnaires 5
  or 7 point scale

12
Open-ended versus closed questions

• Open-ended questions allow the respondent free
  choice in their answer, but this can then leads
  to more difficulty in categorising and analysing
  answers
• Closed (forced-choice) questions provide a series
  of possible answers for the respondent to choose
  from, although it is usual to include an other
  choice to allow the respondent to include their
  own if different from the selection provided.
  Easier to precode answers

13

• Behaviour questions establish what people do
• Belief questions ascertain what the respondent
  thinks is true
• Attitude questions try to establish what they
  think is desirable
• Attribute questions collect information about the
  respondents characteristics
• (Vaus 1996, pp81-83)

14
Response rates
Number returned
Response rate
X 100
N in sample (ineligible unreachable)
15
Dillmans method (Bernard 2000, Vaus 1996)

• Mailed questionnaires must look professional,
  research suggests light green paper gets the best
  response rate
• Dont put any questions on the front or back
  covers just title, graphic designs (not photos)
  and thanks on the back with invitation for
  further comments
• Question order is important. Start with a
  directly relevant question that is interesting,
  easy to answer and non-threatening.
  Socio-economic and demographic questions at the
  end

16

• Never allow a question to break at the end of a
  page. Use plenty of paper. Standard convention is
  all capitals for instructions and mixed for
  questions
• Package questions into logical sections in
  self-administered questionnaires and through
  lists, tables and matrices
• If you need respondents to check just one box
  make sure the possible responses are exhaustive
  and mutually exclusive

17

• Keep unthreatening questions short, but include
  preambles to lessen intimidation of threatening
  questions
• Avoid loaded questions dont you agree that..
• Do not use double barrelled questions
• Do not assume always test the question to check
  that you have covered all the possibilities
• Do not take emotional stands on questions
• If conducting it in another language than your
  own use back translation to ensure concepts are
  kept

18

• Keep mailed questionnaires down to 10 pages in
  length with no more than 125 questions
• The cover letter should explain in the briefest
  terms the nature of the study, how the respondent
  was selected, who should complete it, guarantee
  of confidentiality and possibly anonymity, who is
  funding the survey and why it is important for
  the respondent to return the questionnaire
• Personally sign the letters in blue ballpoint
• Enclose prepaid return address envelopes

19

• Consider how you will know if someone has
  responded if anonymous replies use a separate
  card with a number, or number the return
  envelopes
• Package the questionnaire, cover letter, reply
  envelope and postcard in another envelope with a
  direct typed address rather than mailing label.
  Use first class stamps especially commemorative
• Inducements can be taken in various ways and may
  dissuade respondents, if anything include a pen
  to complete the questionnaire

20

• Contact people before sending out the
  questionnaire and have a follow-up procedure to
  chase non-responses. Each follow-up wave nets
  approximately 50 of the previous waves amount
  again
• Establish that a questionnaire is the most
  appropriate method of data collection for what
  you are trying to study
• Does it need to be representative? And if so can
  you identify the population?

21
Some points from experience

• How will you analyse the data when you get it in?
• How much time and money do you have?
• Think of a time period to develop the
  questionnaire then double it
• Timing and respondent interest
• Pilot questions as much as possible
• Range of answers rather than all at one end of
  the response range

22
Piloting questionnaires

• Variation does the question produce spread
  Meaning do they understand the intended meaning
  of the question and you the answer?
• Redundancy are you asking the same item in
  different questions? (But can be useful for
  checking consistency of answers)
• Scalability if a set of questions forms a scale
  do they all fit?
• Flow do the questions fit together?
• Question skips and filters do they work?
• (Vaus 1996, pp100-103)

23
Probability v non-probability

• Individual-attribute data gathers data about the
  individuals within a population and requires
  probability sampling to be able to generalise
  back to the entire population
• Cultural data requires experts who can offer
  explanations on how the group does things, or
  experiences of the group, and can use
  non-probability sampling as it is not generalised
  back to the whole population

24
Scales

• Single indicator scales assign units of analysis
  to categories of variables
• Composite measures or complex scales/indexes are
  made up of several items all of which count the
  same thing, such as IQ tests, Dow Jones index,
  life assurance tables (Bernard 2000, pp 287- 289)
• Guttman scales (scalogram analysis)measure items
  which have a pattern which indicates that they
  are a unidimensional variable (Bernard 2000,
  pp289-293, Oppenheim 1992, pp201-205)

25

• Likert scales are 5 point or 7 point scales based
  on polar opposites such as agree-disagree,
  excellent-bad (Bernard 2000, pp294-296 Oppenheim
  1992, pp 195-200 Vaus 1996, p88, p252-257)
• Semantic differential scales differ from a true
  Likert scale in that they name the item, rather
  than ask a question about it, and ask the
  respondents to rate their feelings towards a set
  of variables, usually on a 7 point scale (Bernard
  2000, pp 303-304 Vaus 1996, p88)

26
Sampling

• Probability sampling
• Simple random
• Systematic
• Stratified
• Multistage cluster
• Non-probability sampling
• Purposive
• Quota
• Convenience or Availability
• Snowball
• (Bernard 2000, pp144-180 Vaus 1996 pp 60-79)

27
References/Bibliography

• Bernard, H.R. (2000). Social Research Methods.
  Qualitative and Quantitative Approaches. London
  Sage.
• Lincoln, Y.S. Guba, E.G. (1985). Naturalistic
  Inquiry. London Sage.
• MacRae, S. (1994). Describing Interpreting
  Data. Leicester BPS Books
• MacRae, S. (1994). Drawings Inferences from
  Statistical Data. Leicester BPS Books
• MacRae, S. (1994). Models Methods for the
  Behavioural Sciences. Leicester BPS Books
• Newton, R.R. Rudestam, K.R. (1999). Your
  Statistical Consultant. London Sage.
• Oppenheim, A.N. (1992). Questionnaire Design,
  Interviewing and Attitude Measurement. London
  Cassell.
• Vaus, D.A. (1996). Surveys in Social Research.
  London UCL Press