Sampling and Questionnaires

1
Sampling and Questionnaires

• Bill Reimer
• reimer_at_alcor.concordia.ca
• http//reimer.concordia.ca/teaching

2
Outline

• Sampling
• Questionnaire construction

3
Sampling Objective

• Get a lot of information with little cost

Population
4
Sample sizes for a population of 10,000
5
3 Rules of Scientific Sampling

• each unit in the population has an equal chance
  of being chosen
• we must know the chance of each member being
  chosen
• each selection must be independent from the others

6
To what do you want to generalize?

• People
• Households
• Trees
• Leaves
• These are the Units of Analysis

7
To which units do you want to generalize?

• People in Canada?
• Women in Québec?
• Citizens of Montréal?
• Trees in boreal forests?
• This is the Population

8
The Population

• Is the collection of units to which you want to
  generalize
• It may be abstract
• It may be concrete
• But it must be clear

9
From what will you choose your sample?

• A list of units?
• A region or place?
• A drawer or computer file?
• A particular time?
• This is your sample frame

10
The Sample Frame may not match the Population
Population
Sample Frame
Units in the SF but not the P
Units in the P but not the SF
but try to get them close
11
Sampling Requirements

• Identify the population
• Is the sample frame reasonable?
• Does it meet the 3 conditions of scientific
  sampling?

We can seldom meet all 3 conditions!
12
Types of Samples

• Probability
• Generalizability
• Efficient
• Non-probability
• Exploratory
• Strategic

13
Simple Random Sampling

• Equal chance of being chosen
• We know the chance
• Each selection independent

From OLeary, Z (2004) The Essential Guide to
Doing Research, London Sage, Ch 8.
14
Systematic Sampling

• Determine sample rate (8/40)
• Random selection of 1st case
• Every nth case (n5)
• Watch for regularities in the sample frame

From OLeary, Z (2004) The Essential Guide to
Doing Research, London Sage, Ch 8.
15
Stratified Random Sampling

• Separate sample frame into strata
• Take random (or systematic) sample from each
  strata
• May be proportionate or disproportionate

From OLeary, Z (2004) The Essential Guide to
Doing Research, London Sage, Ch 8.
16
Cluster Sampling

• Divide population into clusters
• Randomly select clusters
• Collect data on all cases in cluster
• Watch cluster composition

From OLeary, Z (2004) The Essential Guide to
Doing Research, London Sage, Ch 8.
17
Cluster Sampling - Montréal
18
Non-Probability Samples

• Accidental
• Purposive

19
Convenience Sample

• Choose most convenient people
• Subject to multiple biases

From OLeary, Z (2004) The Essential Guide to
Doing Research, London Sage, Ch 8.
20
Volunteer Sampling

• Ask for volunteers
• Directly
• Ads
• Selected groups
• Unclear biases

From OLeary, Z (2004) The Essential Guide to
Doing Research, London Sage, Ch 8.
21
Strategic Sampling

• Handpick respondents for strategic purposes
• Useful for marginalized or small groups
• Control biases by specifying criteria

From OLeary, Z (2004) The Essential Guide to
Doing Research, London Sage, Ch 8.
22
Snowball Sampling

• Identify strategic respondent
• Ask for referrals
• Continue the process
• Use multiple starts to avoid network biases

?
From OLeary, Z (2004) The Essential Guide to
Doing Research, London Sage, Ch 8.
23
Probability Samples Multiple

• Mixture of various techniques

24
Sample Size Depends On

• Size of the population
• Accuracy desired
• Confidence desired
• Variation in the phenomenon investigated

25
Assignment

• Specify a research question you would like to
  answer.
• Describe a sampling procedure along with the data
  you would collect to answer the question.
• Identify the following
• the population
• the sample unit
• the sample frame
• the type of sample

26
Samples

• Non-probability
• Accidental
• Purposive
• Probability
• Simple random
• Systematic
• Stratified
• Proportionate
• Disproportionate
• Cluster
• Multiple

27
Constructing Questionnaires
28
Types of Research Interviews

• Questionnaires
• Closed and some open-ended questions
• Semi-structured interviews
• Open and some closed-ended questions
• Unstructured interviews
• Open-ended questions

Research Interviews are NOT what you see on TV
29
Advantages and Disadvantages

• Questionnaires
• Economic
• Speedy
• Minimal interviewer bias
• Anonymity
• Low response rates
• Limited exploration
• Comparison easy

• Semi-structured
• High response rates
• Can ask complicated questions
• Can follow-up Qs
• Can observe
• Expensive
• Time consuming
• Comparison difficult

30
Preparing Questions

• Resist the temptation to start with specific
  questions
• Clarify the RESEARCH question
• Identify the types of information required to
  answer the research question
• Within each of the types of information
  brainstorm for specific questions to ask
• Rearrange questions

31
Question Order

• Leave sensitive issues for later - once rapport
  has been established
• Follow from general to specific
• Use tables and grids where appropriate
• Use cards for sensitive issues if appropriate
• Prepare your probes if interview
• Follow a natural flow

32
Anticipate the Analysis

• Produce dummy tables
• Work backwards from your analysis

33
Format examples

• Contingency question
• Full filter question
• Quasi-filter question
• Closed question
• Partially open question
• Open question
• Probes

34
Simple, Mutually Exclusive

• What is your gender? (Circle the appropriate
  number)
• Female . . . . . . . . . . . . . . . 1
• Male . . . . . . . . . . . . . . . . . 2

5
35
Multiple Choice, not Mutally Exclusive

• What modes of transportation did you used to get
  to school last week? (Circle all that apply)
• Metro . . . . . . . . . . . . . . . . . 1
• Bus . . . . . . . . . . . . . . . . . . . 1
• Car . . . . . . . . . . . . . . . . . . . 1
• Bicycle . . . . . . . . . . . . . . . . . 1
• Walk . . . . . . . . . . . . . . . . . . 1
• Other (please specify)
• _____________________ 1

6
7
8
9
10
____ 11-12
36
Rank-order

• Rank order the 3 most important aspects of your
  job (Place 1 beside the most important, 2 beside
  the next most important, and 3 beside the next
  most important)
• Money . . . . . . . . . . . . . . . . . . ___
• Security . . . . . . . . . . . . . . . . . ___
• New experiences . . . . . . . . . . ___
• Power . . . . . . . . . . . . . . . . . . ___
• Excitement . . . . . . . . . . . . . . ___
• Other (please specify)
• ________________________ ___

13
14
15
16
17
____ 18-19
37
Pretest, pretest, pretest

• Use your friends and family
• Discuss
• Where it was clear or not clear
• What they thought about when responding to each
  question
• What they thought you were getting at
• Move to colleagues
• Check it out on people who are similar to those
  you will eventually survey

38
Problem Questions

• Jargon, slang, abbreviations
• What is your Internet browser?
• Ambiguity, confusion, vagueness
• What is your income?
• Emotional language and prestige bias
• Should we put murderous terrorists in jail?
• Do you support Prime Minister Chretiens policy
  on world trade?
• Double-barrelled questions
• Should marijuana be legalized for medical or
  other purposes?

39
Problem Questions cont

• Leading questions
• Do you feel that governments should have less
  power to interfere in business?
• Beyond respondents competence
• Does your mother support extramarital sex?
• False premises
• How can we halt the rising crime rate?

40
Problem Questions cont

• Future intentions
• How will you vote in the next federal election?
• Overlapping or unbalanced response categories
• What is your opinion regarding abortion?
• Very favourable
• Favourable
• Neutral
• Opposed

41
Materials

• http//artsandscience.concordia.ca/soci613br
• Check the Readings link
• SPSS Keywords support for questions
• Jackson article questionnaire design
• Selltiz checklist
• This Powerpoint presentation
• Texts

42
Questionnaire Construction

• The amount of time spent in the preparation of
  the questionnaire is directly related to the
  value of the results.
• Surveys usually cost a lot and you can't redo
  them.
• Know what you are wanting to discover

43
Research Strategies

• Questionnaires for generalizability
• Exploration well selected, in-depth interviews
• Use interviews to design survey
• Use comparisons

44
Sampling and Questionnaires

• Bill Reimer
• reimer_at_alcor.concordia.ca
• http//reimer.concordia.ca/teaching

45
Assignment