Non-Experimental designs: Surveys

1
Non-Experimental designs Surveys

• Psych 231 Research Methods in Psychology

2
Non-Experimental designs

• Sometimes you just cant perform a fully
  controlled experiment
• Because of the issue of interest
• Limited resources (not enough subjects,
  observations are too costly, etc).
• Surveys
• Correlational studies
• Quasi-Experiments
• Developmental designs
• Small-N designs
• This does NOT imply that they are bad designs
• Just remember the advantages and disadvantages of
  each

3
Surveys

• What are they?
• Questionnaires and interviews that ask people to
  provide information about themselves
• Why conduct them?
• Best way to collect some kinds of information
• Descriptive, behavioral, and preferential
• e.g., demographic information, recreational
  behavior, and attitudes
• To compliment experimental work
• Good/common first step, can collect a lot of data
  about a lot of variables
• Do not have to directly observe behaviors

4
Surveys

• Advantages
• One can investigate internal events (for example,
  attitudes opinions)
• Can generalize about an entire population based
  on relatively small samples of individuals
• Large amounts of data can be collected quickly
  with relatively little cost (effort, time, etc.)
• But theyre often not as cheap as you may think

5
Surveys

• Disadvantages
• Correlational causal claims shouldnt be made
• Note surveys are sometimes used as part of an
  experimental design, which may allow testing of
  causal claims
• Non-response bias
• Why doesnt everybody respond?
• Does response rate interact with variables of
  interest?
• Large data sets are sometimes difficult to
  analyze
• Self-reports may not be truthful
• Response set - tendency to respond from a
  particular perspective
• Social desirability bias (e.g., how a moral
  person would answer)

6
Stages of survey research

• Stage 1) Identify the focus of the study and
  select your research method
• What are the objectives of the research?
• Is a survey method the best approach?
• What kind of survey should be used?

• Group administration
• Mail surveys
• Internet surveys
• Telephone surveys
• Face-to-face interviews
• Focus group interviews

7
Stages of survey research cont.

• Stage 2) Determining the research schedule and
  budget
• Stage 3) Establishing an information base
• Find out whats been done, whats known
• E.g., Find other related surveys
• Stage 4) Identify the sampling frame
• The actual population that the sample is drawn
  from (as opposed to the ideal population)
• Think of it as operationalizing the conceptual
  level population
• Be aware of potential coverage error when the
  sampling doesnt lead to a good
  representativeness

8
Stages of survey research cont.

• Stage 5) Determining the sample method and
  sampling size
• Review Probability and Non-Probability methods
• Voluntary response method
• Importance of sample size

9
Voluntary response methods

• A kind of convenience sampling methods commonly
  used

• Problem Typically only individuals with strong
  opinions respond, so the results are often
  extremely biased

Daily show clip
Another fun poll result
10
Importance of sample size

• Sampling error - how is the sample different from
  the population?
• Confidence intervals
• An estimate of the mean or percentage of the
  population, based on the sample data
• John Doe has 55 of the vote, with a margin of
  error 3
• Margin of error (that 3 part)
• The larger your sample size, the smaller your
  margin of error will be.
• Which would you be more likely to believe
• We asked 10 people
• We asked 1000 people

11
Importance of sample size

• Sampling error - how is the sample different from
  the population?

• Response rate
• What proportion of the sample actually responded
  to the survey?
• Hidden costs here - what can you do to increase
  response rates
• Non-response error (bias)
• Is there something special about the data that
  youre missing (From the people who didnt
  respond)?

12
Stages of survey research cont.

• Stage 6) Designing the survey instrument
• Question construction How the questions are
  written is very important
• Clearly identify the research objectives
• Do your questions really target those research
  objectives (think Internal and External
  Validity)?
• Take care wording of the questions
• Keep it simple, dont ask two things at once,
  avoid loaded or biased questions, etc.
• How should questions be answered (question type)?

13
Good and poor questions

• Good

Poor

• Was the FDC negligent by ignoring the
• warnings about Vioxx during testing
• and approving it for sale?
• Yes
• No
• Unsure
• Do you favor eliminating the wasteful
• excess in the public school budget?
• Yes
• No
• Unsure

• If the FDC knew that Vioxx caused serious side
  effects during testing,
• what should it have done?
• Ban it from ever being sold
• Require more testing before approving it
• Unsure
• Do you favor reducing the public school budget?
• Yes
• No
• Unsure

14
Good and poor questions

• Good

Poor

• Should senior citizens be given more
• money for recreation centers and
• food assistance programs?
• Yes
• No
• Unsure

• Should senior citizens be given more money for
  recreation centers?
• Yes
• No
• Unsure
• Should senior citizens be given more money for
  food assistance programs?
• Yes
• No
• Unsure

15
Good and poor questions

• Good

Poor

• Are you against same sex marriage
• and in favor of a constitutional
• amendment to ban it?
• Yes
• No
• Unsure

• What is your view on same sex marriage?
• I think marriage is a matter of personal choice
• Im against it but dont want a constitutional
  amendment
• I want a constitutional amendment banning it

Problem Biased in more than one direction
Problem Asks two questions
16
Survey Questions

• Question types
• Open-ended (fill in the blank, short answer)
• Can get a lot of information, but
• Coding is time intensive and potentially
  ambiguous
• Close-ended (pick best answer, pick all that
  apply)
• Easier to code
• Response alternatives are the same for everyone
• Rating scales
• Used for how much judgments
• e.g., measures attitudes, agree/disagree
• Take care with your labels
• Range of scores, anchors

17

• Closed-ended

• Open-ended

• What is the best thing about ISU? (choose one)
• 1. Location
• 2. Academics
• 3. Dorm food
• 4. People who sell things between Milner and the
  Bone

• What is the best thing about ISU?

Survey Questions
18

• if closed-ended
• decide number/label of response alternatives
• should use odd number (5 or 7 best)
• labels should be clear
• decide scale
• rating PSY 231 is an important course in the
  major.
• 1 2 3 4 5
• Strongly Agree
  Neutral Disagree Strongly
• Agree
  Disagree

Survey Questions
19

• if closed-ended
• decide scale
• semantic differential PSY 231
• Important _____ _____ _____
  _____ _____ Unimportant
• Boring _____ _____ _____ _____
  _____ Interesting
• nonverbal scale for children
• Point to the face that shows how you feel about
  the toy.

Survey Questions
20
Stages of survey research cont.

• Stage 7) Pre-testing the survey instrument
• Fix what doesnt seem to be working
• Stage 8) Selecting and training interviewers
• For telephone and in-person surveys
• Need to avoid interviewer bias
• Stage 9) Implementing the survey
• Stage 10) Coding and entering the data
• Stage 11) Analyzing the data and preparing a
  final report