Use of Surveys in Program Evaluation

1
Use of Surveys in Program Evaluation
2
Types of Surveys

• Mail
• Phone
• Face-to-face
• Magazine
• E-mail
• Internet
• Call in
• Fax

3
Does Method Matter?Roanoke Times 1998 Survey of
Best Motion Pictures

• Mail Responses
• Gone with the Wind
• Sound of Music
• Wizard of Oz
• Its a Wonderful Life
• To Kill a Mockingbird

• E-mail Responses
• Gone with the Wind
• Star Wars
• Schindlers List
• Wizard of Oz
• Shawshank Redemption

4
Increasing Response RatesMail Surveys

• Include a small token of appreciation (25 or a
  pen)
• Precontact participants
• Use a first-class stamp (15 more likely to be
  opened)
• Send follow-up letters
• These factors dont affect response rates
• Survey length
• Personalization
• Deadlines
• Promising anonymity

5
Increasing Response Ratese-mail Surveys

• Compared to regular mail, e-mail
• Faster
• Cheaper (5-20 of regular mail cost)
• Results in longer, more candid open-ended
  responses
• Has similar response rates (about 30)
• Survey length does not affect response rates

6
Increasing Response RatesPhone Surveys

• Immediately identify self and affiliation
• Provide a phone number if participant is
  suspicious
• Stress the importance of the information
• Keep the interview short
• Limit the number of response options
• Speak clearly

7
Question Considerations

• Will the participant understand the question?
• Will the question itself change the way a person
  thinks?
• Do the response options cover the construct?
• What are we going to do with the data?
• What question are we trying to answer?
• How much time, effort, and money are we willing
  to spend in coding and analyzing responses?
• Does the format increase or decrease the
  probability of responding?

8
Question Types

• Open-ended items
• Provide richer quality
• Difficult to analyze
• Restricted items
• Easier to analyze
• May limit responses

9
Open v. Categorical Questions

• Age _____
• Age
• Under 21
• 21 25
• 26 30
• 31- 40
• 41 50
• Over 50

10
Asking About Education

• Years of education
• Degrees
• Years of college
• College credits

11
Rating Scales

• Yes/No
• Agree/Disagree
• Like/Dislike
• Like me/Not like me
• True/False
• Frequency (e.g. never, often)

12
Rating Scales

• Number of points on the scale
• Benchmarks
• At each end
• At each number

13
What is Wrong With These Questions?

• In the past year, how many times did you play
  golf?
• How many times per week do you drink alcohol?

14
Ethical Issues

• Anonymity
• Confidentiality
• Purpose of survey
• Bias