Get What you Need from Online Surveys

1
Get What you Need from Online Surveys

• Janelle Commins, MSPH
• Research Associate
• EMT Associates, Inc.
• 15720 Ventura Blvd., Penthouse
• Encino, California 91436

2
Topic Overview

• When is an online survey appropriate?
• What do I have to choose from?
• Commercial or custom-built
• How do I get people to take MY survey?
• Case Study LA County Service Provider Survey.

3
When are Online Surveys Appropriate?

• Respondent factors.
• Internet access, geographic spread.
• Questionnaire factors.
• Type of question, length of questionnaire.
• Evaluator factors.
• Time frame, budget, technological expertise.

4
What Are My Options?
5
How Do I Get a Good Response?

• What IS a good response?
• Depends on population and relevance.
• For service providers 2-45 median 13.
• For market research lt 2.

6
Recruiting Strategies

• Get executive buy-in
• Send a personal letter to the Executive or
  Research Director
• Provide summary sheet with quick facts.
• Present at collaborative meetings.
• Cold-calling.

7
Response Strategies

• Distribute survey invitations through a trusted
  intermediary.
• Send a pre-announcement email.
• Emphasize relevance and simplicity
• Avoid using the word survey in the Subject
  line.

8
Case Study The Los Angeles County Service
Provider Online Survey
9
What is the Los Angeles County Service Provider
Online Survey?

• First 5 LA-sponsored.
• Countywide survey of agency decision-makers.
• Custom-built (JAVA).
• Evidence-based recruitment, distribution, and
  formatting.

10
Recruitment Strategy

• Executive buy-in at public institutions.
• Presentation at collaborative meetings.
• Cold calling.
• Outcome Distributed to 1,800 service providers
• (150 agencies)

11
Response Strategy

• Subject line asked for help and assistance.
• 7 Day Contact
• Day 0 Pre-announcement email.
• Day 2 Invitation email.
• Day 7 Reminder email.
• Outcome 23 response rate (N461)

12
Item Response Strategy

• Survey loaded upfront allowing users to glide
  through questions.
• Uncluttered appearance.
• Invisible drop-downs appeared depending on users
  selection.

13
Potential Obstacles
14
1 Coordinating Revisions

• Development Problem Unpredictable turnaround
  time.
• Solution Pilot survey and bundle revisions for
  programmer, understand complexity of revisions,
  set a realistic deadline.

15
2 Limited incentive

• Incentive What do agencies stand to gain from
  participating?
• Solution
• Offer agency administrators the opportunity to
  include a question of interest.
• Provide a summary of findings.

16
3 Technical incompatibility

• User Problem Slow loading time for those
  operating old PCs or early web browsers.
• Solution
• Know the user IT environment! What kind of
  browsers are they operating?
• Use graphics conservatively.

17
Scenario

• Very important stakeholder.
• 600 candidates
• Problem Dated web browsers took 1-2 minutes to
  load survey.
• Solution Eventually loaded survey in
  departments commercial software.
• Resolved lag time
• Lost smooth look and feel
• Required more data cleaning
• 25 response rate (N165 respondents)
• Typically institution receives a 2-12 response
  to internal surveys.

18
Comparison of Labor Time

• Custom built

• Off-the-shelf software

• 26 developer hours
• 461 responses
• 0.06 developer hours per response

• 8 developer hours
• 165 responses
• 0.05 developer hours per response

19
Lessons Learned

• Facilitating Recruitment.
• Get executive buy-in.
• Boosting Response.
• Use trusted intermediary.
• Send pre-announcement.
• Emphasize relevance and simplicity.
• Know the IT environment EARLY.

20
Take-Home Message

• Unless you are conducting a survey with unique
  questionnaire options, special formatting
  requirements, or complex data analysis,
  commercial software will probably suit your needs.

21
Questions?
Contact information Janelle Commins,
M.P.H. 818-990-8301 jcommins_at_emt.org