Survey Completion Rates and Resource Use

1

Survey Completion Rates and Resource Use at Each
Step of a Dillman-Style Multi-Modal Survey
Andreal Hassol, Holly Harrison, Ricki Jarmon,
Brenda Rodriguez, Austin Frakt Abt Associates
Inc.
2
Introduction

• Designing DC efforts becoming an increasingly
  complex task.
• Declining participation in surveys
• Projects must be competitive in costs
• Rely on multi-mode DC efforts to achieve high
  response rates
• Designing DC efforts we must weigh resources
  against expected returns.
• Seek mode(s) of DC yielding high quality data
  and high response rates.
• Minimizing nonresponse bias (increase
  generalizability of data)
• Delivered on schedule, on budget
• Total Design Method widely known / cited in
  the industry (Dillman, 1978).

3
Background Elements of the Total Design Method
(TDM)

• Respondent friendly questionnaire
• Topic of survey salient to survey recipient
• Up to 5 contacts with survey recipients (via
  mail, poss. with phone follow-up)
• Inclusion of a stamped return envelope
• Personalized Correspondence
• Token Financial incentives

4
Utilizing the TDM in this Project

• Lit. review - sought out resources that
  demonstrated results of TSD implementation and
  related costs and returns.
• Minimal work tangibly demonstrating effects of
  reminder post cards or additional mailings over
  time.
• This research seeks to fill that gap.

5
Project Overview

• Client Centers for Medicare and Medicaid
  Services (CMS)
• Response Rate Goal 70
• Mode of DC Mail Survey with Phone Follow-up.
• Field Period 21 weeks. (Mid-July to December
  2002).
• Sample 7,730 Community-dwelling Medicare
  Beneficiaries
• Topic Evaluation of beneficiaries experiences
  with a new Medicare
  Insurance Option.
• Design 6 contacts by mail with phone follow-up.
  
• No incentive.

6
Methodology

• 7,730 cases from Medicare Enrollment database,
  across 25 states.
• Included three subgroups
• Those enrolled in new Private Fee for Service
  (PFFS) plan
• Those remaining in traditional Medicare managed
  care plans.
• Those with Medigap supplemental policy
• Proxies allowed (spouse / partner / adult
  child)
• Toll-free number staffed by interviewers where
  recipients could dial to do survey or ask
  questions
• Conducted English and Spanish
• 13 page SAQ / CAPI

7
Timeline of Project

• Advance Letter
• Survey Packet I First Class Mail
• Reminder Postcard I
• Survey Packet II First Class Mail
• Postcard II
• Survey III Priority Mail
• Phone follow-up (47 of non responders matched)
• Continued receipt of mail quex

• Week 1
• Week 1
• Week 5
• Week 6
• Week 9
• Week 13
• Weeks 14-21
• Weeks 14-21

8
Sample Management

• Integral to success of multi-modal project.
• Database designed to fulfill receipting and
  reporting needs for mail and CATI completes.
• Downloads of CATI efforts each night
• Receipt of mail completes
• Enable us to track progress carefully by mode,
  sample type, site
• Minimized burden on sampled beneficiaries
• Avoiding re-mails to already complete or
  identified ineligible
• Avoid phone follow-up to those submitted by mail
• Minimized number of duplicate completes (mail and
  phone)

9
Completion Rate through the Field Period
Quex mailing Weeks 1, 6, 13 Postcards Weeks 5,
9 CATI 14-21
10
Completes by Week

• 87 of completes by mail
• 13 of completes by phone

11
Costs Incurred Through the Field Period

• DC Costs
• (87 of budget)
• 23 on phone
• 64 on Mail

Remaining 13 budget for DP
12
Results on Salience of Survey Topic
79
69
67
73
13
Conclusions

• TDM worked, minor adjustments to tailor to our
  population and study.
• Each remail effort saw spike in returns,
  peaking within 2 weeks of mail out
• Less noticeable returns following postcards
• Telephone efforts were 13 of completed
  surveys but 23 of costs.
• Salience of instrument proves true as highest
  RR in PFFS group (79)
• Respondent friendly design in quex skip
  pattern confusion yielded extra data
• Database essential component to success
• Minimized burden
• Maximize labor resources in phone follow-up
  effort to nonresponders only.
• Over 7 week dual effort 21 cases completed
  phone and mail.
• Attained a 73 response rate (goal 70), using
  AAPOR calc. standards.

14
(No Transcript)