1

• Advanced Data Collection
• January 27, 2005

2
Who We Are Innovation Network

• National nonprofit organization
• Committed to evaluation as a tool for empowerment
• Build evaluation capacity of nonprofits and
  funders so they can better serve their
  communities
• Practice a participatory approach

3
Who We Are Innovation Network

• www.innonet.org/services/LSC_Train

4
Objectives of This Session

• Review the five major groups of data collection
  methods
• Discuss some of the subtleties of web-based data
  collection

5
Data Collection

• Program Plan
• Evaluation Plan
• Data Collection Strategy

6
Data Collection 3 Steps

• Choose the method
• Decide which people or records will be the source
  of the information
• Determine the level of effort involved in using
  that method with that population

7
Data Collection Steps 1 2Choose Method,
Identify Source

• Review documents
• Observe
• Talk to people
• Collect written responses
• Pictorial/multimedia

8
Data Collection Step 3Level of Effort

• Instrument development
• Cost/practicality of actually collecting data
• Cost of analyzing and presenting data
• Also consider
• Communication Power
• Proxy Power

9
Document Review

• Examples
• Attendance records
• Application information
• Feedback forms
• Web statistics
• Curricula or content

10
Document Review

• Efficient and cost-effective
• Dependent on the existence of documentation
• Better for process evaluation than outcome
  evaluation

11
Observation

• When conducting and evaluation the observer
• Does not intervene in any way
• Does not stand out or draw attention
• Does not record names or any identifying
  information
• It is important that observations be held over
  time (and at different places/days/times)

12
Observation

• Inexpensive way to obtain an outside perspective
• Highly susceptible to bias
• Subjective relies on anecdotal data

13
Interviews

• Structured
• Follow a predetermined interview protocol
• Identify broad evaluation questions and specific
  interview questions in advance
• Unstructured
• Begin with one or two open-ended questions
• Proceed based on the responses received

14
Interviews

• Provides the story behind someones experiences
• Useful as a follow-up to a survey to further
  investigate responses
• Time consuming (both in terms of collection and
  analysis)
• Respondents susceptible to leading by
  interviewers

15
Interview Tips

• Make the respondent feel comfortable
• Introduce yourself and the evaluation
• Start with easy, short answer questions end
  with more revealing, long answer questions
• Record the interviews
• Important for capturing great quotes

16
Focus Groups

• A small group of people (who have something in
  common) who are led through a conversation by a
  moderator and discuss their opinions and provide
  suggestions on a topic.

17
Focus Groups

• Efficient and cost-effective way to gather
  qualitative data
• Requires a skilled moderator/facilitator
• Collects consensus rather than individual
  perspectives

18
Focus Group Tips

• Ideally 8-10 person groups
• Maximizes amount of information you can gather
  without the suppression of voices
• Dont get bogged down
• Rewards
• Breakfast/Lunch/Dinner
• Gift certificates

19
Surveys

• Delivery options
• By mail
• In person
• Online
• By telephone

20
Surveys

• Collect quantitative information
• Efficient
• Open-ended questions (qualitative data) can be
  cumbersome for respondents and data entry

21
Survey Tips

• Introduce the survey
• Describe the purpose and what will be done with
  the results
• Reinforce that it is confidential and that you
  request they respond candidly
• Lead with easy, closed-ended questions
• Group categories of questions together
• Consider how long it will take for someone to
  respond rather than the length of pages

22
Web Statistics

• of hits
• Most popular pages
• Most popular downloads
• Common drop off points
• Where are they coming from?

23
Dont Be Afraid to Ask More!

• Why are they there?
• Who are they? What do they do?
• What are they looking for?
• Did they find it?
• What are they going to do with it?
• Are they willing to help you with future
  evaluation projects?

24
Usability Testing

• We can help
• ereed_at_innonet.org
• 202.728.0727 x 109

25
Next Steps

• We can do more!
• Online, instructor-led trainings
• Online, self-paced modules
• Individual technical assistance
• www.innonet.org/services/LSC_Train

26
Next Steps

• We can do more!
• Survey Builder
• Survey Library
• Data Analysis Tool

27
Thanks for Your Participation!
Measure results. Make informed decisions.
Create lasting change.
Innovation Network, Inc. 1625 K St. NW, 11th
Floor Washington, DC 20006 (202)
728-0727 www.innonet.org
Ehren Reed ereed_at_innonet.org www.innonet.org/ser
vices/LSC_Train