Rest of Course

1
Rest of Course
Proposals Research Design Measurement Sampling S
urvey methods Basic Statistics for Survey
Analysis Experiments Other Approaches-
Observation, qualitative methods
2
Topics for Today
1. Evaluation Research Proposal Format 2.
Introduction to Research Design Primary
Secondary Data Qualitative Quantitative
Approaches Survey Experiments On-Site
Household designs 3. Common Problems/Approaches
in PRTR
3
Purposes of Proposal

• Communicate with Client
• Demonstrate your grasp of problem
• Plan the study in advance, so others can evaluate
  the study approach
• will it work?
• have you overlooked something?
• will results be useful to client?
• Can we afford it?

4
Proposal Format
1. Problem Statement - define program to be
evaluated/problem to be studied, users uses of
results. Justify importance of the
problem/study. 2. Objectives Concise listing .
In evaluation studies, the objectives usually
focus on the key elements of program to be
evaluated the evaluation criteria. These are
the study objectives NOT the program
objectives. 3. Background/Literature Review -
place for more extensive history/structure of
program. Focus on aspects most relevant to
proposed evaluation. Discuss previous studies or
the relevant methods. 4. Methods - details on
procedures for achieving objectives - data
gathering and analysis, population, sampling,
measures, etc. Who will do what to whom, when,
where, how and why? 5. Attachments - budget,
timeline, measurement instruments, etc.
NOTE Most programs must be narrowed to
specific components to be evaluated. Think of a
Program of studies rather than a single
evaluation study. The proposal should define this
specific study how it fits into a broader
program of studies.
5
Sample Objectives
1. Estimate benefits and costs of program 2.
Estimate economic impacts of program on local
community (social, environmental, fiscal). 3.
Determine effects of program on target
population. 4. Describe users and non-users of
program 5. Assess community recreation needs,
preferences 6. Determine market/financial
feasibility of program 7. Evaluate adequacy or
performance of program
6
Typical Research Objectives
Describe a sample or population Identify/test
relationships between variables in a population
statistical cause-effect Quantify the
relationship
Average income of MI Snowmobilers in 1998 is 45K
SB with higher incomes spend more money After
safety program, SBs have fewer accidents SB
spend per day 25 .4 Income
7
Variable Terminology
Variables any characteristic that varies across
individuals in a population (i.e. takes on
different values for different individuals). Depen
dent variable is the one you are trying to
predict or explain, usually the focus of your
study Independent variables are the ones that
help explain the dependent variable. In Program
evaluation, the outcomes are generally the
dependent variables and characteristics of
program or target populations are independent. In
a cause effect relationship, cause is independent
effect dependent.
8
Methods Choices

• Overall Approach/Design
• Qualitative or Quantitative
• Primary or secondary data
• Survey, experiment, case study, etc.
• Who to study - population, sample
• individuals, market segments, populations
• What to study - concepts, measures
• behavior, knowledge, attitudes
• Cost vs Benefit of Study

9
Primary or Secondary Data

• Secondary data are data that were collected for
  some purpose other than your study, e.g.
  government records, internal documents, previous
  surveys
• Choice between Primary /Secondary Data
• Costs (time, money, personnel)
• Relevance, accuracy, adequacy of data

10
Qualitative vs Quantitative Approaches
Qualitative Focus Group In-Depth
Interview Case Study Participant
observation Secondary data analysis Quantitative
Surveys Experiments Structured
observation Secondary data analysis
11
Qualitative vs Quantitative
Quantitative Genl Laws Test Hypotheses Predict
behavior Outsider-Objective Structured formal
measures probability samples statistical
analysis
Qualitative Unique/Individual case Understanding M
eanings/Intentions Insider-Subjective Unstructur
ed open ended measures judgement samples
interpretation of data
Purpose Perspective Procedures
12

• Survey vs Experiment
• Survey - measure things as they are, snapshot of
  population at one point in time, generally refers
  to questionnaires
• (telephone, self-administered, personal
  interview)
• Experiment - manipulate at least one variable
  (treatment) to evaluate response, to study
  cause-effect relationships
• (field and lab experiments)

13
Research Designs/Data Collection Approaches
14
General Guidelines on when to use different
approaches

• 1. Describing a population - surveys
• 2. Describing users/visitors - on-site survey
• 3. Describing non-users, potential users or
  general population - household survey
• 4. Describing observable characteristics of
  visitors - on-site observation
• 5. Measuring impacts, cause-effect relationships
  - experiments

15
Guidelines (cont)

• 6. Anytime suitable secondary data exists -
  secondary data
• 7. Short, simple household studies - phone
• 8. Captive audience or very interested population
  - self-administered survey
• 9. Testing new ideas - experimentation or focus
  groups
• 10. In-depth study - in-depth personal
  interviews, focus groups, case studies