General Elements in Evaluation Research

1
General Elements in Evaluation Research
2
Types of Evaluations
3
Outcome (or impact) Evaluation

• One looks at what difference certain things
  (e.g., services and supports) make in outcome
  variables of interest.
• Outcomes typically include beneficial changes for
  those receiving services and fiscal and other
  effects for public entities.

4
Process Evaluation

• Typically, one looks at policies, procedures, and
  practices in terms of milestones, planned
  accomplishments, efficiency, and effectiveness.
• Process evaluations often relate to teamwork,
  project management, continuous improvement,
  and/or policy development.

5
Note

• Those who fund evaluations typically have a set
  of questions to which they want timely, relevant,
  and credible answers through systematic inquiry,
  using
• quantitative (counts proportions statistics)
  and/or
• qualitative (observations, interviews) techniques.

6
Research Designs
7
Background

• Valued outcomes depend on a whole host of
  factors.
• Factors in successful transitions from school
  include (1) person (2) experience in school
  (3) physical and social environment (4) paid
  people whose jobs include helping with
  transitions (5) opportunities for postsecondary
  education and training and (6) the local labor
  market.

8
Random Assignment

• Difficult to establish causality between an
  independent and a dependent variable.
• Threats to inferences are reduced if some of
  those interested in the services receive them,
  while others do not, and assignment to treatment
  and control groups is truly random.
• Such experimental designs typically reduce
  differences known to affect outcomes.

9
Random Assignment

• If there is control of the experiment so that
  enhanced services are only received by those in
  the experimental group, one may be more confident
  in assertions of likely causality.
• Such control, however, can be very difficult to
  achieve in natural settings.
• Enhanced services may inadvertently end up being
  provided to members of both groups.

10
Quasi-experimental Designs

• One may construct matched comparison groups,
  and follow both.
• Comparison groups should be similar in terms of
  factors known to be related to (predictive of)
  outcomes.
• The groups should be receiving similar services,
  except for the enhanced services being studied
  in terms of impact.

11
Quasi-experimental Designs

• Comparison group members should also be in
  similar schools receiving much the same services,
  and face similar postsecondary opportunities and
  labor market conditions.
• If these things are true, one may be able to
  control statistically for any remaining
  differences that make a difference.
• Reasonable inferences as to the apparent
  influence of the enhanced services can then be
  made.