Evaluation%20research

1
Evaluation research

• Using research methods in combinations
• Policy analysis

2
Evaluation

• Examining the effects of programs, policies and
  laws
• Programs providing services to clients
• Policies rules for handling clients, actions
  taken i.e., policy on deadly force for fleeing
  felons
• Laws what are the real impacts

3
Evaluation

• People assumed that particular effects would take
  place
• However, those effects may or may not happen
• There may be unanticipated side effects to a
  change, both good and bad

4
Policy

• Usually some issue is raised
• Demands are made, and there is support and
  opposition
• Goals should be formulated and ways established
  to attain those goals (objectives)
• Policy output the program, resources allocated

5
Policy

• Then the impact is determined
• Did the changes in policy outputs have an impact
  on the problem?
• Hypothesis If some policy action is taken, then
  we expect a particular result
• To test the hypothesis, must do an evaluation

6
Evaluation components

• Process evaluation and impact
• Process
• Needs assessment and monitoring part of the
  process evaluation
• The first step should be needs assessment
• Frequently this is not done

7
Needs assessment

• Is the program/policy/intervention needed?
• Is the problem serious? Will it get better on
  its own?
• Who are the members of the target population?
  (those most in need of services, deterrents,
  etc.)
• What are their characteristics?

8
Needs assessment

• Programs will have little impact if they dont
  affect the target population
• Programs will have little effect if they do not
  take into account the characteristics of the
  target population

9
Needs assessment

• i.e., a school based program will not help
  chronically truant delinquents
• Adding 11th and 12th grade classes at a reform
  school will not be useful if very few delinquents
  at the school are functioning at even a 9th grade
  level
• Welfare to work requires child care

10
Needs assessment

• There is a need to gather information to
  determine the characteristics of the target
  population
• May not be readily apparent, as the target
  population might not be in the system
• Assessment methods records, key informants,
  gatekeepers, data collection

11
Needs

• If the target population is not reached, even
  good programs, policies and laws may not have an
  impact
• Weed and Seed example
• Adolescent drug offender example

12
Needs

• Establishment of goals and objectives
• Exercise what are the goals of a prison
• Mechanisms (objectives, activities) for meeting
  those goals
• Develop a list

13
In an evaluation

• The research must ask
• What does the program intend to accomplish?
• How do existing staff determine whether they have
  attained their goals?
• Have formal goals and objectives been identified
  and prioritized?

14
Evaluation

• What performance measures are currently used?
• Are they adequate, or must additional measures be
  developed as part of the evaluation?
• It would be very useful if evaluators were
  included at the start of a program to establish
  measures, seldom happens

15
Monitoring (process)

• What is actually happening in the intervention?
• What happens on paper, theoretically, may be
  different from what actually happens
• Who is actually served by the program?
• Are they members of the target population?

16
Monitoring

• If they are not, the program will probably have
  little impact
• how are people selected for inclusion, or how do
  they decide to participate?
• How do participants differ from members of the
  target population?
• How do potential participants differ from
  nonparticipants?

17
Monitoring

• How do staff feel about the program or policy?
• Example laws and prosecutors
• Juvenile judges and juvenile justice programs
• Is the program or policy actually being
  implemented?

18
Monitoring

• Are all the components being addressed?
• Example of drug courts
• Program components and benchmarks, means of
  assessing whether program components are being met

19
Monitoring

• Are there problems that were not anticipated?
• i.e., lack of community support, resistance to
  the program or policy, reactions of criminal
  justice personnel, unexpected obstacles

20
Monitoring

• Unanticipated side effects
• Could be positive or negative
• Positive conditions improve for people not in
  the target population
• Mental health centers, nutrition requirements for
  WIC
• Negative example of net widening
• Program may do something else