Sustaining Community Projects: Logic Model Construction and Implementation

1
Sustaining Community Projects Logic Model
Construction and Implementation

• CYFAR Evaluation Team
• CYFAR, 2005

2
Introduction

• Across the United States there are a multitude of
  programs designed to improve the quality of life
  for individuals, families, and communities.
• Evaluation is essential for successful
  programming to document what happens in the
  program, demonstrate which strategies work best,
  and assess the short-term and long-term outcomes
  of the program.

3

• Understanding how a program achieves results is
  critical to determine if program elements should
  be modified and provide stakeholders with
  evidence that the program is effective.
• Evaluation theory provides the basic conceptual
  framework for thinking about problems and how
  change should occur.
• Logic models provide valuable ways to organize
  that thinking around evaluation theory.

4
Logic Model

• Program theory articulates program impact and
  program processes.
• Logic models provide a visual picture of program
  theory in action (Mancini, Huebner, McCollum and
  Marek, 2005).
• Of primary importance in developing logic models
  is establishing and maintaining a focus on
  results rather than on activities.

5

• Logic models help program professionals,
  evaluators and stakeholders reach consensus about
  which elements are essential to the program
  (Millar, Simeone, Carnevale, 2001 Orthner
  Bowen, 2004).
• Consequently program professionals and
  researchers are more able to identify faulty or
  implausible links early on.

6
SCP Logic Model

• The logic model you are asked to complete
  includes five elements Identified needs and
  assets, Desired results (short-term and
  long-term), Indicators (short-term and
  long-term), Activities, and Resources.
• These are found on the Logic Model Template
  Worksheet.

7

• Desired Results Worksheets are provided to help
  you detail information necessary to describe
  anticipated results.
• An Analysis column is included on the Desired
  Results Worksheets that allows you to discuss how
  you will analyze the data that are collected.
• To assist you in completing your Program Logic
  Model, please refer to the sample logic model of
  the After School Program.

8
Step 1 Identified Needs and Assets

• The first step is to identify needs and assets in
  the community which allow program personnel to
  clearly identify and understand the issues.
• Because this is the basis for the entire program,
  it is important to use accurate and valid
  information rather than uninformed hunches.
• Be sure to look at needs and assets at many
  levels, from the community systems such as
  health, education, legal, etc. to families and to
  individuals.

9

• Needs and assets analysis is a systematic way of
  gathering information that helps set priorities
  for action while also recognizing the assets that
  already exist in a community.
• In the logic model process the information that
  evaluators are gathering to understand severity
  of a problem also assists them in determining
  program priorities, and results that accrue from
  program activities.
• Census data, observations, records and existing
  data bases, and literature reviews are sources of
  existing information to identify needs and assets
  (Witkin Altshuld, 1995).

10
Step 2 Desired Results

• The second step in the logic model process is to
  articulate the desired result or change you
  expect to see because of your program.
• This is arguably the most important aspect of a
  logic model because it provides the focus that
  all successful programs need.
• At the completion of the program, what will be
  different?

11

• Results should be defined as short-term and long-
  term.
• Specifying results is useful because it enables
  program professionals to have a more realistic
  sense of how parts of the program theory fit, and
  what can reasonably be expected to change over a
  specific time period.
• Demonstrated results are not expected to be
  documented by following the same participants for
  five years, but rather are focused on all
  participants, or each group, as they complete
  participation in programs.

12
Short-Term Results

• Changes in knowledge, attitudes, and/or behaviors
  that are anticipated as a result of participating
  in programs can be considered short-term results.
• These results should be reported at the end of
  each of the five years.

13
Long-Term Results

• System change or changes in organizations,
  community capacity, program support, and/or
  program sustainability may be considered
  long-term results.
• Progress toward long-term results can be reported
  for each or any of the five years but MUST be
  reported in year 5.

14
Step 3 Indicators

• Indicators flow logically from the desired
  results. How will you know if the results have
  been achieved, with what evidence?
• By determining program indicators, program
  activities and results can be measured and
  tracked. Consequently notions, hunches, hearsay,
  and anecdotes are not good indicator candidates.

15

• Indicators can be far-ranging, including results
  from surveys and focus groups, agency
  administrative data, population data, as well as
  other measures (DeVellis, 2003 McKillip, 1998).
  
• Evaluation Work Sheets are included to help you
  specify program indicators for each of your
  program goals.

16
Step 4 Activities

• Activities describe programs that are implemented
  to achieve the desired results.
• Program activities become important only when
  they are linked to desired results (Bowen et al.,
  2001).
• Attention to the choice of programs, research
  upon which it is based, population to be served
  and critical elements of successful programs must
  all be considered.
• By referring to the Guiding Principles for the
  National Outcome that you have chosen for your
  project, you will be assisted in determining
  appropriate activities for your projects based
  upon the desired results that you have determined.

17
Step 5 Resources

• Resources include the people, stakeholders,
  curricula, spaces, approvals, funds and anything
  else needed to conduct the activities to achieve
  the desired results to meet the identified needs
  and capitalize on the available assets.

18
Monitoring and Evaluation

• An important part of the logic model is how the
  intervention will be monitored and evaluated,
  including program implementation and its intended
  results.
• Monitoring and evaluation permeate all parts of a
  logic model.

19

• There are multiple objectives for monitoring and
  evaluation
• to identify the initial extent of need
• to track how the need may change over time
• to document how well program implementation
  protocols are being followed
• to mark progress toward achieving results
• to know when results have been met and
• to discern how resource use has supported
  particular program efforts and subsequent results.

20
Logic Model Work Sheets

• The Logic Model Work Sheets allow you to put your
  Program Logic Model to work for you.
• These worksheets are provided to help you
  identify the anticipated results (short-term and
  long-term) for your program as well as
  indicators.
• You will also need to detail the data collection
  process sources of data, time of collection, and
  method of collection.
• Finally, you will also note the analyses you
  anticipate conducting.

21

• Please refer to the Program Logic Model Example
  for more detail on how you can complete these
  sheets for your project.
• For example, in the sample logic model, the
  Short-Term Desired Result Work Sheet shows the
  indicators or evidence for knowledge, attitude,
  and/or behavior change.
• The work sheet clearly shows what data are
  needed, when it will be collected, by whom, and
  how it will be analyzed.
• Completion of these evaluation work sheets will
  guide the entire data collection, analysis and
  reporting processes over the course of the
  project.

22
Logic Model Template
Identified Needs and Assets
Indicators Short Term Long-term
Results Short Term Long-term
Activities
Resources
Note Monitoring and evaluation activities
permeate all aspects of the logic model.
23
Example of Logic Model and Work Sheets

• The following provides an illustration and
  application of the logic model using an after
  school program as the example.
• This example is designed to stimulate your
  thinking about how to most effectively integrate
  your plan with this results-oriented logic model
  template.
• We anticipate that your logic model will contain
  more detail and be more comprehensive.

24

• The Logic Model is based upon the School-Age
  (K-8) national outcome.
• The example shows the relationships between the
  identified needs and assets, the short-term and
  long-term desired results, indicators, activities
  and resources.
• The following work sheets for the Short-Term and
  Long-Term Desired Results provide further details
  regarding the indicators, including the sources
  of data, times and methods of data collection,
  followed by planned analyses.

25
Step 1 Identified Needs and Assets

• Lack of parental monitoring after school
• Many children in park and on streets in groups
  after school
• Church building available for program use
• Low homework completion rates among 3rd to 5th
  graders

26
Step 2 Desired Results

• Short-Term
• Homework completion rates for children in grades
  3-5 will increase
• Parents will gain knowledge of after school
  resources
• Parents will gain knowledge of the importance of
  parental monitoring

27
Step 2 Desired Results

• Long-Term
• Community support for after school programs will
  be sustained
• Parents will monitor their children

28
Step 3 Indicators

• Short-Term
• Attendance rate at after school program
• Increased rates of homework completion
• Increased parental requests for after school
  programs
• Long-Term
• Community collaboration and support for space
  rental, staff training, recruitment, parent
  education and funding
• Fewer children on streets in park after school

29
Step 4 Activities

• Develop and implement a structured tutoring
  program
• Develop and implement a multi-component after
  school program, including recreation
• Develop community media campaign on importance of
  parental monitoring and after school care
• Educate school, faith and business communities
  about importance of after school programs through
  information sessions

30
Step 5 Resources

• School, faith business community leaders
• Parents
• Children in K-5
• Staff time of existing organizations
• Local media cooperation
• Grant funds

31
Work Sheet Completion Short-Term Desired
Results

• Homework completion rates for children in grades
  3-5 will increase.
• Short-Term Indicator Homework completion rates
  in grades 3, 4, 5.
• Source of Data Teachers classroom records.
• Time of Collection These records are currently
  kept and reported quarterly to parents.
• Methods of Collection
• Individuals will be coded for confidentiality.
• Enrollment in the after school program will be
  coded yes or no, and homework completion rate
  will be reported as percentage of assignments
  turned in.
• Teachers will report to evaluator each quarter.
• Data Analysis
• Completion rates for each participant will be
  compared to the rate for the previous year using
  paired t-tests.
• Completion rates for children who are and are not
  in the after school program will be compared.

32
Work Sheet Completion Long-Term Desired Results

• Community support for after school programs will
  be sustained.
• Long-Term Indicators
• Increased community collaboration membership
• Increased sources of fiscal responsibility for
  program
• Source of Data Collaboration meeting minutes,
  Annual budget and financial sources for program,
  Administrator interviews
• Time of Collection Prior to program
  implementation and annually thereafter.
• Media and other relevant materials to be
  collected or documented in a portfolio as they
  occur.
• Methods of Collection
• Project evaluator and director will work together
  to document and collect minutes of collaboration
  meetings and other key events that influence
  system change and stakeholder involvement.
• Portfolios will be used to document progress.
• Data Analysis
• Formative evaluation during the first two years
  of the program will help refine and direct
  efforts at building community support.
• Evaluator will report at least bi-annually to the
  collaboration on progress made and key areas that
  need to be strengthened.