Moving Towards Inclusive Evaluation

1
Moving Towards Inclusive Evaluation

• Julie Moore Kelsey Deane
• Foundation for Youth Development
• University of Auckland

2
FYD and Project K

• A 14-month programme
• for Year 10 students
• Age 14-15
• Experiential and intensive
• positive changes are sustained
• 3 stages

For Young New Zealanders

Stars
Kiwi Can
Project K
FOUNDATION FOR YOUTH DEVELOPMENT
3
Project K

• Wilderness Adventure
• 3 weeks
• Students are physically and mentally challenged
• can include tramping, abseiling, kayaking,
  camping and mountain biking

4
Project K

• Community Challenge
• Adapt lessons learned to their community
• Discover the local resources, opportunities and
  support available
• Undertake a community project to give something
  back
• Set academic, health, fitness and personal goals

5
Project K

• Mentoring
• 12 months of one-on-one mentoring
• Positive changes from first two stages are
  sustained and strengthened
• Mentors are screened, trained, approved and
  supported during mentoring

6
Evaluation of Project K

• Designed by FYD
• Assess effectiveness
• Learn and develop
• Carried out in partnership with the Ministry of
  Social Development (MSD).
• September 2004 through June 2007
• To evaluate outcomes for young people who
  participate in Project K

7
Evaluation of Project K
Baseline
End of programme
One/Three Year post
Intervention
Self-Efficacy
End of Wilderness Adventure questionnaire
presentation
Baseline Measures

Health Lifestyle
End of Community Challenge questionnaire
presentation
Goal Achievement
NCEA Results
Mentoring Goal setting Mentoring contacts
relationships Goal review
Enrolment Status
Student Transition
Social Competence
8
Randomised control trial

• A randomised control trial (RCT)
• Participants were randomly allocated to Project K
  or control group and outcomes of the groups
  compared

9
Experimental Designs

• Acknowledged as a superior research design
  (Lipsey Cordray, 2000)
• But is challenging to use in the real-world

10
The Way Forward

• Inclusive evaluation
• Research partnerships
• Youth Advisory Group

11
The Evaluation Literature

• Two predominant aims Discovering the truth
  through scientific inquiry and producing useful
  information
• BUT
• Problems with the usefulness of evaluations in
  the past (Patton, 1997 Weiss, 1998)

Source Angelica, Emil, (2001). The Wilder
nonprofit field guide to crafting effective
mission and vision statements. Saint Paul, Minn.
Amherst H. Wilder Foundation
12
Including Stakeholders

• intended users are more likely to use
  evaluations if they understand and feel ownership
  of the evaluation process and findings they are
  more likely to understand and feel ownership if
  theyve been actively involved (Patton, 1997, p.
  22).

13
Tools Used to Include Stakeholders

• The Communication Network

WIDER COMMUNICATION NETWORK
2nd Reporting Cycle
2nd Reporting Cycle
FYD
1st Reporting Cycle
RESEARCHER
14
The Evaluation Questions Voting Survey

• You have the opportunity to use 5 votes to
  select your priority questions. You may divide
  the votes any way you like.
• What is Project K in concept (what is it supposed
  to be doing) in practice (what is it actually
  doing)? Are Project Ks operations consistent
  with Project Ks conceptual theory?
•  
• To what extent was each Project K objective
  achieved? Were these outcomes maintained (at
  one-year and three-years post-programme)?
•  
• What are the long-term (broad) impacts of Project
  K?
• Is Project K working? Should it be revised?
•  
• What changes in the Project Ks design or
  implementation might produce better outcomes?
• Why does Project K work (or if it doesnt, why
  not)? How does Project K produce outcomes?
•  

15
The Programme Logic Model

• a program logic model is a picture of how your
  program works the theory and assumptions
  underlying the programThis model provides a road
  map of your program, highlighting how it is
  expected to work, what activities need to come
  before others and how desired outcomes are
  achieved. (W.K. Kellogg Foundation Evaluation
  Handbook, p. 35).

16
(No Transcript)
17
(No Transcript)
18
An Integration of the Primary
Evaluation Goals

• Program Theory-Driven Evaluation (Donaldson,
  2007)
• Tools described above were uplifted from this
  approach and adapted to suit the current
  evaluation context of Project K
• Program impact theory developed with stakeholders
• List of questions generated and prioritised
• Data collected in rigorous way while paying
  attention to the practical constraints

19
End Goal
Everyone Heading in the Same Direction
Source Angelica, Emil, (2001). The Wilder
nonprofit field guide to crafting effective
mission and vision statements. Saint Paul, Minn.
Amherst H. Wilder Foundation
20
References

• Donaldson, S. I. (2007). Program Theory-Driven
  Evaluation Science Strategies and Applications.
  New York, NY Lawrence Erlbaum Associates.
• Lipsey, M. W., Cordray, D. S. (2000).
  Evaluation Methods for Social Intervention.
  Annual Review of Psychology, 51, 345-375.
• Patton, M. Q. (1997). Utilization-focused
  evaluation the new century text (3rd ed.).
  Thousand Oaks, Calif. Sage Publications.
• Weiss, C. W. (1998). Have we learned anything new
  about the use of evaluation? American Journal of
  Evaluation, 19(1), 21-33.
• W.K. Kellogg Foundation (2000). Logic Model
  Development Guide. Battle Creek, Mich. W.K.
  Kellogg Foundation.