Healthy Schools Leadership Program (HSLP) Evaluation Plan

1
Healthy Schools Leadership Program
(HSLP)Evaluation Plan

• February 2006

2
What Im going to talk about

• Present get your feedback on Evaluation Plan
  for HSLP 06-08 cohort
• Generate ideas for your own assessment/evaluation
  work
• Ideas for using the Healthy Youth Survey
• ? Please consider this a discussion!

3
George tried to be a good monkey
4
The Primary Evaluation Question

• Is the Healthy Schools Leadership Program (HSLP)
  effective for making schools healthier?
• Note that this is a state-level evaluation
  question
• I will present an ambitious scenario for
  evaluation
• We will attempt to be as rigorous as possible
• Im not a schools expert you are the schools
  experts

5
Caveats to Evaluation Model

• Small numbers of schools (n15)
• Change takes time especially for long-term
  outcomes
• Our comparison group is the rest of the state
• If findings are positive, some will argue that
  this is the results of your existing
  capacity/readiness
• The plan is ambitious looking for additional
  resources to support evaluation

6
Id like to say something stronger

• What we can say right now with certainty is
  that if you spend resources implementing
  coordinated school health it doesnt appear to
  make your academic achievement any worse.

7
Describe the Program
8
Coordinated School Health Goals

• Health Education
• Physical Education
• Counseling/Support Services
• Food/Nutrition Services
• Healthy Environment
• Health Services
• Family/Community Involvement
• Staff Wellness

9
Simplified Logic Model
B. Activities Asst Plan Evaluate
A. Inputs Teams HSLP training
C. Outputs Asst results Action Plan
E. Intermediate Outcomes Policy, procedure,
environment changes Sustainability
D. Short-term Outcomes Results from action plan
activity
F. Long-term Outcomes Staff/studnt behavior
change
G. ImpactsHealth improvemt Academic achievemt
Catalysts
Barriers
Non-HSLP External Factors
10
Getting the Right Perspective Evaluation
Questions
11
Specific Aim

• 1. Describe the extent to which the inputs
  and activities are sufficient to generate outputs
  assessment results, action plans (A/B?C,
  Descriptive)
• Were teams able to assemble wellness committees?
  How did they function? Were they able to complete
  assessment and planning? What priorities were
  identified? What were the barriers and motivating
  factors related to output completion?
• Are there differences among HSLP teams in
  satisfaction with outputs? What are the factors
  (HSLP or other) associated with high/low
  satisfaction? Was HSLP a critical factor in
  achievements of high-quality outputs?
• Data Sources Team interviews, action plans,
  Healthy Schools Report Card

12
Hypothetical Findings

• All schools assembled teams X included
  community members
• 12 of 15 teams created formal action plans
• All schools identified at least one nutrition
  priority
• School teams that were very satisfied with
  their plans were more likely to have involved
  community members and had a clear process for
  translating assessment results into an action plan

13
Specific Aim

• 2. Describe the extent to which participation
  in the HSLP is associated with short-term
  outcomes early results of action plans (A/B?D,
  Descriptive)
• Were school wellness teams able to use plans to
  initiate change? How did implementation activity
  begin? Which priorities did schools take action
  on first? How were responsibilities identified?
  What factors were associated with transition from
  planning to action?
• Are there differences among HSLP schools in their
  satisfaction with/quality of early
  implementation? Were some kinds of activities
  more easily translated to quality action? What
  are the factors (HSLP or other) associated with
  high/low satisfaction? Was HSLP a critical factor
  in initiating high-quality work?
• Data Sources Team interviews, action plan
  reports, Healthy Schools Report Card

14
Hypothetical Findings

• 10 of 15 schools increased the overall number of
  maintenance items on their Healthy Schools
  Report Card
• Teams reported highest satisfaction with their
  work on XXX priorities
• XX Teams (X) said they strongly agreed that
  change would not have occurred without the HSLP
  training and technical assistance

15
Specific Aim

• 3. Describe the extent to which schools that
  participated in HSLP change their environments
  (A/B?E, Pre/post comparison to non-HSLP schools)
• Did HSLP school environments change, and were
  changes different than those of non-participating
  schools? Which changes were achieved?
• Are there differences among HSLP schools that
  achieved policy/procedure/environment change vs.
  those that did not?
• How has sustainability been addressed? What
  additional resources have been attracted for
  school health? Has institutionalization of school
  health work been achieved?
• Data Sources SHEP, Healthy Schools Report Card,
  HYS (perceived school environments), Staff
  wellness survey (if perceived environment
  indicators included)

16
Sample Findings
Source SHEP 2004
17
Specific Aim

• 4. Describe the extent to which schools that
  participated in HSLP change their student/staff
  behaviors (A/B ? F, Pre/post comparison to
  non-HSLP schools for students, self-comparison
  for staff)
• Did HSLP schools student behaviors change, and
  were changes different than those of
  non-participating schools?
• Did HSLP schools staff behaviors change?
• Are there differences among HSLP schools that
  achieved behavior change vs. those that did not?
• Data Sources HYS, Staff Wellness Survey

18
Sample Findings
Source Healthy Youth Survey 2002-2004
19
Specific Aim

• 5. Describe the extent to which schools that
  participated in HSLP have had health impact (A/B
  ? G, Pre/post comparison to non-HSLP schools for
  WASL, self-comparison for absenteeism)
• Did HSLP schools student academic achievement
  change, and were changes different than those of
  non-participating schools?
• Did HSLP schools student and/or staff
  absenteeism rates change?
• Are there differences among HSLP schools that
  achieved impact vs. those that did not?
• Data Sources WASL, Student absenteeism measures,
  Staff absenteeism measures

20
Sample Findings
21
Data Sources Gathering Credible Evidence
Not everything that counts can be counted and
not everything that can be counted counts.
Albert Einstein
22
Data Sources

• Proposing to use data you already collect, or new
  data collection related to your HSLP
  implementation
• Exception interviews of team/administration
• For anything where we want to look at everyone
  together, methods have to be standard
• For some things it would be fine to have sentinel
  schools
• See school-specific data suggestions later

23
Team Interviews

• During Spring 06 develop interview protocols for
  assessing function and satisfaction among HSLP
  participants (see evaluation questions)
• Repeat in Spring 08
• Augment with interviews with other wellness
  committee members (or lead administrators?)
• Your role participate in interviews

24
Healthy Schools Report Card

• Conduct assessment to identify priorities for
  action planning
• ASCD (Association for Supervision and Curriculum
  Development) has a clear process/instrument
• Your role use good assessment to develop plans,
  provide a copy of whatever baseline (spring 06)
  and follow-up (spring 08) assessment results

25
HSRC Component Areas

• School health program policy strategic planning
• Coordination of school health programs
• Social and emotional climate
• Family and community involvement
• School facilities and transportation
• Health education
• Physical education and physical activity
• Food and nutrition services
• School health services
• Counseling, psychological and social work
  services
• School-site health promotion for staff

26
HSRC Indicator Priority Rankings

• Multiple specific topics within components
• Rate with your team for each current status,
  perceived benefit, perceived effort
• Color-coded result report for combined scores

• Green short-term high priority
• Red short-term lower priority
• Lime long-term, highest priority
• Blue long-term, high priority
• Orange long-term, lower priority
• No color Maintenance

27
HSRC Sample Component-Topic

• School health program policy strategic planning
• All staff members are provided with
  time/resources to comply with health program
  policies
• Coordination of school health programs
• School health program members regularly inform
  the principal, district superintendent, or school
  board of current developments in the school
  health program.

28
Annual Action Plans Reports

• Design your action plan based on assessment
  results
• Your role provide action plan annually, report
  of progress annually (format?)

29
School Health Education Profile (SHEP)

• Survey of Secondary School Principals and Lead
  Health Teachers
• Sponsored by CDC
• Will be sent in Spring 06 and Spring 08
• Your role encourage completion of the survey

30
SHEP/Profiles Data ExamplesSchool Health
Advisory Groups

• Have a School Health Advisory Group
• 59 of high schools
• 56 of middle schools
• 46 of combined secondary schools
• 55 overall
• Similar to 52 total in 2002

31
SHEP/Profiles Data ExamplesTobacco

• A designated individual has primary
  responsibility for seeing that the tobacco
  prevention policy is enforced
• 64 high school
• 68 middle school
• 59 secondary combined
• 65 overall
• Similar to 68 overall in 2002
• No association with school advisory presence

32
SHEP/Profiles Data ExamplesNutrition

• Students can purchase fresh fruits/vegetables
  from vending machines or at the school store,
  canteen, or snack bar
• 49 high school
• 45 middle school
• 39 secondary combined
• 46 overall
• Slight improvement from 39 overall in 2002
• No association with school advisory presence

33
SHEP/Profiles Data ExamplesAsthma

• Obtain and use an Asthma Action Plan (or
  Individualized Health Plan) for all students with
  asthma
• 77 high school
• 77 middle school
• 74 secondary combined
• 76 overall
• Improvement from 62 overall in 2002
• Schools with advisories are twice as likely to
  obtain asthma plans as schools without advisories

34
Staff Wellness Survey

• Designed by CDC available in March
• Assuming we can modify (shorten) if needed
• Assuming we (DOH/OSPI) can provide as online
  survey and give data back to school teams
• Are there other ways to measure staff absences?
• Your role engage staff to participate
• Caution poor participation will not yield useful
  data!

35
Healthy Youth Survey (HYS)

• Administered Fall of even years
• Grades 6,8,10,12
• How many HSLP schools participated in 2002, 2004?
• Most powerful comparison will be 02-04-06
• Your role register to participate in 2006 (soon!)

36
Student Absenteeism

• NCLB requires measurement of excused/unexcused
  absences
• Assuming no change in non-illness excused
  absences, we can monitor changes in this rate
  over time
• Your role provide annual absence rates

37
Academic Achievement/WASL

• Annual, building-level
• Grades 4,7,10
• Math, Reading, Writing
• Your role none needed

38
Other School Factors

• Free reduced lunch enrollment this may be an
  outcome, consider different measure of SES
• Percent minority enrollment
• School/district size/enrollment
• School level (elementary, middle, high,
  combination)
• Urban/rural location
• Baseline levels of outcomes
• Other (baseline) funding sources for health
  improvement
• Your role none we can obtain

39
Too much information?
40
Timeline
41
School-level Evaluation

• We wont evaluate you as individual teams
• Evaluating yourself on a few points would be
  helpful in telling your story later
• Suggest designating a lead for local evaluation
  (just like any implementation piece)

42
Sample School-level Evaluation Data

• vending revenue data
• key informant interviews with students or staff
• tracking of policy enforcement activities
• other data gathering from communities
• health room utilization data
• student absences use more advanced tracking
  system (Public Health Tracking System grant
  available from DOH environmental health)
• lots of (digital) pictures of your environment
  now and later

43
Still Awake??
44
Healthy Youth Survey
45
Why do I need HYS now?

• Contribute to assessment, planning
• Identification of your priorities
• Describing affected students
• Gain attention for student health as a priority

46
HYS 2004 Collaborative Effort of OSPI, DOH,
DSHS, CTED, FPC, and RMC Research

• State-level simple random sample of schools
• County-level samples drawn as appropriate
• Non-sampled schools also invited to participate
• Survey consistently administered in the Fall of
  even years
• Survey given to 6th, 8th, 10th, and 12th graders
• Survey booklets have one-page tear-off answer
  sheet
• Survey for 8th, 10th, and 12th graders uses 2
  form interleaved administration

47
HYS 2004 Participation

• The Healthy Youth Survey 2004 was completed

By 185,095 students
In 1,013 schools
In 235 school districts
In all 39 counties.
48
Risk Protective Factor DataConsistent with CTC
MTF SurveysDevelopment led by DSHS/DASA OSPI

• WSSAHB Questionnaire
• Demographics
• Alcohol, Tobacco, other drugs
• School risk/protective factors
• Community risk/protective factors
• Peer-Individual-Family risk/ protective factors

49
Youth Health Behavior DataConsistent with
National YRBS YTSDevelopment led by DOH

• YRBS Questionnaire
• Demographics
• Alcohol, Tobacco, Other drugs
• Nutrition
• Physical Activity
• Safety Behaviors (Helmets, Seatbelts)
• Mental Health/Depression/Suicide
• Additional Tobacco Indicators

50
Getting the Best of Both
Form A WSSAHB-like
Form B YRBS-like
Survey Core
51
6th Grade Survey
One Form
Survey Core Mutually Agreed Items
52
Core Survey Items

• Demographics
• Alcohol, Tobacco and Other Drug Use Items
• Key Violence-Related Items
• School-Specific Asset Items
• Depression

53
Administration procedures

• Safe and confidential environment
• Standardized administration procedures (e.g.,
  coordinator training, teacher training, written
  instructions, teacher stays in room but at desk,
  single class period to avoid discussion, absent
  students do not make up)
• Students informed of importance of survey
• No student name or other identifying information
• Students place own answer sheet in envelope
• Students provided resource list

54
Data Cleaning Procedures

• Inconsistent answers
• Evidence of faking high level of substance use
• Dishonesty
• Wrong grade

55
What is a Confidence Interval?

• The reported value is probably a little different
  than the true value for all your students
• We are 95 confident that the true value is
  within the /- range, called the confidence
  interval

56
Interpreting CIs Graphically
57
Local Report Contents

• Introduction and Overview
• Highlights of Local Results
• Selected Results by Gender
• Responses to Items
• Risk and Protective Factors
• List of Core Items Item index

58
For your continued exploration

• http//www.rmccorp.com/HYS06/index.htm

http//www3.doh.wa.gov/hys/
http//www3.doh.wa.gov/hys/
59

• Students with severe persistent asthma symptoms
  had
• increased odds for academic risk

Source 2004 Healthy Youth Survey, state sample
only. Graph uses unadjusted estimates odds
ratios are for estimates adjusted for grade
(8-10-12) and maternal education (3-level).
60

• Students with diabetes had increased odds for
  academic risk

Source 2004 Healthy Youth Survey, state sample
only. Graph uses unadjusted estimates odds
ratios are for estimates adjusted for grade
(8-10-12) and maternal education (3-level).
61

• Students who smoked any cigarettes during the
  past month
• had increased odds for academic risk

Source 2004 Healthy Youth Survey, state sample
only. Graph uses unadjusted estimates odds
ratios are for estimates adjusted for grade
(8-10-12) and maternal education (3-level).
62
Healthy Youth Survey Website

• Some quick crosstabs examples you could run
• Kids who smoke are less likely to report having
  access to an ATOD counselor (61 among smokers v.
  71 among non-smokers, 10th grade state sample)
• Kids who report being bullied are also more
  likely to report depression (46 depression among
  bullied v. 29 among non-bullied, 10th grade)
• Kids who drank more than 2 soda pops per day were
  more likely to report getting Ds and Fs (7 vs.
  15 among heavy pop-drinkers, 10th grade)

63
Healthy Youth Survey 2006 HSLP related measures

• Physical Education
• Number of days of PE per week
• How long doing actual activity in PE class
• Counseling/Student Support Services
• Is there a school counselor to talk about ATOD
• Is there someone you could go to for help
• Food/Nutrition Services
• New junk food module in 2006 asks about soda pop
  and junk/fried food consumption at school, also
  breakfast
• Fruit vegetable 5-a-day consumption
• Soda pops per day (previous day)

64
HYS 06 HSLP-related measures

• Healthy Environment
• Feeling safe at school
• Enjoy being in school/hate being in school
• Carry weapon on school property
• Bullying
• Teachers care about me
• Using tobacco on school property
• Been drunk or high at school
• Tobacco rules are usually enforced
• Health Education
• Number of times received tobacco education
• Number of times practices ways to say no
• How good in school at educating about HIV

65
HYS 06 HSLP-Related Measures

• Family/Community Involvement
• Dinner with family at home
• Parent/guardian discussed tobacco at home
• How easy to get tobacco/alcohol if you wanted
  some
• Student Support Services
• Ever had an asthma action plan

66
Your Questions

• What if I cant find my school or district
  report?
• What if I just need some general help getting
  started with HYS data or the website?
• What if I want more analysis done with our data?
• What if I have specific questions about HYS data
  and our Health Department staff cant help me?

67
You made it!
68
What we talked about

• Present get your feedback on Evaluation Plan
  for HSLP 06-08 cohort
• Generate ideas for your own assessment/evaluation
  work
• Ideas for using the Healthy Youth Survey

69
Here to help!

• General Questions
• Lori, Martin, Pam
• HSLP Evaluation Detail
• Julia Dilley 360-402-7877
• julia.dilley_at_state.or.us
• Healthy Youth Survey
• Diane Pilkey 360-236-3526
• diane.pilkey_at_doh.wa.gov