Evaluating Health

1
Evaluating Health Wellbeing

• Hunter Institute of Mental Health

Presentation available on Hunter Institute of
Mental Health website as from next Monday
www.himh.org.au
2
Workshop Overview

• In this workshop we will
• Clarify and define the topic
• Look at the important steps in evaluation and
  briefly apply these to an example relevant to
  schools
• Look at the evaluation of MindMatters as an
  example and provide an overview of the evaluation

3
Clarifying the topic

• Practical ways of going about
• Evaluating
• programs, projects and/or activities which are
  aimed at enhancing
• health and wellbeing ..
• in schools

4
Clarifying health and wellbeing

• Could refer to a number of broad concepts
• such as
• Positive mental health
• Psychological wellbeing
• Resilience
• Social and emotional learning
• Prevention of mental health problems
• Social support
• Its not important to decide now, but clarity
  about the end goal is usually necessary for
  evaluation

5
Measuring health

• Usually measured by assessing health indicators
• risk factors (eg family history)
• risk behaviours (eg drug use)
• risk environments (eg ETS)
• protective factors (eg marital status)
• protective behaviours (eg clothing)
• capacities (strength, endurance)

6
Activity 1

• Write down a simple and straightforward idea for
  an activity that could be done in schools to
  enhance health and wellbeing

7
Example

• In a compulsory Year 9 Personal Development
  Course, teachers will work with students
  throughout Term 2 to organize a Mental Health
  expo for the whole school in which health
  promotion workers and service agencies will come
  into the school and provide activities and
  information related to mental health, mental
  illness and services available.

8
Clarifying Evaluation

• Evaluation means different things in education,
  social science, and in health science
• In education, the goal of evaluation has
  usually been to provide information to enable an
  overall judgement of a program (summative), to
  enable the program to be improved and delivered
  (formative), and to identify any unintended
  consequences

9
Clarifying Evaluation

• Social science has contributed such concepts as
  process and impact evaluation, and
  cost-benefit analysis
• Health science has emphasised the use of the RCT
  in determining the evaluation of treatment
  effectiveness.
• There are ongoing debates about the effectiveness
  of different evaluation designs to determine
  effectiveness of health promotion programs.

10
M Q Patton
Evaluation is any effort to improve human
effectiveness through systematic data-related
enquiry
11
M Q Patton
Evaluation is a process of making reasonable
estimations that particular activities have
contributed in concrete ways to observable
effects May not always create certainty about
causation but may provide guidance in an
uncertain and complex world.
12
M Q Patton
A successful evaluation is one that is useful,
practical, ethical and accurate emerges from
the special characteristics and conditions of a
particular situation a mixture of people,
politics, history, context, resources,
constraints, values, needs, interests and chance.

13
M Q Patton
There is no one best way to conduct an
evaluation.
14
Key steps

• Clear description of the program and how it is
  supposed to work (program logic)
• Clarifying the purpose of the evaluation
• Deciding what data will be collected, when and
  how and by whom?

15
Key steps

• Collecting the data
• Analysing the data
• Identifying implications and making
  recommendations
• Dissemination

16
Step 1 - Program logic

• What is your program trying to do?
• Outline how the program activity is related to a
  health and wellbeing outcome?
• Similar to outcome-based lesson planning.

17
Example Program Logic
Students participate in the health expo
Students have information about mental health
problems and the services that are available to
help and how to access them
Students access services when they feel they have
a mental health problem
Students mental health problems are dealt with
Students mental health improves
18
Activity

• Write down a draft program logic for your
  activity.
• What are the steps that link your activity to
  your health and wellbeing outcome?

19
Step 2 - purpose of evaluation

• What do we really want to know?
• Possible answers
• How coherent is the program logic? Does it make
  sense? (Coherence)
• Is the program needed, relevant to the target
  group? (Relevance)
• What was achieved? How well was the program
  delivered? Did the target group participate?
  (Achievements)
• What problems arose and how they addressed?
  (Responsiveness)
• Did the outcomes expected at each step actually
  eventuate? (Results)

20
Activity

• Look again at your idea for promoting health and
  wellbeing.
• What evaluation questions would be important to
  you?

21
Step 2 - Example

• Possible questions would be
• Did the program logic model make sense in
  practice? Does it need to be modified?
• Did the students see the need for the activity?
• What went well and what did not? What would you
  do differently again?
• Did the students participate?
• Did they learn anything?
• Did they ever use the information?
• Did they access any services?
• If so, did their problems get addressed?
• Was their health improved

22
Step 3 - Design

• What data do you need to collect? When? Who will
  collect it? How will it be collected?
• Considerations include relative importance of
  the questions, resources available and timeframe
• Methods will be determined by the questions and
  feasibility considerations

23
(No Transcript)
24
Reference for previous slide
25
Step 3 - Example

• Possible strategies
• Pre-post questionnaire of awareness of mental
  health problems and support services
• Focus groups with students (to determine program
  relevance)
• Monitoring of participation by services
• Monitoring participation by students
• Monitoring follow-up use of services

26
Next steps

• 4. Collecting the data
• Analysing the data
• Identifying implications and making
  recommendations
• Dissemination

27
MindMatters Evaluation

• Commenced in 2001
• Managed by a National reference Group comprising
  evaluation experts, representatives of health and
  education sectors

28
1. MindMatters Program Logic
MindMatters provides Professional Development
Kit
Schools attend MindMatters Professional
Development and/or receive the Kit
Schools implement MindMatters model (changes in
curriculum and/or other structural changes,
partnerships
Students learn knowledge and skills and have
more supportive environments
Students mental health improves
29
2. Purpose of the evaluation

• to evaluate the relevance, quality and any
  outcomes of the Professional Development Program
  being rolled out across Australia (PD evaluation)
• to describe in detail the use of the MindMatters
  program/resourcesat the school level including
  any outcomes (School level evaluation)

30
3. Evaluation methods

• PD evaluation
• Mail-out survey to participants in PD in each
  state/territory
• Key informant interviews (telephone,
  tape-recorded)
• School-level evaluation
• 15 school case-study
• Individual and group interviews during school
  visits
• Student questionnaires

31
Methods continued

• Healthy Kids Survey
• Protective factors
• Family Connection
• School Connection
• Community Connection
• Autonomy Connection
• Pro-social Peers
• Pro-social Group

32
Methods continued

• Healthy Kids Survey
• Resilience factors
• Communication Cooperation
• Self-esteem
• Empathy
• Effective Help-seeking
• Self-awareness
• Goals Aspirations

33
Data Collection analysis

• PD questionnaire
• lists provided by MindMatters
• Random sampling
• Follow-up
• Descriptive statistics and comparisons across
  states and systems
• Key informant Interviews
• Interviews tape recorded
• Individually summarised and coded
• Themed according to the structure of the
  interview

34
Data collection analysis

• Field visit interviews
• Audio-taped
• Transcribed
• Themed according to structured template of issues
• Further identification of similarities and
  differences across schools
• Identification of critical success factors and
  important barriers
• Identification of perceived impacts and outcomes
• Ongoing

35
Data collection and analysis

• Student Questionnaires
• Data collected by schools and sent to Hunter
  Institute of Mental Health
• Data entry
• Analysis using SPSS
• Comparisons between baseline and 2 year follow up
  for grade and sex groups.
• Ongoing

36
Reporting Dissemination

• Six-monthly progress reports to the Reference
  Group
• Reports available on MindMatters website
• http//cms.curriculum.edu.au/mindmatters/evaluatio
  n/evaluation.htm

37
Difficulties encountered

• MindMatters not a prescribed program
• -gtthe Program Logic is too general for
  practical evaluation of outcomes
• Each school adopts a different model of
  implementation therefore a different logic
  model applies -gt case study approach required to
  relate any impacts or outcomes (or lack of) to
  schools own model of implementation

38
Useful Resources

• Integrated health promotion resource kit
• Victorian Department of Human Services
• http//www.health.vic.gov.au/healthpromotion/resou
  rces_links/integrated.htm
• Evaluation in health promotion, Principles and
  perspectives
• WHO
• http//www.euro.who.int/InformationSources/Publica
  tions/Catalogue/20010911_43