Evaluation of the efficacy of the Tweenees lessons

1

• Evaluation of the efficacy of the Tweenees
  lessons
• Ian Barron
• Teaching Fellow
• Child Care and Protection

2
School-Based Abuse Prevention Programme
EvaluationTraditional Literature Review 1990 -
2007

• 22 Studies (3 secondary) evaluated school-based
  abuse prevention programmes over a 17 year
  period. The following outcomes were identified
• Children displayed high levels of prior
  knowledge
• On average small gains in knowledge and
  behavioural intention (telling) were achieved
• Cognitively able middle class girls made the most
  gains
• There were small increases in the number of
  disclosures (8 studies)
• At least 4 lessons, boosted annually were
  necessary to sustain knowledge gains
• There was no evidence that school-based abuse
  prevention programmes cause any harm

3
Meta-analyses (1990 2007)

• Outcomes
• 4 identified meta-analyses (most recent Davis and
  Gidycz, 2000) with effect sizes ranging from d
  0.47 to 1.07 moderate to high effect size)
• Moderator variables
• Duration (4 lessons and booster) and content of
  the programme (modelling and skill rehearsal)
• Age (older pupils achieved greater gains)
• SES (pupils from middle class families achieved
  greater gains than working class)

4
Gaps in the research to date

• Participants (survivors, upper primary and
  secondary aged pupils response to programmes)
• Lack of systematic recording (demographic,
  attrition rates, disclosures, effect sizes)
• Assessment measures (limited standardised
  measures, over reliance on questionnaires,
  childrens views not sought, childrens negative
  experiences not asked for, no cost-effectiveness
  measures, virtually no programme integrity
  measures)

5
Current study

• Evaluation of a local school-based abuse
  prevention programme the Tweenees (VIP package)
• Design Pre and post-test measures with waiting
  list control group, matched for numbers
• School and year group sampling purposeful
• Sampling for interviews - stratified sampling for
  age and gender and random sampling for SES
• Bias towards urban, working class, Caucasian
  pupils

6
School-Based Abuse Prevention Evaluation The
Local Study

• 4 lessons were delivered from the Tweenees
  programme over a 4 week period to a small group
  of survivors, as well as primary and secondary
  aged pupils. All schools had volunteered to take
  part of the evaluation.
• Survivors Group Intervention group 18u
  delivered lessons (10 pupils) and waiting list
  control group (10 pupils), aged 6 to 13.
• Primary Group (Primary 7) 18u delivered lessons
  (68 pupils), class teacher delivered lessons (20
  pupils) waiting list control group (59 pupils)
• Secondary Group S1 (118) and S2 (114) including
  waiting list control groups for each year.
  Equivalent numbers in each condition. Guidance
  teachers delivered the lessons.

7
School-Based Abuse Prevention Evaluation Local
Study Outcome Measures

• Knowledge behavioural intentions (CSKS-Q
  adapted, psychometric properties, used in 3
  previous studies)
• Perception of risk assessment (perception of harm
  and capacity to keep self safe exploratory
  signs at home, in school and in the community)
• Emotional impact (explicit vs implicit,
  positive/negative)
• Disclosure rates (within and beyond lessons -
  generalisation)
• Customer satisfaction from teachers and pupils
• Implementation integrity of class lessons
  (exploratory)
• Tools Questionnaires, interviews, disclosure
  forms and video observation

8
OutcomesSurvivors Group Knowledge and Skills
Questionnaire Pre and post-test Total Scores
(114 total score)

• Intervention group - significant difference on
  post-test total scores on knowledge/skills
  questionnaire found (t -4.035, p 0.003)
  compared to the control group (t 0.079, p
  .939),
• 6.7 points vs 0.1 points, SD 5.25
• Survivors who received the Tweenees lessons
  demonstrated on average, significant small gains
  in knowledge and behavioural intention to tell
  compared to survivors who had not received the
  lessons.
• Effect size d 1.00 (high)

9
OutcomesSurvivors Group Knowledge and Skills
Questionnaire - Specific Questions

• Significant difference found on question 10 (t
  -2.862, p lt .05) and 11 (t -4.000, p lt .01)
  post-test scores compared to control group (t
  1.861, p .096 t .557, p .591)
  respectively
• Question 10 What if your uncle was giving you a
  big tight hug and you dont like it. What would
  you do?
• Question 11 What if someone touches you in a
  way that makes you feel uncomfortable inside and
  you ask them to stop but they dont. What would
  you do?
• Survivors who received the Tweenees lessons were
  more likely on average to suggest more effective
  strategies to these questions compared to
  survivors who didnt receive the lessons, e.g.
  saying no, getting away and telling.

10
OutcomesSurvivors Group Knowledge and Skills
Questionnaire Pre and post-test Moderating
Factors

• When comparing the scores of male and female
  survivors, who had received the Tweenees lessons,
  a significant difference was found for Gender on
  question 21 F (1,8) 7.273, p lt .05 and
  question 24 F(1,8) 6.316, p lt .05. Given the
  control group was all female there was no control
  group comparison.
• Female survivors who received the Tweenees
  lessons were more likely on average to report
  higher overall sense of safety and feeling
  good about themselves scores compared to the
  male survivors who received the lessons.

11
OutcomesPrimary Evaluation (P7) Knowledge and
Skills Questionnaire - Pre and post-test Total
Scores

• 18u worker delivered lessons - significant
  difference found (t -5.433, p lt .01), 3.4
  points, SD 5.13, effect size d 0.60
• Class Teacher delivered lessons significant
  difference found (t -3.107, p lt .01), 4.1
  points, SD 5.90, effect size d 0.65
• Waiting list control no significant difference
  (t 0.692, p .492), -0.5 points, SD 5.45
• Primary 7 pupils who received the Tweenees
  lessons, whether delivered by 18u workers or
  their own class teacher, achieved significantly
  more knowledge and behavioural intention to tell
  gains on average than pupils who had not received
  the lessons.

12
OutcomesPrimary Evaluation (P7) Knowledge and
Skills Questionnaire - Pre and post-test
Specific Questions

• 18u worker delivered lessons - significant
  results found for
• Question 10 Intervention group (t -2.065, p lt
  .05) compared to control group (t 0.435, p
  0.665).
• Question 14 What if a friend of your
  parent/carer is hugging and kissing you and then
  asks you to keep it a secret. What do you do?
  Intervention group (t -2.031, p lt .05) compared
  to control group (t -0.195, p 0.845).
• Class Teacher delivered lessons - significant
  result found
• Question 13 What if a 17yr old is asking you to
  do something you know you shouldnt do. You say
  No! Im going to tell. Then he says if you do Im
  going to beat you up. What would you do?
  Intervention group (t -2.486, p lt .05) compared
  to control group (t -0.286, p 0.776).
• On average primary 7 pupils who received the
  Tweenees lessons were more able to suggest more
  effective solutions (telling and not keeping the
  secret) to these specific scenarios than pupils
  who did not receive the lessons.

13
OutcomesPrimary Evaluation (P7) Knowledge and
Skills Questionnaire - Pre and post-test
Moderating Factors

• On pre and post-test scores, no apparent
  significant differences were found between the
  pupils who received the Tweenees lessons and
  those who did not on the moderating factors of
  age, ethnicity, socio-economic status, and gender
• Age (within the P7 year group), gender,
  socio-economic status and ethnicity (too small a
  sample) did not appear to be factors which
  effected the childrens response to the Tweenees
  lessons.

14
OutcomesSecondary Evaluation Knowledge and
Skills Questionnaire - Pre and post-test Total
Scores

• S1 Intervention group - significant difference
  found (t - 2.248, p lt .05) 1.6 pts, compared to
  control group (t 1.547, p 0.127) 0.1pts
• S2 Intervention group - significant difference
  found (t - 2.703, p lt .01) 2.2pts, compared to
  control group (t - 1.594, p .117) 1.3pts
• S1 and S2 Intervention group - significant
  difference found (t - 3.523, p lt .01) 1.8pts,
  compared to control group (t - 0.393, p .695)
  0.2pts.
• Effect size d 0.33
• In conclusion secondary S1 and S2 pupils who
  received the Tweenees lessons made small average
  knowledge and behavioural intention gains to
  tell, compared to same age pupils who did not
  receive the Tweenees lessons.
• The S2 group received an abuse prevention lesson
  a month prior to the 4 lessons. It is unclear
  whether this made any impact on the S2 results.

15
Secondary Evaluation Knowledge and Skills
Questionnaire - Pre and post-test Specific
Questions

• No significant results were found for specific
  questions in either S1 or S2 for pupils who
  received the Tweenees lessons and those in the
  waiting list control

16
OutcomesSecondary Evaluation (S1) Knowledge and
Skills Questionnaire Pre and post-test
Moderating Factors

• A significant result was found for the moderating
  factor of Presenter F(2,57) 3.182, p lt .05
  from the post-test Total Scores.
• Significant results were found for three specific
  questions for Presenter
• Q 3 When you are on your own who is taking care
  of you?
• Q15 Do you have to do everything your baby
  sitter tell you to do?
• Q17 What if a 17yr old is asking you to do
  something you shouldnt do. You say No, Im going
  to tell my parent/carer. Then he says if you do
  Im going to beat you up. What do you do?).
• Findings suggested that the extent of knowledge
  and skill gains for S1 pupils may be partly
  dependant upon the experience of the teacher
  presenting the lessons.

17
OutcomesSecondary Evaluation (S2) Knowledge and
Skills Questionnaire - Pre and post-test
Moderating Factors

• No apparent significant results were found for
  the moderating factors of age, gender,
  socio-economic status, ethnicity and presenter
  for S2 pupils.
• Age within the S2 year group, gender,
  socio-economic status and ethnicity did not
  appear to be factors which impacted on pupils
  response to the Tweenees lessons.

18
OutcomesSecondary Evaluation (S1 S2)
Knowledge and Skills Questionnaire - Pre and
post-test Moderating Factors

• No apparent significant results were found in the
  whole S1/S2 group for the moderating factors of
• gender, socio-economic status, ethnicity,
  presenter and year group.
• The above factors did not appear to significantly
  effect pupils responses in S1 and S2 (when taken
  as a whole group) to the Tweenees lessons

19
Outcomes Survivor Group Disclosures over 4
lessons

• Intervention Group 12 Disclosures
• 2 Child Sexual Abuse (previously known)
• 8 Bullying
• 2 Physical Assault
• Waiting List Control 0
• All disclosures in the lessons were verified
  from video analysis of the sessions and from
  adults/peers within the situations in which the
  harm occurred.

20
OutcomesPrimary Group Disclosures during the 4
lessons

• 18u worker delivered lessons (68 pupils) 65
  Disclosures
• 15 - physical abuse 2 - child
  sexual abuse 1 sexual assault
• 14 bullying 2
  grooming 1 - erotica
• 11 - physical assault 1 -
  attempted stealing 1 - abduction
• 10 - domestic violence 1 -
  emotional abuse
• 5 - attempted abduction 1 rape
• Class Teacher delivered lessons (20 pupils) 4
  Disclosures
• 2 attempted assault
• 1 domestic violence
• 1 domestic violence/attempted assault
• Waiting List Control (59 pupils) 0 Disclosures
• Taking into account the numbers of pupils in each
  group, there appears to be a significant
  difference between the number of disclosures
  received by 18u workers compared to the P7 class
  teacher. A number of disclosures could refer to
  the same incident.

21
OutcomesSecondary Group S1 and S2 -
Disclosures during the 4 lessons

• S1 Intervention Group 5 Disclosures
• 1 domestic violence
• 1 bullying
• 1 physical assault
• 1 physical emotional abuse
• 1 abduction
• S2 Intervention Group 0
• S1 Waiting List Control 0
• S2 Waiting List Control 0
• S1 disclosures occurred in the 1st session of
  one class, the only secondary lesson led by an
  18u worker. All disclosures were verified as
  having been said via video analysis.

22
Outcomes GeneralisationPrimary and Secondary
Evaluations Telephone calls to survivors
helpline (Oct April) by school area

• School area
  2004/5 2005/6
• Primary 7 18u delivered schools
  0
  44
• Primary 7 Class Teacher delivered school
  0 0
• Primary 7 Waiting List Control schools
  0
  0
• Primary 7 Other area schools
  0
  28
• Secondary S1/2 Guidance Teacher delivered
  school 2 5
  
• Secondary S1/2 Waiting List Control school
  0 0
• Of the 77 calls
• 36 to check the number out 4
  reported female friend raped (all female)
• 16 bullying (14 female, 2 male) 2 sexual
  assault (all female)
• 7 domestic violence (all female) 1 rape
  (female)
• 5 physical abuse from parents 1 fear of
  stranger who hurts children (boy)
• (3 male, 2 female) 5
  calls no data
• Available information analysed from Survivors
  Service helpline records

23
OutcomesPrimary and Secondary Evaluations
Telephone calls to survivors helpline some
themes

• Records for 2004/5 showed that children did not
  use the helpline. Phone calls were from adults
  seeking support for children.
• Some primary 7 children who received the Tweenees
  lessons subsequently used the helpline number
  (applying skills in real life). Some simply
  checked out the helpline, others disclosed abuse.
  
• A significant number of children in Primary 7
  phoned the helpline from other school areas not
  part of the evaluation. Given the specificity of
  the age group, this may be evidence of children
  talking to each other across schools leading some
  children, not involved in the Tweenees lessons
  seeking help (generalising knowledge and skills
  from child to child?).
• Five pupils from secondary (S1) phoned the
  helpline 4 reported the same incident of rape.

24
OutcomesSurvivor, Primary and Secondary
Interviews (childrens learning and experiences)

• 20 interviews (6 survivors, 6 primary pupils and
  8 secondary pupils matched for gender)
• Nearly all children reported new learnings, e.g.
  what to do in a risky situation, discovering some
  people having a tough time and hearing different
  perspectives
• Little to no negative experiences or outcomes
  were reported despite being asked this directly
• Some felt they could talk about their feelings
  more openly because they heard others doing so.
• After experiencing the lessons most reported
  feeling more confident in keeping themselves safe
• Some judged themselves to be more assertive in
  telling because of their experience of adults
  listening, believing them and giving them more
  choices within the lessons.

25
OutcomesSurvivor, Primary and Secondary
Interviews (childrens learning and experiences)

• Nearly all children affirmed telling as the
  best strategy
• Most felt that following the lessons, they were
  more able to keep themselves safe less likely
  to be harmed
• On hearing others stories of harm most described
  their reaction as feeling sad and wanted to do
  something about it
• All thought none of the ideas within the
  lessons were difficult to understand

26
Presenters Teachers (6) and 18u Workers (3)
main themes from the interviews

• 18u and teachers held different goals, the former
  emphasised disclosure, the latter improving
  childrens behaviour towards others.
• 18u workers and teachers held different
  definitions of abuse. Teachers tended to operate
  what could be described as a narrow definition
  of abuse, in terms of pupils disclosing to the
  teacher and requesting that the disclosure be
  passed on. 18u workers operated a wide
  definition of abuse incorporating any stories of
  harm that were shared.
• Such differences in definition may have led to
  the significant differences in the recognition
  and reporting of disclosures.
• Teachers thought half the pupils benefited
  whereas 18u workers thought all the pupils
  benefited. This was underpinned by a difference
  in the definition of what benefit meant.
• Most teachers reported that they were confident
  to run the lessons next year.
• Because teachers reported to experiencing few
  disclosures during the lessons, none of the
  teachers felt the lessons had better prepared
  them to receive disclosures from children in the
  future.
• All thought parents should be more actively
  involved.

27
Presenters Teachers and 18u Workers main
themes from the interviews

• The only detrimental effects on pupils mentioned
  was that some boys were showing off during the
  lessons.
• The concept seen as easiest for children to learn
  was to say no, go and tell.
• The concept seen as most difficult to learn was
  that violence is preventable.
• There were differing views on what limited the
  effectiveness of the lessons. Teachers reported
  on childrens troublesome behaviour and 18u
  workers reported on children getting in trouble
  at times for their behaviour.
• There were shared views on making the Tweenees
  more effective for the future. These included
  more lessons, smaller groups and more preparation
  and training for teachers.
• Another suggestion was to use multi-agency teams
  to resource the delivery of the lessons. This was
  seen as an opportunity to facilitate closer
  partnership working between schools and child
  protection services.

28
Survivor, Primary and Secondary Groups Customer
satisfaction pupils views

• All survivors (100) and most primary aged pupils
  liked the Tweenees lessons a lot (74).
• A third of secondary pupils either liked or
  liked the lessons a lot, with the rest finding
  the lessons ok. A small number of S2 pupils
  (7) felt bored at times
• The favourite parts of the Tweenees lessons were
  the discussions (15) and the games (13),
  especially -Truth Dare Scare , Iffy Jiffy and
  Traffic lights.
• Pupils liked the writing worksheets the least
  (19)
• Despite being specifically asked, only 2 primary
  pupils felt frightened and 3 embarrassed during
  the lessons. No pupils reported experiencing
  upset.

29
Survivor, Primary and Secondary Groups Customer
satisfaction pupils views

• Almost half (40) the pupils did not talk to
  their parents about the Tweenees lessons. Those
  that did, spoke about it being fun and
  learning about safety.
• How to keep safe were the most frequently
  reported useful ideas (14) in the lessons.
• Most pupils found the Tweenees lessons
  understandable to very understandable (over
  90)
• Most pupils felt they had been given enough
  information to decide whether to tell if they
  were being harmed in some way (over 90).
• Most reported to feeling good, comfortable and
  relaxed during the lessons (80). A small number
  felt sad (2) on hearing stories of harm.

30
Presenters Teachers and 18u Workers - Customer
satisfaction

• Most presenters reported
• Most pupils looked forward to the lessons
• Truth Dare Scare was the favourite part of the
  lessons for pupils
• The lessons were understandable
• No parts of the lessons were upsetting or anxiety
  provoking for pupils
• Pupils did not become unduly afraid of strangers
  nor overly assertive
• Children of all ages should be taught such
  lessons as part of the school curriculum
• They would like to teach these lessons next year,
  although one teacher responded no in the
  evaluation form and yes within the later
  interview.

31
Presenters Teachers and 18u Workers - Customer
satisfaction

• There was a difference between 18u workers and
  teacher responses in that 18u workers also
  reported that children
• became more assertive through the telling of
  abuse
• increased in their frequency in approaches to
  them to tell their stories of harm
• applied the strategies they learned through
  telling within and out-with the lessons
• applied their strategies to other contexts, e.g.
  at home or in a small survivors group.

32
Implementation IntegrityVideo Analysis Summary

• From video analysis 18u worker led delivery of
  lessons included
• Open body language and orientation and movement
  around the class towards pupils
• High frequency of interactions from the support
  adult with individual pupils within the class
• Support adult giving individual pupils an
  opportunity to tell their story to an affirming
  adult before giving the pupil the choice to tell
  the class.
• Facilitation of peer to peer talk with a high
  frequency of spontaneous contributions (more akin
  to the playground setting)
• Space and time given for longer turn-taking
• Specific and explicit questions about harm
• All pupil responses explicitly received without
  judgement
• Validation of all viewpoints as right for that
  child
• Different pupil responses summarised to enable
  pupils to come to their own conclusions
• Disclosures received without judgement or
  blocking
• Frequent recognition and reporting of actual
  disclosures
• Space and time given for spontaneous class
  discipline to occur among pupils rather than
  active adult intervention

33
Implementation Integritythe Prevention Mind

• 18u worker delivery (from interview) was
  underpinned by
• A high level of motivation and unity of purpose
  for the safety and protection of children
• An explicit goal of disclosures as an outcome
• An expectation that disclosures would occur
• The motivation and lack of fear to notice and
  record disclosures
• A thorough understanding of the materials and how
  they work, i.e. the ownership of an active
  process to facilitate disclosure
• An absence of adult myths about abuse
• A high level of knowledge about child development
  and child protection, e.g. definition, signs and
  symptoms and what they mean
• A high knowledge and skill level in child-centred
  communication

34
Measure of Cost-effectiveness

• In terms of additional costs the cost of
  delivering the Tweenees lessons in the
  intervention group was worked out per pupil, per
  school and per disclosure. This worked out as
  follows
• Per pupil 7.73 (There were 205 pupils in the
  primary and secondary intervention group. This
  figure was divided into the additional total
  costs to get the cost per pupil for the
  intervention group).
• Per disclosure 21.71 (The figure selected for
  disclosures was the 73 disclosures made within
  the four lessons in the primary and secondary
  intervention groups. The additional total cost
  was divided by this figure to get a cost per
  disclosure).
• Per school 396.25 (There were four schools in
  total. This figure was divided into the
  additional total costs to get the cost per
  school).

35
Implications and Recommendations

• The Tweenees lessons appear to be one of the most
  effective abuse prevention programmes to date
  when delivered with skill (no other programme has
  evidenced such levels of disclosure)
• Teachers effectively delivered the Tweenees
  lessons to a primary prevention level
  (increased knowledge and behavioural intention)
• 18u workers effectively delivered the lessons to
  both a primary and secondary prevention level
  (disclosure of abuse)
• Teachers will need focused training in the
  Prevention Mind in order to deliver the
  Tweenees materials to a secondary prevention
  level.

36
Implications and Recommendations

• A video should be developed demonstrating the
  effective delivery of the Tweenees lessons,
  incorporating how to recognise and respond
  appropriately to disclosure.
• Abuse prevention lessons should be delivered
  annually.
• Methods for active parental involvement should be
  built into delivery of the lessons.
• A multi-agency model of lesson delivery should be
  explored.
• Longer running groups are required for survivors
  of abuse.

37
Conclusions

• The Tweenees programme matches other abuse
  prevention programmes in achieving on average
  small knowledge and behavioural intention gains.
• The Tweenees lessons go substantially beyond
  these other programmes by further evidencing the
  transfer of these skills into daily life, i.e.
  childrens use of telling behaviour.
• Such developments appear to be achieved by a
  highly knowledgeable and experienced prevention
  mindset coupled with a child-centred interactive
  class process.
• Teachers need focused training and support to
  develop the knowledge, mindset and child-centred
  communication skills that facilitate such a class
  context and process.
• The most effective and pragmatic way forward is
  to draw on the collaborative and joint-expertise
  of teachers and child protection workers
  (survivor services and statutory professionals)
  in the delivery of the Tweenees lessons.