Evaluating the Children

1
Evaluating the Childrens Commissioner for
WalesA case study in research withyoung people
as partners

• Nigel Thomas and Mathew Taylor
• September 2007

2
Aims of the research

• Aims
• to evaluate effectiveness of Childrens
  Commissioner and impact on children and young
  people
• to work with Commissioners office to develop a
  process for continuing evaluation
• to plan and carry out research with active
  participation of children and young people
• Three-year project (May 2005 to May 2008)
• Based at Swansea University
• Funded by Childrens Commissioner

3
Timetable for the research

• Year 1
• Planning, design and exploratory work
• Year 2
• Main data collection
• Year 3
• Analysis and work to establish model for
  continuing evaluation

4
Children and young peoples involvement in
planning the research

• Began from when funding agreed
• In appointing the research officer
• In designing the research plan

5
The appointment group

• Young people (recruited through informal
  networks) involved in
• writing job description, person specification and
  advertisement
• shortlisting applicants
• interviews
• appointment decision (including membership of
  university appointment panel)
• Mandy Cook appointed

6
The steering group

• Fifteen children and young people recruited using
  variety of networks (youth forums and similar
  organisations)
• ages from 12 to 20 (at outset)
• from all across South Wales
• also working with group in North Wales
• Met four times in first year produced plan and
  design for the research

7
Method used in designing the research

• Started by looking at Commissioners job as set
  out in Acts of Parliament and in regulations made
  by National Assembly
• Divided job into different components, and asked
  what was best way to evaluate each component
• communicating with children and young people
• monitoring policy and services
• helping in individual cases

8
1st Steering Group meeting July 2005

• Question and answer session with staff from
  Commissioners office
• Exercise to get children and young peoples ideas
  on how to do the evaluation
• Worked in groups using summaries of
  Commissioners key tasks
• Indicators of success what would count as doing
  a good job?
• Data what information do we need collect or
  examine?
• Method how could we go about finding this
  information?

9
Participants comments on the exercise

• Objectives should be simplified and reduced
• Time needed to explain, especially for younger
  children
• Indicators, data and method need to be
  explained put definitions up on wall
• Time of day young people work best first thing
  in morning or late at night!(?)
• Children and young people may not know about the
  Commissioner information needed in schools

10
2nd Steering Group meeting August 2005

• Enthusiasm for project, so agreed to hold second
  meeting earlier than planned
• Agreed to take whole day more time to discuss
  things, provide lunch and trip to bowling alley

11
2nd Steering Group meeting

• Repeat of exercise with reworded summaries of
  objectives and reworded tasks
• What should be happening if this objective is
  being met?
• How do we find out whether it is happening?

12
2nd Steering Group meeting

• Introduction to Research Methods
• Basic introduction to research methods
• Discussion of possibilities for children and
  young people to be actively involved
• Most had some experience of one or two of the
  methods described
• All keen to be involved in some aspect of the
  research process particularly interviewing and
  data analysis

13
3rd Steering Group meeting October 2005

• Presented research plan worked up from everyones
  ideas and suggestions
• Accepted by the group as fair reflection of what
  they had said
• Asked group members to think about how they would
  like to be involved in doing the research
• Each member signed up to some element of the plan

14
3rd Steering Group meeting

• After lunch divided into 2 working groups based
  on expressions of interest, and looked in detail
  at two elements of the communication functions
  research
• Tracking what the Commissioner and his team are
  doing to communicate with children and young
  people
• Surveying children and young peoples knowledge
  and understanding of the Commissioner using a
  questionnaire

15
Following 3rd meeting

• By end of session we had outlines of
• tool for tracking the Commissioner and his team
• questionnaire to use with children and young
  people
• Fed this work back to adult advisors agreed to
  start piloting these two pieces of work in
  collaboration with the two working groups.
• Also set up MSN groups to provide easier way to
  keep group members in touch with each other
• Began work on pilot research

16
4th Steering Group Meeting April 2006

• Progress report and update
• Reviewed research plan
• Reported on use of tracking tool
• Reported on results of questionnaire pilot so far
• Worked on improvements to questionnaire design
• General discussion of issues in monitoring policy
  and services

17
Position at end of 1st year

• Questionnaire pilot (200) completed group
  members involved in distributing and collecting,
  offered chance to help with data input
• Some tracking of Commissioners team group
  members also involved with this
• Discussions begun with Commissioners office
  about best way to carry out evaluations of other
  functions monitoring policy and services and
  helping individual cases

18
Successes

• A group of young people who are genuinely
  interested in the project and who have made a
  real contribution to its design.
• The research is making progress probably on
  target, although it does not always feel that way

19
Problems

• Difficulties in maintaining contact outside
  formal meetings
• Practical issues in including young people in
  data collection timing, education commitments,
  travel limitations
• Grossly underfunded project 1 half-time
  researcher, very limited budget for running costs
• Current design much too ambitious will have to
  be prioritised
• Tension between commitment to involving children
  and young people and commitment to delivering
  other research outcomes

20
5th Steering Group Meeting July 2006

• Reflective discussion on the role of the steering
  group eg how it differed from a youth forum
• Designing the interviews
• compiled list of people to interview
• discussion on what type of questions to ask
• discussion on how to record interviews
• What do you think of it so far? Session with
  adult advisors to evaluate work of group

21
6th Steering Group Meeting October 2006

• Interviewing workshop
• discussion on interview techniques
• discussion about rights of interviewee
• writing an interviewing code of conduct
• Meeting with Peter Clarke
• For some of the group this was the first time
  they had met Peter this was a chance to chat
  with him and to ask questions about his work
• Interview workshop part 2
• looked at different types of recording equipment
  and tried them out
• role play interviews practiced asking
  questions, using recording equipment, making
  notes

22
Typical Steering Group Meeting

• News update
• Snack break
• Focal task
• Lunch break
• Activity(?)
• Focal task
• Finish

23
7th Steering Group Meeting February 2007

• Residential meeting over two days
• Day One
• Report of progress on survey
• Data Analysis part 1
• Exercise 1 working with ideas and opinions
• Exercise 2 reading pilot interview transcripts
• Update on website
• Session led by Participation Unit
• Swimming, supper, film etc

24
7th Steering Group Meeting

• Day Two
• Data analysis part 2
• Coding the interviews identifying key issues
• Planning the interviews
• Revising the questions
• Who to interview allocating interviews, looking
  at logistics
• Coding the interviews identifying key issues
• Evaluation
• of residential
• of project in general

25
8th Steering Group Meeting April 2007

• School survey 1302 questionnaires now completed
  looked at summary of quantitative findings, did
  some qualitative analysis in group
• Credits for involvement in research discussion
  of options agreed to go for simple letter of
  credit
• Reporting agreed to produce Interim Report of
  research (NT to write, other contributions
  invited)
• Update on stakeholder interviews
• Analysis of policy documents

26
9th Steering Group Meeting July 2007

• Work on analysing interview transcripts
• using key words
• using templates
• Work on planning interviews with big cheeses
• what questions to ask (small group exercise)
• who to interview (brainstorm)
• Repeat of What do you think of it so far?

27
Position at end of 2nd year

• Questionnaire survey (1300) completed data
  analysed, some involvement from group
• Stakeholder interviews (40) carried out, most by
  employed researcher, significant number by group
  members. Analysis beginning, 2nd wave interviews
  being planned.
• Evaluation of individual casework little
  progress
• NB Death of Peter Clarke (January 2007)

28
Reflections on young peoples involvement

• Consistency of group membership
• Serious engagement with task
• Limited knowledge of topic
• Readiness to learn new skills
• Work involved in planning sessions
• Lack of follow-through
• Importance of evaluation and reflection

29
Work outside the group meetings

• Much more of a feature of last few months
• Individuals conducting interviews, analysing
  data, taking initiative
• Growing confidence?
• Limited by time commitments
• Limited by travel restrictions etc

30
Questions

• How far have children and young people really
  influenced the research plan?
• They were very clearly active in the process
• But do we know that we would have arrived at a
  different plan without them?
• ...and does that matter?

31
Child protection issues

• Not a major feature of project
• Members recruited through youth councils/forums
• High age range
• Ethics protocol agreed early
• Staff had CRB checks
• Consent formalised for residential

32
And now.

• the truth!

33
The Reality Of Being Involved

• Confidence building
• Enabled to see things differently
• Increased understanding
• Commitment
• Recognition that work will impact others
• Ability to listen and relate to what others are
  saying

34
The Good Bits

• Meeting other young people
• Range of experiences
• Relationship building
• Level of involvement
• Increased knowledge
• Recognition for work
• Residential

35
The harder bits

• Time scales
• Geographical issues
• Ability to be involved outside meetings
• Understanding others experiences
• Support when out interviewing
• Keeping on track and meeting deadlines

36
Contact details

• Nigel Thomas
• University of Central Lancashire
• npthomas_at_uclan.ac.uk
• Mathew Taylor
• Voices from Care Cymru
• matthew_at_vfcc.org.uk
• Mandy Cook
• a.j.cook_at_swan.ac.uk
• Website www.cceval.co.uk