Example: Evaluating School Effectiveness

1

• Example Evaluating School Effectiveness
• in Lancashire LEA using
• Value Added Measures
• Dr Sally Thomas
• Plan
• UK context
• What is value added
• Key issues in school effectiveness research
• Lancashire LEA Value Added Project
• Conclusions
• Further research

2

• UK CONTEXT
•  
• In early 1990s political attention and market
  driven reforms sought to increase the
  accountability of schools in the UK.
• Since 1992 schools' raw examination performance
  has been published annually by the Department for
  Education and Skills. These league tables have
  been widely criticised as unfair to schools with
  disadvantaged intakes.
• Other reforms in the last 10 years include a
  national inspection system (OFSTED), a national
  curriculum and national testing via standard
  assessment tasks and examinations.
• In this context school effectiveness and
  improvement research has been given a far greater
  emphasis than it enjoyed previously.
• For example, the UK government has now accepted
  the need for value added measures of student
  progress and the importance of school
  self-evaluation has been emphasised.

3
Sir Ron Dearings Report to Secretary of State
for Education (1993) Without a value added
dimension, the obvious basis for judgement is
that higher scores represent better practice
and lower scores worse. This could lead to
unwarranted complacency on the part of some
schools ... and, conversely, to despair on the
part of others ...  
4
Education White Paper 1996 (Page 53) The
Government's priority is to foster the internal
will and capacity of schools to generate their
own improvement ... staff and governors of every
school should feel that it is directly for them
to monitor the quality of the education they
provide and improve schools ...
5
The Autumn Package 1999 DfEE QCA
OFSTED Practice in the classroom is being
sharpened as teachers make considered
professional judgements based on enhanced, more
informative performance data available to
them. Jacqui Smith Parliamentary Under Secretary
of State for Education and Employment
6
(No Transcript)
7

• What Is Meant by Value Added?
• The rationale for the value added technique is
  that
• Raw results describe the grades that students
  have obtained
•  
• Raw results do not describe how well a school or
  college has performed
• Value added results describe how effective a
  school or college is in promoting students'
  achievement

8

• What Is Meant by Value Added?
• Value added is a measure of the relative progress
  made by pupils in a school over a particular
  period of time (usually from entry to the school
  until public examinations in the case of
  secondary schools, or over particular years or
  curriculum stages in primary schools) in
  comparison to pupils in others schools in the
  same sample.
•  
• It compares outcomes after adjusting for varying
  intake achievement and reflects the relative
  boost a school gives to a pupils previous level
  of attainment in comparison to similar pupils in
  other schools.
• The concept of value added is, therefore, both an
  indicator of a schools effectiveness and a tool
  for head teachers and their staff to use to
  analyse the extent to which they have effectively
  raised pupil achievement.
• However, it is not a magic wand. It has real
  limitations, which need to be well understood.

9

• Value Added Technique can be applied
• across any phase of education
• Primary
• Secondary
• Post 16
• The common aim of these studies is to look at
  factors outside the control of the school or
  college (such as students' prior attainment,
  gender, ethnicity and social class) that may have
  an impact on assessment or examination results,
  and where appropriate, to control for these
  factors in the analysis.

10
(No Transcript)
11

• Methodology Multi-Level Modelling
• Methodological advances have facilitated the
  development of value added measures of school
  effectiveness. These include Multi-level
  modelling techniques which are considered the
  most accurate and flexible tools for examining
  the hierarchical nature of pupil attainment data
  (Goldstein, 1995).
•  
• Essentially these methods are a generalised form
  of multiple regression.
• This approach allows the statistical analysis to
  compare different models and separate out the
  effect of the school experience on individual
  pupil outcomes (what pupils achieve) and the
  extent to which pupil intake characteristics
  (prior attainment, socio-economic background)
  affect pupil outcomes.
• Therefore, accurate baseline information about
  pupils' prior attainment is crucial to calculate
  the value added component.
• Value added measures can also be fine-tuned
  using additional background information about
  pupils, such as their gender, ethnicity and
  social class.
• Using this approach, the residual or value added
  score for each school in the sample can be
  calculated.

12

• School Effectiveness Research
• Some Key Issues have been addressed using
  multilevel modeling
• Is there evidence of internal variations in
  effectiveness (eg subject/departmental
  effectiveness and differential effectiveness for
  different groups of pupils, different curriculum
  stages)
• How stable are school effects over time? What are
  the trends in schools raw and value added
  performance?
• What are the long term effects of schools? Can
  the continuity of previous school effects on
  students later performance be examined? What is
  the impact of pupil mobility?
• What is the impact of classroom grouping on pupil
  performance?
• Do national or regional differences exist in
  school effectiveness?
• See also Scheerens 1992 Teddlie Reynolds
  2000

13

• The Lancashire Value Added Project
• The Lancashire value added project was set up in
  1992 and has involved over 130 secondary schools
  and over 140,000 pupils.
• The project aims to provide an innovative system
  of school evaluation and self-evaluation via the
  feedback of student performance, attitude and
  other data such as teacher and parent attitudes.
• Crucially, the evaluation process is not intended
  for external accountability purposes, rather a
  confidential tool for internal accountability and
  school improvement.
• Since the early beginnings, the project has
  expanded to incorporate a number of different
  types of value added evaluation feedback and the
  methodology employs state of the art
  statistical techniques such as multilevel
  modelling.

14

• The Data Employed for Lancashire Analysis
  (1993-2005)
• Outcome measures
• Total GCSE Score (excluding GNVQs)
• Total GCSE Score (including GNVQs)
• 5 Best GCSE score
• English
• Mathematics
• Science
• (Calculated using GCSE grades treated
  numerically, A 8, A 7, B 6, C 5,
• D 4, E 3, F 2, G 1, Other 0)
• Are used to create a total of 44 separate value
  added measures including for each outcome
  measure
• separate measures for each year and a three year
  rolling average
• separate measures for all pupils and pupils
  grouped according to their previous attainment in
  three bands
• From 2004 28 new value added scores for seven
  optional GCSE subjects

15

• The Data Employed for Lancashire Analysis
  (1993-2005)
•  
• Variables controlled for in value added analysis
• Verbal CAT sub-test
• Quantitative CAT sub-test
• Non-verbal CAT sub-test
• Age in months
• Male or Female
• Entitlement to free school meals
• Mobility - attending more than one secondary
  school
• Ethnicity using DfES categories
• Special Education Needs

16
(No Transcript)
17
(No Transcript)
18
(No Transcript)
19
(No Transcript)
20
(No Transcript)
21
(No Transcript)
22
  Innovation in School Evaluation The Lancashire
VAP The Case Study School I think it was
really fairly quickly that we began to realise
what a powerful tool that we had got. For first
of all, I suppose selfishly, I was their head and
for me as a management tool it was the one thing
I would never give up, because it has enabled me
to understand so many things better than I have
understood before, and to deal with them, in a
more dynamic and positive way (Head teacher)
23

• Limitations of Value Added Methodology
• Importance of statistical significance and
  confidence intervals when making comparisons
  between schools and departments.
• Data accuracy and measurement error.
• Validity of student outcome measures.
• Validity of background factors. (e.g. How well
  can we control for factors outside the control of
  the school, such as additional private tuition?)
• Retrospective nature of the data.
• Finally, are we measuring what we value or
  valuing what we measure? . it is important to
  emphasise that value added measures of student
  progress are not the only criteria that should be
  employed to judge school effectiveness, other
  more qualitative measures of school processes (eg
  quality of teaching) are also vital for this
  purpose.

24
A Critique of SESI A mechanistic methodology,
an instrumentalist view of educational processes
and the belief that educational outcomes can and
should be described independently of such
processes. Elliot (1996, pg. 200)
25

• Conclusions
• The importance of taking account of background
  factors and prior attainment using appropriate
  value added models in order to estimate the
  influence of the school is clear. Current
  evidence points to the importance of schools
  utilising value added information as part of a
  framework for self-evaluation, alongside other
  data or evidence (such as that provided in the
  DFES autumn package or via pupil, teacher and
  parent questionnaires).
•  
• However, the evidence also shows that
  effectiveness is best seen as a feature that is
  outcome and time specific. Therefore judgements
  about schools need to address at least five key
  questions
• Effective in promoting which outcomes?
• Effective over what period of time?
• Effective for whom?
• Effective for which curriculum stage?
• Effective in what educational policy or regional
  context?

26

• Further Research Is Needed to
• Examine the Nature and Dimensions of School
  Effectiveness and Improvement
•  
• Develop a broader range of outcomes (e.g.,
  measures of pupil affective and vocational
  outcomes in addition to academic outcomes).
• Examine what school characteristics or processes
  are likely to enhance a broader range of
  outcomes.
• Examine limitations of the data. For example, how
  well can we control for factors outside the
  control of the school such as private tutoring?
•  
• Analyse the national value added databases to
  further examine the patterns, conditions and
  constraints of school effectiveness and
  improvement across a range of regional, policy or
  socio-economic contexts.
• Explore the relationship between effectiveness of
  different levels of education system (e.g.,
  national, regional, local, school, department,
  classroom, individual).

27

• Further Research Is Needed to
• Examine the Implications for Policy and Practice
• Examine the long-term impact of accountability
  and school self-evaluation processes on the
  quality of teaching and learning?
• Examine what conditions appear to enhance, or
  alternatively form barriers, to school
  effectiveness?
• Examine the strategies, or levers, appear to
  improve school effectiveness? For example, what
  is the affect of providing feedback data to
  schools on their effectiveness?
• Examine the impact of value added approaches on
  organisational culture and learning, assessment
  policy and practices or equal opportunities.
• Explore the application of value added
  methodology to broader issues in educational
  research and evaluation.