Grading and Awarding Improving reliability

1
Grading and Awarding Improving reliability?

• Colin Robinson
• Social and Strategic Market Research
• (SMSR)

2
What is a grade?

• A grade is an inadequate report of an inaccurate
  judgement by a biased and variable judge of the
  extent to which a student has attained an
  unidentified level of mastery of an unknown
  proportion of an indefinite amount of material.
• (adapted from Paul Dressel 1976)

3
An inadequate report

• Loss of information
• Combining across components
• Reducing performance judgments to grades
• Imprecise
• The smaller number of grades that are available
  for reporting, the more imprecise each grade
  becomes.

4
An inaccurate judgement

• Markers have to match the candidates responses
  to the mark scheme or performance criteria
• Both responses and mark schemes are subject to
  interpretation
• Some answers will be accepted that might have
  been rejected and some will be rejected that
  could have been accepted.
• These are not mistakes but legitimate
  differences in interpretation

5
A biased and variable judge

• Marker variability
• Markers judgments vary according to the time of
  day and personal circumstances
• One markers judgment will differ subtly from
  anothers

6
An unidentified level of mastery unknown
proportion of an indefinite amount of material.

• Guidance to candidates and teachers comprises
• The syllabus
• The grade descriptions
• Past papers/mark schemes
• Examiners reports
• None gives exact detail of what is required

7
The syllabus -1858 style

• From Oxford Regulations for 1858
• Mathematics
• Euclid, Books I. II.
• Arithmetic
• Algebra to Simple Equations inclusive
• This amount of knowledge will enable a Candidate
  to pass in this subject
• Questions will also be set in Euclid Books III.
  IV. VI., in Quadratic Equations, Progressions,
  and proportion, Plane Geometry not beyond the
  Solution of Triangles, the use of Logarithms,
  Mensuration and Practical Geometry.
• From Acland 1858

8
The syllabus 2008 style

• GCSE Mathematics 2007

9
Marking 1858

• Each of the examiners was instructed to take 100
  as the maximum value to be assigned to the paper,
  and to mark the work of the each candidate
  according to that scale.
• From Acland 1858

10
Marking 2005

• Code of practice April 2008

11
Awarding 1858

• The Examiners for the Preliminary Examination
  were requested not only to give marks, but also
  to reject or pass. But this part of the work was
  considered so important that it was thought
  better that the examiners should come to Oxford
  and confer with the Subdelegacy on the standard
  to be adopted.
• From Acland 1858

12
Table showing marks assigned to the Exercises of
Senior candidates in the Preliminary Examination.
(adapted from Acland 1858)
13
The current awarding process

• AQAGUIDETOSTANDARDSETTING.pdf

14
Determining the grade cut-off(AQA 2005)
15
Applying the indicators to two equally weighted
components
16
Using the Uniform Mark Scale 1
17
Using the Uniform Mark Scale 2
18
After the award

• The role of the Accountable Officer
• Borderline review
• Enquiries and Appeals

19
Conclusions

• Awarding and grading processes have a range of
  mechanisms to improve reliability
• Statistical and judgmental evidence are both
  essential
• Borderline review may reduce reliability
• Regrading on the basis of Enquiries are justified
  only in extreme cases
• Reducing grades is hardly ever appropriate

20
References

• Acland, T.D. 1858 Some account of the origin and
  objects of the new Oxford Examinations for the
  title of Associate in arts and certificates for
  the Year 1858. Ridgeway London 1858
• AQA 2005 A basic guide to standard setting. AQA
  Guildford 2005
• AQA 2006 Uniform Marks in GCE, VCE, GNVQ and GCSE
  Examinations. AQA Guildford 2006
• Dressel, Paul. (1976). Grades One more tilt at
  the windmill. Bulletin. Memphis Memphis State
  University, Center for the Study of Higher
  Education

21

• colinrobinson_at_tinyworld.co.uk
• www.smsr.co.uk