Are some GCSEs harder than others

1
Are some GCSEs harder than others?

• ASCL Annual Conference
• March 2007

2
Five different methods for estimating grade
severity
From Nuttall et al, 1974, based on 1968 O-Level
results from one board
3
CEM Monitoring Projects (England)
Age
Year
18
13
17
12
16
11
15
10
14
9
13
8
12
7
11
www.cemcentre.org
6
10
5
9
4
8
3
7
2
6
1
5
R
PIPS Baseline
4
N
ASPECTS
3
4
Regression segments
5
Yellis rainbow wallchart
6
Severity of grading at A level (Fitz-Gibbon and
Vincent, 1994)
7
2003

• John Dunford
• It's a consequence of people perceiving that
  maths, physics, chemistry and modern foreign
  languages are harder subjects. Since most
  university courses do not require particular
  subjects, an A or a B in psychology is worth the
  same.
• Statistically its easy to show that psychology
  is an easier A-level than maths. It is incredibly
  worrying ... we are producing a nation of
  psychologists when the country desperately needs
  scientists and linguists.
• It's a hidden scandal.
• David Milliband
• Every A-level subject meets rigorous standards
  and several international panels have shown this
  to be true.
• Ken Boston
• There is no such thing as an 'easy' A-level
• Psychology A-level questions
• Critically consider whether multiple
  personality disorder (dissociative identity
  disorder) is an iatrogenic or spontaneous
  phenomenon.
• Compare and contrast two or more explanations
  of one anxiety disorder.

8
My analysis

• National dataset all GCSEs taken in 2004
  (625,000 students)
• Rasch model
• treats different subjects as indicators of a
  common trait (general academic ability)
• What level of general academic ability
  corresponds to each grade in each subject?
• 34 subjects included
• GNVQs failed to fit
• Creative subjects failed to fit (music, art,
  perf. stds)
• Grade U did not fit

9
Results
10
Gender bias in different subjects
11
Problems with statistical approaches to
comparability

• Other factors statistical differences may arise
  from many things other than difficulty (eg
  teaching, special interests)
• Multidimensionality different examinations
  measure different things so cant be equated
• Unrepresentativeness statistical samples are
  unrepresentative of those who take (or might
  take) a subject
• Subgroup differences relative difficulties
  vary by subgroup
• Inconsistency different statistical methods
  dont agree
• Forcing equality anomalous distributions and
  changeover difficulties would arise from making
  grades comparable and which definition of
  comparability would be privileged?

12
Conclusions (1)

• Are some subjects harder than others?
• Yes, if by harder we mean that the same grade
  corresponds to different levels of general
  academic ability
• But some subjects/examinations are not a good
  indication of general academic ability
• The differences are big
• Spanish, French, German are overall about a grade
  harder than drama, textiles, child development
• Grade F in Spanish, IT or history is almost the
  same as a D in textiles, PE or drama
• Rank order of difficulty depends a lot on which
  grade you are looking at
• At A and A, English is harder than history at
  F, it is 1.5 grades easier
• Correlation between F and A difficulties is 0.0

13
Conclusions (2)

• Gaps between grades are not equal
• Average A-A gap is twice C-D, D-E, E-F
• Difficulty varies for males and females
• Physics 0.5 grade easier for boys
• But subject differences outweigh gender
  differences
• Grade U is not one grade below G

14
Should we make them all comparable?

• Would delay awarding process
• Grade profiles for some subjects would be skewed
  (half Latin candidates would get A)
• Which definition of comparability would we use?
• Purely statistical methods lead to some anomalies
• Would destroy comparability over time
• In Australia they avoid all these problems by
  scaling