The Pretest effect in National Curriculum Tests

1
The Pre-test effect in National Curriculum Tests

• Ezekiel Sweiry and Malcolm Hayes
• Edexcel

2
National Curriculum Testing

• Key stage 2
• Ages 7-11
• Subjects English, Maths Science
• Key stage 3
• Ages 11-14
• Subjects English, Maths Science

3
Test Development

• Pre-test 1
• Item trials
• Pre-test 2
• Whole test trials

4
What is the pre-test effect?

• The situation in which pupils score differently
  on their pre-test when compared with how they
  would be expected to perform had they taken the
  same test under live test conditions.
• Information about pupil performance in the
  pre-tests allows us to determine the level of
  difficulty of the tests and therefore to maintain
  standards over time.
• It is crucial that inferences made from data
  collected under pre-test conditions remains valid
  in the context of the live tests.

5
Aims

• To understand the causes of the pre-test effect
• How can we manage the risk to test equating that
  is posed by the pre-test effect?

6
Methodology

• Literature review focusing on factors that are
  known to affect test performance
• Consultation with test developers
• Analysis of pre-test data
• Model of the causes of the pre-test effect

7
Causes of the pre-test effect
8
Causes of the pre-test effect
9
Model of the pre-test effect
10
Motivation

• Of all the psychological concepts, motivation is
  the one most often cited in the literature as
  being responsible for differences in pupil
  performance between low and high stakes tests.
• Test-taking motivation has been defined as an
  individuals engagement and expenditure of
  energy toward the goal of attaining the highest
  possible score on the test. (Wise and DeMars,
  2003).

11
Expectancy-value model of motivation

• (1) Expectancy the students estimated
  likelihood of success on the task
• (2) Value how important the task is to the
  student
• (3) Affect the students affective and
  emotional reaction to the task
• (4) Required mental effort mental effort needed
  to complete the task.
• (Wolf et al. 1995)

12
Motivation and Performance

• Most studies of test-taking motivation have used
  a self-reporting scale that is completed by
  pupils immediately after they have taken the
  test.
• Two consistent findings are that motivation is
  correlated with test performance (e.g. Wise and
  DeMars 2003) and test consequences (e.g. Wolf et.
  al 1995).
• Motivation filtering

13
Preparation

• The different key stage tests vary in relation to
  the period in the year when pre-test 2 is
  administered.
• Sturman (2003) found that the vast majority of
  year 6 teachers prepared their pupils for the KS2
  Science tests.
• Meta-analyses (e.g. Becker, 1990) have shown that
  coaching boosts test performance.

14
Individual Differences

• Test Anxiety
• High levels of test anxiety have been shown to be
  negatively related to test performance (e.g.
  Hembree, 1988).
• Self-Efficacy
• The belief in one's capabilities to organise and
  execute the courses of action required to produce
  given attainments.
• Personality
• Conscientiousness, competitiveness
• Perception of Stakes
• How do pupils view the test stakes?

15
School Effects

• Test Environment
• Setting, noise level, weather
• Teacher Instruction
• Does the teacher follow prescribed administration
  instructions?
• Testing Schedule
• Are the tests taken on separate days?
• Curriculum coverage
• Has the whole curriculum been covered?
• Amount of preparation

16
Test Effects

• Feedback
• Will pupils be given their test scores?
• Test Length
• How many tests are there? How long is each test?
• Question Type
• Motivation has been found to be related to degree
  of effort required to answer particular types of
  question (e.g. Wolf et. al, 1995).
• Test stakes
• Effectiveness of preparation
• Length of preparation period

17
Key Stage 2 Maths

• Each pupil completes two 1-hour written test
  booklets and a mental maths test.
• The pre-tests are taken within a 1.5 week window
  that ends just prior to the live test.
• Within this window, the test schedule is the
  choice of the school.

18
DIF Analysis Mental maths test
19
Mental Maths and Test anxiety

• Reduced working memory is thought to lead to a
  pronounced increase in reaction time for more
  complex mental maths problems (e.g. Ashcraft,
  2002).
• The same effect was seen in the KS3 mental maths
  tests and the KS2 spelling test.

20
DIF Analysis Tests A and B
21
The Test B Drop

• There is a notable difference in the number of
  items showing DIF between test A and test B.
• In KS2 Maths pre-tests, pupils often take test A
  and B on the same day.
• A comparison of pre-test performance on test B
  between pupils taking both written tests on the
  same day and those taking the tests on separate
  days was undertaken.
• There appears to be a significant effect.

22
Is there a Preparation Effect?

• The pre-test can be taken up to 1.5 weeks before
  the live test is sat.
• A statistical analysis was carried out to
  investigate the relationship between pre-test
  score and the day within the 1.5 weeks when the
  pre-test was taken.
• There was no evidence of a preparation effect
  over this time scale.

23
Test Equating

• The expected outcome for any child taking one of
  our tests is independent of the test taken
• Maintenance of standards over time

24
(No Transcript)
25
Equivalent score on B
Cut-score on A
26
Equivalent score on Test B
Cut-score on Test A
27
Assumptions

• Tests are measuring the same thing
• Tests are valid and reliable
• Tests are built to the same specification
• Experimental conditions are constant

28
Experimental design

• Live test to live test
• Pre-test to live test
• Pre-test to pre-test

29
Live test to Live Test
30
Pre-test to Live Test
Live test B
Live test A
Pre-test B
31
Pre-test to Pre-Test
32
Pre-test effects

• Low motivation
• Lack of preparation
• Environmental factors
• Fatigue
• Different or less well?

33
(No Transcript)
34
(No Transcript)
35
(No Transcript)
36
(No Transcript)
37
(No Transcript)
38
(No Transcript)
39
(No Transcript)
40
(No Transcript)
41
(No Transcript)
42
(No Transcript)
43
(No Transcript)
44
(No Transcript)
45
Conclusion

• Pre-test effect can be modelled by a simple shift
  on the ability scale
• Calibrating tests using pre-test data is a valid
  approach
• Pre-test to pre-test equating translates to live
  test to live test equating.

46
References

• Aschcraft, M (2002). Maths anxiety personal,
  educational, and cognitive consequences. Current
  Directions in Psychological Science
• Becker, B (1990). Coaching for the Scholastic
  Aptitude Test further synthesis and appraisal.
  Review of Educational Research, 60, 373-417
• DeMars, C (1999). Does the relationship between
  motivation and performance differ with ability?
  Paper presented at the April 1999 NCME annual
  meeting, Montreal
• Hembree, R (1988). Correlates, causes, effects,
  and treatment of test anxiety. Review of
  Educational Research, 58, 7-77
• Sturman, L (2003). Teaching to the test science
  or intuition? Educational Research, 45 (3),
  261-273
• Wise, S and DeMars, C (2003). Examinee motivation
  in low-stakes assessment problems and potential
  solutions. Paper presented at the annual meeting
  of the American Association of Higher Educational
  Assessment Conference, Seattle, June, 2003
• Wolf, F, Smith, J, and Birnbaum, M (1995).
  Consequence of performance, test motivation, and
  mentally taxing items. Applied Measurement in
  Education, 8 (4), 341-351