Modelling the Predictive Validity of Selection Tests

1
Modelling the Predictive Validity of Selection
Tests

• John F Bell
• Research Division
• Cambridge Assessment

2
Predictive Validity

• Usually measured by a correlation coefficient
• Usually disappointingly low
• US Department of Labor, Employment Training and
  Administration
• Above 0.35 very beneficial
• .21 - .35 likely to be useful
• .11 - .20 depends on circumstances

3
Why so low?

• Effect of selection
• Uncorrected correlation decreases with greater
  selection
• Measurement errors
• Ordinal Categorical Variables
• A perfect relationship can have a correlation lt 1
• Variation in the wrong place
• Not interested in the fact that the test does not
  predict lower outcome scores very well
• Limit to what a selection test can predict

4
Effect of ordinal variable
r .87
5
Why so low?

• Effect of selection
• Uncorrected correlation decreases with greater
  selection
• Measurement errors
• Ordinal Categorical Variables
• A perfect relationship can have a correlation lt 1
• Variation in the wrong place
• Not interested in the fact that the test does not
  predict very high outcome scores well (applicants
  would be accepted any way)
• Limit to what a selection test can predict

6
Effect of Method of Selection (Simulated data)
7
What is observed?
8
Effect on Correlation
9
Worse Case Self- Selection
Correlations near zero
Can happen in real world (Linn and Dunbar, 1982)
10
Should we use the correlation?

• Tukey (1954) claimed to be an early member of
  Charles Winsors informal society for the
  suppression of correlation coefficients - whose
  guiding principle is that most correlation
  coefficients should never be calculated. He
  asked the question Does anyone know when the
  correlation coefficient is useful, as opposed to
  when it is used? If so, why not tell us?
• He recommend regression instead of correlation

11
Landys (1986) validation argument

• People who do well on test x will do well on
  activity y, or yf(x). In the null form, the
  hypothesis would assert that there is no
  relationship between test performance and
  activity performance.
• i.e. regression models not correlation

12
Asking the right question?

• If we use the test will we get more good
  students?
• Using Landys concept, the objective is to select
  the n highest values of y

13
Landys formulation and selection tests

• f(x) is not necessarily observed for a selection
  test
• h(x) the function for accepted applicants is
  observed
• g(x), the function for rejected applicants is not
  oberved
• yf(x)i.h(x)(1-i).g(x)
• I 1 if candidate accepted, 0 otherwise

14
Landys formulation andsimulated data
15
Extrapolation
16
Example MVAT/BMAT

• UK medical admissions tests for medical schools
  with very scientifically demanding courses
• Originally used in Cambridge (as MVAT).
• Now used more widely as the BMAT
• (www.bmat.org.uk)

17
Result of pilot year
18
Using the selection test only

• Take the best n candidates
• Where n is number of places
• Use a cut score of 43
• Assumes that rejected candidates have a
  probability of success of less than 0.11
  regardless of MVAT score

19
Relationship between first and test score
E.g. To accept a student with a MVAT total of
less than 34, it is necessary to believe that
the probabilities for all rejected candidates
less than 0.07
20
An unrealistic assumption

• Probabilities of success are the same for
  selected and de-selected applicants
• Same as saying that the current system is a
  lottery
• Obviously not true remember the histogram
• Predicted number of 1sts is sum of probabilities
  for the given group

21
Tau statistics
22
Results
23
More realistic assumption
24
Differing values of k
25
Results for varying k
26
Extensions

• Possible to predict future years
• Allows evaluation of k
• Use when test is used to supplement existing
  procedure

27
Predicting future years
Where g(x) is the probability of being accepted
on a given mark
28
Supplementing an existing procedure
29
Supplementing existing procedure

• Width of the decide on existing procedure
  interval decreases with decreasing confidence
• Test is still useful at lower levels of
  confidence
• Iterative procedure can be used to determine k

30
Koszonom hogy meghallgattak

• Three papers are available from me
• On correlation and selecting
• The tau-statistic replacing an existing procedure
• The tau-statistic supplementing an existing
  procedure
• Bell.j_at_cambridgeassessment.org.uk