Priority Scores and Percentiles

1
Priority Scores and Percentiles

• James Onken, Ph.D.
• Chief, Office of Program Analysis and Evaluation
• NIGMS

2
Priority and Percentile Scores

• Priority Score The average of individual
  reviewers scores on 1.0 - 5.0 scale (two
  significant digits), rounded to three significant
  digits, and multiplied by 100
• Percentile Score The rank of an applications
  priority score, relative to all priority scores
  assigned in the three most recent review cycles
• Example Average reviewer score 1.53
• Priority score 153
• Percentile score 20.7 (MGA)

3
(No Transcript)
4
Scoring Issues

• How reliable are they?
• Are they being expressed at an appropriate level
  of precision?
• Do the average priority score and percentile
  provide enough information?
• - To applicants
• - To NIH staff
• In one way or another, all three of these issues
  involve score variation and what this variation
  means.

5
Sources of Variation in Scores

• Individual Reviewer Scores
• True differences among applications
• True differences of opinion among reviewers
• Error
• Priority Scores
• True differences among applications
• Error
• Percentile Scores
• Variation in Priority Scores
• Variation in score base used to calculate
  percentiles

6
Reliability of Group Averages

• Reliability of individual reviewer score (rxx)
• True variance
• True variance Error variance
• Expected reliability of group average (rxx)
• rxx N rxx
• 1 (N-1) rxx

7
Expected Reliability of Priority ScoresAs
Function of Group Size
8
Misapprehension of the Advantages of Averaging

• Averaging estimates is an effective way to
  improve accuracy when combining expert judgments,
  integrating group members judgments, or using
  advice to modify personal judgments... people
  often hold incorrect beliefs about averaging,
  falsely concluding that the average of two
  judges estimates would be no more accurate than
  the average judge. The experiments confirmed that
  this misconception was common across a range of
  tasks...
• - Larrick and Soll, 2005
• Even subjects who were statistically
  sophisticated showed limited appreciation of the
  aggregation principle...
• - Kunda and Nisbett, 1986

9
Describing Variation in Reviewers Scores

• Standard Deviation
• Range

10
Variance Indicators
11
Variance of Proportions (Percentiles)

• Sampling errors in proportions are a function
  of
• proportion (largest around 0.5)
• denominator (smaller with larger N)

12
Standard Errors of Proportions
13
Round-to-Round Variation in Percentile
ScoresAssociated with Priority Scores within the
Same Study Section
14
Priority Scores Among Applications of Equivalent
Relative Rank
15
Priority Score VariationAmong Applications with
Equivalent Relative Rank

• Coefficient of Variation
• Standard Deviation
• Average Priority Score
• 0.10