PilotOnline Review of R21 Applications

1
Pilot-Online Review of R21 Applications

• Daniel F. McDonald, Ph.D.
• Chief, Musculoskeletal Oral Skin Sciences and
  Renal Urological Sciences IRGs
• Center for Scientific Review, NIH

2
I. Objectives

• To assess alternate review paradigm given
  increasing overall CSR review burden and
  ultimately limited human/financial resources
• To relieve review burden on standing skeletal
  biology study sections

3
(No Transcript)
4
II. Meeting Mechanics

• Choose application pool
• Administrative review (SRA)
• Recruit reviewers
• Make assignments
• Critique prep (reviewers)
• IAR submission Phase

• Triaging process
• Initiation of chat room
• Monitoring of discussions
• Establishment of final levels of enthusiasm
• Procurement of individual reviewer votes

5
Chatroom Setup
6
III. Post-meeting Activities

• Critique revision (reviewers via IAR)
• Score/Code entry/release
• Summary Statement generation/release
• Process assessment
• gt CSR personnel (SRA, IT support, senior staff)
• gt Reviewers
• gt Program staff

7
Summary of Review Load

• 52 R21, 4 R03 and 2 R15 applications
• 194 assignments for 46 reviewers
• 21 of the 58 applications went unscored

8
Number of Reviewers Participating

• 137 assignments to 37 scored applications
• Average of 3.3 reviewers assigned to each scored
  application

9
Summary of Comment Source

• Average of 3.7 members commented on a given
  scored application in electronic panels
• vs.
• For a regular study section, average number of
  members participating in a discussion per
  application 4.6
• N 75 study sections, with a mean of 19.7
  members per study section Unpublished
  observations made in DRG study sections in 1989

10
Summary of Comment Source (contd)

• Average of
• 5.9 comments per application from assigned
  reviewers
• 0.8 comments from unassigned reviewers almost
  always the Chair
• 6.8 total comments per application

11
Summary of Scoring Profile

• 7 of 37 scored applications were scored by all
  reviewers
• The other 30 applications were scored by all but
  1 or 2 reviewers
• Conflicts counted as scores to differentiate from
  those that didn't score at all

12
(No Transcript)
13
Data Analysis Suggests

• Number of comments tended to increase with range
  of initial scores.
• Once the initial score spread from the assigned
  reviewers gets above some figure between 0.5 and
  1.0, there is an increase in the number of
  comments that are made.
• The additional comments are all from the assigned
  reviewers, not the unassigned reviewers.

14
(No Transcript)
15
Data Analysis Suggests (contd)

• Final score not closely related to number of
  comments.
• Range of final scores from assigned reviewers was
  much narrower than their initial scores.
• Final score range from assigned reviewers
  narrower than from all reviewers.
• Initial range of scores from assigned reviewers
  broader than final range from all reviewers.

16
(No Transcript)
17
(No Transcript)
18
(No Transcript)
19
(No Transcript)
20
IV. Preliminary Post-meeting Reflections (SRA)

• ADVANTAGES
• Maximum flexibility in construction
• Recruiting tool
• Convenience for potential reviewers
• Substantial reviewer cost savings

• SHORTCOMINGS
• Unfamiliar approach to reviewers
• Added security vulnerabilities
• No real-time face-to-face interaction
• Restricted involvement of subgroup of panel
  members in any particular evaluation
• More time-consuming for all parties
  (SRA/Chair/panelists)
• Meeting size constraints