CSR Best Practices Kitt Presentation 12032007

1
CSR Best Practices Committee
Cheryl A. Kitt, Ph.D. Deputy Director
December 3, 2007
National Institutes of HealthU.S. Department of
Health and Human Services
2
CSRs Mission

• Ensure NIH grant applications receive fair,
  independent, expert, and timely reviews
• To ensure that the process continue to operate at
  a high level, the CSR Best Practices Committee
  was formed.

3
Charge to the CSR Best Practices Committee

• Goals
• Review existing policies and procedures
• Ensure alignment with FACA and other regulations
• Ensure consistency of practices and policies
  across study sections
• Promote the sharing of best practices across
  sections at CSR

4
The Committee Authored Five CSR Peer Review Best
Practices Guidelines

• Criteria for Selecting Reviewers and Assembling
  Rosters
• Conduct of Study Section Meetings Roles and
  Responsibilities
• Roles and Responsibilities for Summary Statement
  Production
• Telephone Reviewers at Face-to-Face Review
  Meetings
• Mail Reviews
• These documents are companion pieces to the SRA
  Handbook, a critical compendium of peer review
  policies and practices

5
1. Criteria for Selecting Reviewers and
Assembling Rosters

• Core Values
• The quality of scientific evaluation of a grant
  application is a reflection of the quality of the
  reviewers on the roster.
• SROs must ensure that the study section remains
  responsive to emerging areas of science and
  shifting scientific boundaries, and maintain a
  balanced membership with respect to geographic,
  gender, and minority representation.
• Balance is needed between recruiting generalists
  for their breadth and specialists for specific
  niche research areas.

6
Reviewer Selection

• General Considerations
• Independent and Established Investigator
• Respected by peers (e.g., department chair,
  journal editor, clinical status, plenary
  lectures, keynote speaker, etc.)
• Quality of research accomplishments Impact on a
  field
• Independent publications in peer-reviewed
  journals
• Honors (e.g., Awards, Memberships, etc.)
• Research Support (e.g., NIH, VA, NSF,
  foundations, industry, foreign, etc.)

7

• For Fellowship Committees
• All of the above
• Broad perspective
• Reviewers track record of mentoring e.g.,
  students, pre- and/or postdoctoral fellows
  institutional or unit (departmental, division
  lab) mentoring- years and resources
• For Small Business Committees
• If from academia, criteria listed above
• Rank in the small business concern (e.g., CSO,
  section head, directorship, group/project leader)
  
• SBIR/STTR funding
• Publication record in peer reviewed, high
  visibility journals or Patents
• Recognized as prominent practitioner in a
  professional field (e.g., surgeon, engineer)

8
Sources for Reviewer Recruitment

• Research databases (e.g. CRISP, PubMed,
  PsychInfo, etc.)
• Professional societies
• Scientific community consultation
• NIH Program Staff
• Conference attendance
• Study section chairs
• Study section members

9
Maintaining Reasonable Application to Reviewer
Workload

• Best Practice Statement
• For reviewers attending the meeting, the goal is
  7 assignments per reviewer on average, which
  includes reader assignments (see exceptions
  below).
• A workload of 7 gives the SRO flexibility to
  bring in a limited number of reviewers with light
  loads (e.g., field leaders, reviewers for a small
  group of specialized applications), balanced by
  the rest of the study section who are assigned a
  more substantial number of applications to
  review.
• Exceptions
• For small numbers of temporary members, member
  SEPs, re-review SEPs, multidisciplinary clinical
  trials, etc., that must be approved by the IRG
  Chief.

10
2. Conduct of Study Section Meetings

• Goal
• To provide consistent and clear guidance to SROs
  regarding the expectations of conduct before and
  at study section meetings, and to provide
  guidance for continued oversight of this
  critically important function.
• An effective peer review process requires an SRO
  to be active at the meeting, as well as in the
  weeks before the meeting.

11
Conduct of Study Section Meetings

• How
• Establish contact with reviewers and stay in
  touch with them during the weeks leading up to
  the meeting.
• Foster a team environment with ongoing
  interactions between SRO, IRG Chief, and Division
  Director to enhance the quality of peer review.
• Use Pre- and Post-Meeting Checklists

12
3. Summary Statement Production Best Practices

• High quality summary statements depend both on
  the reviewer and SRO.
• Reviewers effectively communicate in writing the
  critique of an application.
• SRO provides orientation and oversight before and
  during meetings to avoid discrepancies and
  misleading statements.
• SRO monitors critiques posted in IAR and
  provides an opportunity to detect and correct
  problems before the meeting, enhancing the review
  process and subsequent summary statement.
• The SRO uses written critiques and input from
  study section discussions to assemble summary
  statements that are concordant with the scores.

13
4. Telephone Reviewers at Face-to-face Review
Meetings

• Limit number of telephone reviewers initially to
  10 and not to exceed 20 for regular
  face-to-face meetings, barring unforeseen
  emergencies.
• Telephone reviewers should have prior review
  experience.
• SROs must focus on handling the meeting and
  taking notes. Thus, they should be minimally
  involved in communication logistics.
• Telephone reviewers who are not regular members
  preferably should not lead off the discussion.
• Telephone reviewers submit their scores in a
  secure manner, preferably via IAR or Meeting One.
  

14
5. MAIL REVIEWS

• In balancing expert coverage, SROs will on
  occasion have an application where a small part
  of the proposed project involves a special
  expertise or technique not covered by the
  existing members of the panel.
• If the size of the need for the expertise does
  not warrant addition of another member to the
  meeting then a mail review may be appropriate.

15
Parking Lot Issues

• Conflict of Interest
• Streamlining
• Appeals/Rebuttals
• Deferral for Re-Review
• Selection of Study Section Chairs
• Training of Reviewers and Chairs
• Role of Program Staff at Meetings

16
Committee Members

• Shirley, Mariela (NIAAA-Program Director)
• Bradley, Eileen (CSR-IRG Chief)
• Byrnes, Noni (CSR-IRG Chief)
• Cooper, Cathleen (CSR-SRO)
• Edwards, Emmeline (NINDS-Dep. Dir. DEA)
• Etcheberrigaray, Rene (CSR-IRG Chief)
• Fisher, Suzanne (CSR-Division Director)
• Gibson, Joy (CSR-IRG Chief)
• Khan, Mushtaq (CSR-IRG Chief)
• Krishnan, Krish (CSR-SRO)
• Panniers, Richard (CSR-IRG Chief)
• Pyper, Joanna (CSR-Dep. IRG Chief)
• Rigas, Marc (CSR-SRO)
• Schneider, Donald (CSR-Division Director)
• Kitt, Cheryl (CSR), Chair

17
kittc_at_csr.nih.gov 301-435-1112