Decision Making Research and NIH Peer Review

1
Decision Making Research and NIH Peer Review

• Jane Steinberg, Ph.D.

National Institute of Mental Health National
Institutes of Health
2
What is Decision Research?

• The systematic study of how humans come to
  conclusions or a course of action.
• Testing the bounds of rational decision making
  with very interesting results
• Yielding 2 Nobel Prizes in economics

3
What is Decision Research?

• Interesting results showing that
• providing information or instructions is not
  enough in complex or high-risk decision making
• humans turn to certain heuristics or biases to
  simplify complex decisions that may lead to
  non-optimal decisions
• these biases can be minimized or used to
  strengthen decision making and behavior change

4
Heuristics

• Accessibility
• Framing
• Anchoring

5
Decision Research

• People are much happier using their heuristics or
  biases or intuitions than a structured decision
  making tool.
• Much

6
Helpful in Peer Review?
7
Decision Research in Peer Review

• RGA consultants explained that the more
  structure and simplification provided in a
  complex decision, the more accurate the decision.
• They recommended that reviewers address each
  criterion specifically in their reviews, rather
  than simply identifying strengths and weaknesses.
  
• This was implemented and is the current process
  in grant review.

8
Review Criteria

• Significance
• Innovation
• Approach
• Investigators
• Environment

9
Next Steps

• Task RPC to generate an SRA instruction-set for
  reviewers
• Begin with something small and doable (e.g., for
  an RFA, standing committees, or RPG reviews)
• Remember, instructions most important for first
  time groups
• Collect data to determine the incremental value
  of criteria-based scoring.

10
Design
Criteria-based Review
Holistic Review
11
Next Steps

• Task RPC to generate an SRA instruction-set for
  reviewers
• Begin with something small and doable (e.g., for
  an RFA, standing committees, or RPG reviews)
• Remember, instructions most important for first
  time groups
• Collect data to determine the incremental value
  of criteria-based scoring.