Decision Making Research and NIH Peer Review

1
Decision Making Research and NIH Peer Review

• Jane Steinberg, Ph.D.

National Institute of Mental Health National
Institutes of Health
2
What is Decision Research?

• The systematic study of how humans come to
  conclusions or a course of action.
• Tests the bounds of rational decision making with
  very interesting results.
• And 2 Nobel Prizes in economics

3
What is Decision Research?

• Interesting results showing that
• providing information or instructions is not
  enough in complex or high-risk decision making
• humans turn to certain heuristics or biases to
  simplify complex decisions that may lead to
  non-optimal decisions
• these biases can be minimized or used to
  strengthen decision making and behavior change

4
Personal Finances
5
Medical Decision Making
6
Security Issues
Military protocols
Screening procedures

7
Decision Research

• People are much happier using their heuristics or
  biases or intuitions than a structured decision
  making tool.

8
Why Not Peer Review?
9
Decision Research and Review

• NIH Peer Review can be conceptualized as a
  series of judgments, each one a complex decision
• Individual decision a reviewer prepares a
  critique and assigns a draft score
• Group decision the review meeting and
  consideration of reviewers comments
• Individual decision a reviewer assigns a score

10
Decision Research in Peer Review

• RGA consultants told us that reviewers are giving
  us a lot of information, but we werent making
  the most of it.
• NIH implemented one recommendation that was based
  on work in judgment and decision research.

11
Decision Research in Peer Review

• RGA consultants explained that the more
  structure and simplification provided in a
  complex decision, the more accurate the decision.
• They recommended that reviewers address each
  criterion specifically in their reviews, rather
  than simply identifying strengths and weaknesses.
  
• This was implemented and is the current process
  in grant review.

12

• What else would be helpful to try or to evaluate?
  What else does decision science tell us?
• RGA experts recommended going one step further in
  breaking down the complex individual decision
  scoring by criterion.
• Many possibilities

13
Review Criteria

• Significance
• Innovation
• Approach
• Investigators
• Environment

14
Heuristics

• Accessibility
• Framing
• Anchoring
• Preference for discussing shared information

15
Conclusions

• People and groups use heuristics to make easy
  work of difficult decisions.
• Most are successful, but some systematically
  provide less than optimal decisions.
• Thirty years worth of decision research has
  identified how to minimize these biases and
  strengthen decision making.
• The review process involves very complex
  decisions that may benefit from applying this
  knowledge base.