PT VI Rudolph presentation

1
Asynchronous Electronic Discussion
Joseph Rudolph and Ross Shonat
August 27, 2007
National Institutes of HealthU.S. Department of
Health and Human Services
2
Asynchronous Electronic Discussion(AED) What
is it?

• Secure Web Based Review method.
• Threaded Discussion Board.
• Separate server environment from IMPAC II and
  IAR, however data is mined from both IMPAC II and
  IAR.
• Both AED and IAR are utilized.

3
Intended Purpose of AED

• AED is NOT intended to replace the traditional
  large face to face meetings.
• AED is intended to provide an additional review
  platform to SRAs and Reviewers which would fill
  the niche between phone and face to face
  meetings.
• Attract senior and experienced reviewers who are
  not able to travel.
• AED is flexible enough to adapt to the
  requirements of the individual SRA.

4
Advantages of AED

• Greater flexibility for scheduling and running
  peer review meetings.
• Expand potential reviewer base, by eliminating
  travel and time constraints.
• Allows for more thoughtful and thorough
  discussion.
• Efficient mechanism for streamlining.
• Simplifies participation of program staff.
• Improved management of conflicts.
• Website is straightforward for reviewers and
  requires little training.

5
Initial Challenges for AED

• Format is new for both Reviewers, SRAs, and
  Program and will require a familiarization
  period.
• Requires active participation on the part of SRA,
  Chair and reviewers.

6
How Does it Work?

• Reviewers are recruited and grant materials sent
  out.
• Reviewers submit initial scores and critiques to
  IAR.
• AED web site link is given to SRA and meeting
  data is entered.
• All meeting data is pulled from IMPAC II and IAR.

7
How Does it Work (2)?

• The meeting occurs on the website and is
  comprised of 3 phases
• Streamline phase
• Discussion phase
• Final scoring phase.
• Following meeting reviewers revise critiques in
  IAR.
• The threaded discussion is not archived. Once
  the meeting is over, the website is deleted
  including backups.

8
What Does It look Like?
9
Review Page
10
Challenges of AED

• Developing a culture of usage and acceptance from
  SRAs, Reviewers and Program Officers.
• Training both for SRAs, Reviewers and Program.
• Technical challenges, integration and usability.

11
Growth of AED Number of Applications
12
Growth of AED Number of Meetings
13
Increased Efficiency of AED Meetings
14
Increased Usage Among SRAs
15
A Study Section Chair Talks About Asynchronous
Electronic Discussions
Finally, in spite of my initial skepticism
regarding this blog-like review mechanism, all
. . . reviewers produced comments, critiques,
posed questions in a fashion that I would judge
to be the BEST Ive seen in 30 years of chairing
various review groups.
Craig M. Jackson, Ph.D. President and
Principal Scientist Hemosaga Diagnostics Corp.
16
Future Directions

• Scaling
• Number of review panels (100 per round by 2008).
• Number of applications.
• Involvement of other ICs.
• NIGMS, NIMH, NIA, NIDA, NHLBI, NCI, NIDDK
• Improvements to Interface (Version 3.0 to be
  released 11/2007).
• Improvements to hardware.
• Synchronize reviewer authentication with NIH
  Commons.
• Wider adoption and acceptance.

17
Acknowledgements

• Project Management Team
• Rich McKay, Jim Blagaich, Richard Panniers, RV
  Srinivas, George Chacko, Ross Shonat.
• Developers
• Leo Wu, Sara Wu, Dipak Bhattacharyya, Harish
  Vajja, and Yulia Shifrin.
• AED Support
• Bella Kazantseva and Carlton Williams.
• Executive Sponsors
• Toni Scarpa and Cheryl Kitt