Electronic Improvements For Peer Review Present and Future

1
Electronic Improvements For Peer ReviewPresent
and Future
Brent StanfieldActing Director
Center for Scientific Review National Institutes
of Health Department of Health and Human Services
2
Recent Paperless Improvementsfor Peer Review

• Applications on CDs
• Internet Assisted Review (IAR)

3
Recent Paperless Improvements Applications on
CD
Yesterday
Paper everywhere!
It all ended upin here
4
Recent Paperless Improvements Applications on
CD
Today
Reviewers bring laptops
The laptops are generally reused after the meeting
5
Recent Paperless Improvements Applications on
CD

• All paper applications submitted to NIH are now
  scanned into digital format
• Scanned files are saved online in folders
  categorized by the study sections where the
  applications will be reviewed
• Review staff order appropriate number of CDs
  containing applications that will be reviewed
  in their study section
• Lightweight portable CDs sent to reviewers
  rather than bulky and wasteful paper copies

6
Recent Paperless Improvements Applications on
CD
Insert CD into CD-Drive
Action
7
Recent Paperless Improvements Applications on
CD
8
Recent Paperless Improvements Applications on
CD
X
X
X
X
- 01
01
X
X
9
Recent Paperless Improvements Applications on
CD

• Benefits of CDs
• CDs replace wasteful, bulky, hand collated paper
  copies
• Compared to paper copies, the direct costs of
  shipping applications to reviewers is reduced
• CDs are indexed and searchable and include value
  added features such as review guidelines for
  all application types (mechanisms and RFAs) on
  the CD
• CDs are much easier for reviewers to transport
  to a review meeting than stacks of bulky paper
  copies - Most (but not all) reviewers really
  like the CDs

10
Recent Paperless Improvements Applications on
CD

• CD Complications
• Until NIH requires electronic submission a
  certain number of paper applications will need
  to be rendered into digital format, adding to
  the cost and work process associated with
  creating CDs
• Some reviewers dont like CDs (for now CSR sends
  high quality paper copies of applications to
  assigned reviewers)
• Some reviewers dont want to bring laptop
  computers to review meetings
• Conflict CDs are burdensome for staff and
  expensive

11
Recent Paperless Improvements Applications on
CD

• What Does the Future Hold?
• Electronic receipt will improve the quality of
  the grant image on CDs and reduce the number
  of applications that need to be scanned into
  digital format to produce CDs
• All CSR study sections now distribute
  applications to
• reviewers on CD
• At some not too distant point we may be able to
  go entirely paperless

12
Enhancing The Review Process Internet Assisted
Review (IAR)

• NIH is currently using an online peer review tool
  that is a component of NIH eRA Commons and
  integrated with the computer-based information
  system of the extramural program (IMPAC II)
• Today, almost all CSR peer review meetings use
  IAR

13
Enhancing The Review Process Internet Assisted
Review (IAR)

• Using IAR
• Reviewers may
• Electronically submit critiques and preliminary
  scores prior to the meeting
• Read critiques submitted by others online
• Modify their critiques after the meeting
• View scanned grant image online
• Maintain their personal information (single
  point of ownership)
• Scientific Review Administrators (SRAs) may
• Do anything reviewers can do in the system
• Manage meetings and Reviewers IAR access via
  Control Center
• Designate applications as lower half
  (streamline)
• Download preliminary summary statements
  generated from submitted critiques

14
Enhancing The Review Process Internet Assisted
Review (IAR)
Reviewers List of Meetings
15
Enhancing The Review Process Internet Assisted
Review (IAR)
List of Applications Assigned to a Reviewer
Submit Phase
16
Enhancing The Review Process Internet Assisted
Review (IAR)
IAR Submit Screen
17
Enhancing The Review Process Internet Assisted
Review (IAR)
IAR List of Applications Read Phase
18
Enhancing The Review Process Internet Assisted
Review (IAR)
IAR List of Applications Edit Phase
19
Enhancing The Review Process Internet Assisted
Review (IAR)

• IAR Benefits
• Facilitates informed discussion among study
  section members
• - Enhances review meeting efficiency by focusing
  attention on those applications most in need
  of discussion- Enhances review meeting
  effectiveness by allowing reviewers to review
  or check facts used in other reviewers arguments
  before the start of the review meeting
• Expedites the process of generating summary
  statements after the study section meeting
• Provides infrastructure that ensures
  continuation of the peer review process,
  perhaps in combination with telephone/video
  conferences, if travel to review meetings became
  impractical

20
Enhancing The Review Process Internet Assisted
Review (IAR)

• IAR Complications
• Reviewers must register with eRA Commons
• - To avoid exceeding support capacity CSR rolled
  out use of eRA Commons IAR among
  its study sections over a 1-year period
• - Some reviewers have experienced difficulties
  registering and were frustrated
  by limited support capacity
• gt Problem has been largely resolved
• Reviewers must post their reviews several days
  before the review meeting requiring
  modification of some reviewers work habits

21
Enhancing The Review Process Internet Assisted
Review (IAR)

• What does the future hold?
• System could become a single point of access for
  all materials associated with a review meeting
  making any mail out of materials (including
  CDs) obsolete
• - Low resolution application images are already
  available on IAR
• - An important barrier to discontinuing CDs is
  download speeds
• Coming Soon!
• - SRAs will have the ability to directly post
  late arriving and supplemental
  materials in pdf format to a meeting folder

22
Paperless ImprovementsThe Next Wave
23
The Next Wave Electronic Application Receipt
On February 1, 2004 NIH will allow applicants to
electronically submit R01, R03, and R21
applications that have modular budgets and no
sub-contracts

• There will be on the order of 10,000 modular
  R01, R03, and R21 applications submitted in
  the October 2005 round that are eligible
  for electronic receipt

24
The Next Wave Electronic Application Receipt

• To date CSR has participated in several pilots
  that have involved tens of electronic
  applications.
• CSR expects to receive a few hundred electronic
  applications in in February 2005 and
  increasingly more and more in succeeding
  receipt dates in the future

25
The Next Wave Electronic Application Receipt

• Benefits of electronic receipt
• Space, equipment, and staffing needs to receive,
  store, and manipulate the tens of thousands of
  applications submitted to NIH each review
  round will be reduced over time
• Some information about electronically submitted
  applications will be captured automatically,
  reducing staffing requirements and time needed
  for processing
• Need (for capacity) to render hard copy
  applications into a digital format will be
  reduced
• Sets the stage to begin investigating
  integration of electronic tools into the
  referral process
• - Integration of Knowledge Management to help
  streamline and speed the referral
  process?

26
The Next Wave Electronic Application Receipt

• Electronic receipt challenges
• The exact number of applications that will be
  submitted electronically over the next few
  review rounds is unknown
• High resolution hard copies of all applications
  received electronically must be generated for
  distribution to reviewers to ensure that they
  are neither advantaged or disadvantaged in
  review
• - This presents a management challenge because
  the number of electronic
  applications that NIH will receive and need to
  print is somewhat unknown
• Referral of electronic applications requires
  modifications to the workplace
• - For example, referral officers require two
  monitors to refer electronic
  applications, one to view the application and the
  other to enter information into
  the IMPACII database interface while looking at
  the application

27
The Next Wave Electronic Application Receipt

• What does the future hold?
• Electronic receipt is expected to help
  streamline and speed CSRs receipt function
• Electronic receipt of high resolution files will
  facilitate the process of going paperless
• - In the future NIH could establish a system
  where reviewers would
  receive high resolution hard copies of
  applications only if they request the
  copies (perhaps through IAR)
• Electronic receipt may also better position CSR
  to take advantage of other tools that could
  help enhance the peer review process and make
  it more efficient

28
The Next Wave Knowledge Management (KM)

• CSR is eager to collaborate with eRA in
  developing KM tools that can be use to
• Help refer applications to appropriate NIH
  funding components and Integrated Review
  Groups/study sections
• Help identify reviewers for study section and
  help refer applications received by study
  sections to those reviewers whose expertise is
  the best match
• Help applicants identify study sections that
  would be most appropriate to review their
  applications

29
Paperless ImprovementsWhat Are The Goals?
30
Paperless ImprovementsWhat Are The Goals?

• Reduce Waste (paper and unnecessary work
  processes and facilities/business costs
  associated with paper)
• Improve effectiveness
• Improve efficiency
• Improve service for partners and customers
• Provide tools that assist professional staff,
  reviewers, and applicants
• Shorten the review cycle