Purpose of Review Committee

1

Objective Review Training 2009 Sponsored by
Grants Policy Staff/OMS Michelle G. Bulls,
Director

Presenter Ms. Tammy G. Bagley Senior Grants
Policy Analyst Grant Policy Staff, OMS/IHS (301)
443-7172
2
Objective Reviews

• What is an Objective Review?
• An Objective Review is a process that involves
  the thorough and consistent examination of
  applications based on an unbiased evaluation of
  scientific or technical merits or other relevant
  aspects of the proposal.
• Purpose of an Objective Review
• To provide a fair and credible forum for making
  sure that only the applicants selected for
  funding offer the greatest potential for
  furthering the programs goals, objectives and
  purpose.

3
Outcomes and Other Purposes for the Review Process

• Provide an evaluation on the merits of an
  application submitted based on the
  criteria/guidelines provided in the IHS Grants
  Funding Opportunity Announcement.
• Provide recommendations for possible funding of
  the grantee application to the IHS Program Staff.
  
• Provide technical assistance to potential IHS
  grantees on the strengths and weaknesses of
  their proposed project (without re- writing the
  proposal for them).

4
Policy AAGAM Chapter 2.04.104-C

• ORCs must be viewed as credible and fair.
• Any circumstance that might introduce a conflict
  of interest, or the appearance thereof of
  prejudices, biases or predispositions into the
  process must be avoided.
• ORCs are performed by experts in the field of
  endeavor for which support is requested.

5
ORC Members

• Must consist of the following persons
• Chairperson Presides over the ORC meeting, a
  non-voting member of the ORC,
  prepares the Executive Summary
  Statements.
• Program Official Serves as an advisor on
  programmatic
  related issues.
• NOTE POs are not allowed to provide any
  information that would create a bias in the minds
  of the reviewers. No comments on personal
  knowledge of the applicants can be shared.
• Grants Management Official
• 1. Provides orientation to the ORC,
• 2. Secures all score sheets, critiques, and
  minority reports at the end of each review,
  daily,
• 3. Ensures that the ORC is a fair and creditable
  process and,
• 4. Serves as an advisor on grants management
  related issues.

6
ORC Members cont

• The panel must consist of a minimum of at least 3
  people.
• Primary Reviewer
• Introduces the application to the ORC,
• Reads key aspects of the project, including the
  highlights of the project narrative,
• Serves as the lead reviewer for the application,
• Conducts a thorough critique of all strengths and
  weakness of the application and,
• Presents them first to the ORC.
• Most knowledgeable of all aspects of the
  proposal.
• .

7
ORC Members cont

• Secondary Reviewer
• Reviews the entire application and,
• Participates as a discussant based on their
  critiques,
• The open discussion should be geared around
  comments on any additional areas of the
  application that may not have been presented by
  the Primary Reviewer. Clearly the discussions
  must focus on strengths and weaknesses of the
  application.
• Tertiary Reviewer
• Offers a third opinion of the application and,
• Describes the key points of the proposal that
  were not mentioned by the primary and secondary
  reviewers.
• IMPORTANT NOTE Tertiary reviewers are not
  required.

8
The cognizant program office officials CANNOT
serve as reviewers on proposals funded out of
their respective offices.
Federal Employees

• A Federal employee may be from an office within
  the Indian Health Service considering the
  application for funding other than the cognizant
  program office.
• AAGAM 2.04.104C-6-2
• Example
• OPHS- Apples and Oranges Health Review
• An OPHS staff person cannot serve as a
  reviewer

9
Contents of the Reviewers PackageAAGAM Chapter
2.04.104C-7 (D)

• Instructions for Reviewers
• Conflict of Interest and Confidential Form
• Written Agreement of the Scope of Work
• Confidential Financial Disclosure Report
  (OGE-450) for Federal Reviewers
• IHS Program Announcement
• Reviewers Score Sheets
• Reviewers Critique Template
• Objective Review Minority Report Form
• Objective Review Roster
• Review Panel Application Assignments
• Review Meeting Order of Review

10
ApplicationsAAGAM Chapter 2.04.104C-7D-1

• All reviewers should receive a copy of each
  application. This means even the applications
  where the reviewer does not serve as the primary,
  secondary, or tertiary on the application.

11
Order of ORC Meeting

• Meeting is called to order by the Program
  Official.
• Brief description of the program that is up for
  review
• Expresses appreciation for those that will serve
  on the review panel
• Shares the number of applications that are up for
  review
• Introduces the Chairperson.

12
Order of the ORC Meeting

• Chairperson
• Calls for a brief introduction of each Review
  Member.
• Goes over the ORC protocol for conducting the
  meeting briefly, discusses breaks, timelines for
  reviewing applications and presides over the ORC
  process.
• Keeps track of which reviewer had a Conflict of
  Interest, for which grant.
• Turns over the meeting to the GMS for the
  Orientation.

13
Order of the ORC Meeting

• Grants Management Official conducts the
  Orientation
• Describes the roles of each participant within
  the review process,
• Discusses the Conflict of Interest Forms and
  collections all signed forms,
• Reminds ORC members to adhere to the program
  announcement criteria,
• Discusses scoring confidentiality -individual
  scores, collection of score sheets
• Discusses Critiques and Minority Reports,
• Reminds reviewers to dispose of all applications
  after the ORC has concluded,
• Discusses the confidentiality of the review
  process after the ORC ends.

14
Ready to Start?

• Chairperson
• Announces application to be reviewed,
• Ask the panel, if there are any conflicts of
  interest,
• Persons with conflicts must excuse themselves
  from the room,
• Requests primary, secondary, and tertiary (as
  appropriate) reviewer to summarize the project
  narrative and present strengths and weaknesses of
  the application,
• Second reviewer presents, strengths and
  weaknesses,
• Panel discussion, for clarity if needed,
• Call for a motion of approval or disapproval,
• Motion is seconded by a panel member,
• Votes are taken on the motion,
• Reminds reviewers to enter individual scores on
  their score sheet.

15
Scoring Applications

• Panel members should express their range of
  scoring based on the scoring chart provided to
  them by the Program Office.
• Individual scores are confidential and should not
  be shared with the panel at all.
• For example the Primary Reviewer might say
• After careful review of this application, I feel
  that it falls in the fundable range of 80 to 100.
  
• Reviewers must make sure that their numeric
  scores support the strengths and weakness
  outlined in their critiques.
• Note
• If the reviewer gives the application a low
  score, then his/her
• critique must reflect numerous weaknesses to
  support the
• low score given. (same for scoring the
  application on the high end)

16
Scoring Applications (cont)

• Reviewers cannot re-write the proposal. It must
  be scored as is.
• Reviewers can only make recommendations for
  improvement in their critiques, but this will not
  change the score. This is designed to highlight
  the weaknesses of the application.
• Reviewers should not add to or disaggregate the
  project in order to approve it. They should
  approve it and or make recommendations .
• Approval or Disapproval should be based on the
  CONTENTS OF THE APPLICATION ONLY in line with
  the criteria outlined in the Funding Opportunity
  Announcement.

17
Scoring Applications (cont)

• Applications
• In accordance with HHS policy (AAGAM Chapter
  2.04.104C -7G), all applications that are
  reviewed must be scored unless the program is
  using a streamlined method for voluminous
  applications.
• Disapproved applications
• Disapproved applications receive low scores
  that are outside of the fundable range or
  unsatisfactory rating.
• If a reviewer is out due to a Conflict of
  Interest, they should make note of that on their
  score sheets.
• Example
• COI if out due to a Conflict or Interest

18
Scoring Applications (cont)

• The Chairperson is responsible for making sure
  all reviewers place a score on their score
  sheets.
• If using individual score sheets, all sheets
  should be given to the chairperson before moving
  to the next application for review.
• If using a master score sheet, the Chairperson
  will need to collect each master sheet from each
  reviewer voting at the end of the day and provide
  them to the Grants Management Official present.

19
Supposition

• Information contained in the grantee application
  should be the ONLY basis for reviewers scores.

• Reviewers cannot score applications based on
  speculation, or assumption. Applications must
  be scored based on the
• Program Announcement Guidelines
• Facts stated in the actual application
• Application instructions

20
Critiques

• Each reviewer should pass their critiques to the
  Chairperson for collection before moving on to
  the next application for review.
• Note if a critique or minority report needs to
  be written or revised, the Chairperson should
  note who is responsible for this action and make
  sure the reports are completed by the end of the
  day and turned into him/her to be given to the
  Grants Management Official.

21
Minority Reports

Minority Reports are required when two or more
reviewers disagree with the majority.
Reviewers are required to complete a minority
report stating their comments and concerns.
Reports must be signed by all reviewers making
comments. Minority report comments will be
placed in the summary statement and shared with
the applicant.
22
Conclusion of the ORC

• Housekeeping
• Chairperson- Gives the Grants Mgmt Specialist
  all
• Score sheets
• Critiques
• Minority Reports
• All applications should be destroyed.
• Reviewers are reminded that all discussions of
  the ORC are confidential and should not be shared
  outside of the ORC meeting.

23
Executive Summary

• Once the ORC is concluded, the GMS will copy all
  critiques, and minority reports, and give them to
  the Chairperson.
• The program official must not write the Executive
  Summary however, they may designate the
  Chairperson to write the summary. The goal is to
  not have the program perform this function to
  maintain separation of duties and avoid
  conflicts.
• To avoid the appearance of a conflict, the
  program office should not be in possession of the
  reviewers scores or critiques without first
  obtaining them from the Grants Management Staff.

24
Reviewer Payments

• Program Official must contact
• Mr. Dale Burson
• Division of Acquisitions and Policy
• 206-615-2456
• Regarding payment for reviewers. Either by
• P-Card or,
• Cashiers Check
• Note the recommended threshold for IHS
  reviewers is 250.