How to Prepare an Excellent Project Plan

1
How to Prepare an Excellent Project Plan
Surviving Peer Review

• Don Knowles and Mike Strauss
• Office of Scientific Quality Review

2
Why OSQR Review?
Its not our fault!

• 1998 Farm Bill
• ARS research peer-reviewed every 5 years
• Most review panelists external to ARS
• Satisfactory review before beginning research

3
Importance to ARS

• Congressional intent was to enhance research.
• OMB Analyses. ARS must report statistics for
  review quarterly. OMB is looking for average
  score improvement
• Provides for Agency Accountability especially to
  Congress
• Enhances ARS Image
• Improving peer review success
• could mean more support for ARS research

4
1. Stakeholder Workshop
Congressional Mandate
2. National Program developed.
Input
Input
8. Retrospective evaluation.
3. Program Priorities set (PDRAM).
You are here
Assess
Plan
PPO to NPS
4. Project Plan prepared.
7. Annual progress reviews
Implement
5. OSQR Review and Certification.
6. Research initiated
5
What is OSQR Review?
A Dialogue And an External Review Scientific
Review Prospective Review Peer
Review Competitive Review
6
Review Panels

• Panelists are your colleagues.
• They read your peer-reviewed papers.
• Panelists are active scientists.
• Most are academics (per the Farm Bill).
• Panelist often know your work.
• And are often familiar with your excellence.
• Panelists take their task very seriously.
• The devote many hours to each review.
• They dont want to give low scores!

7
How a Panel is Selected

• OSQR Receives suggestions/nominations from NPS,
  Areas, others.
• All potential chairs are screened for COIs.
• SQRO Interviews potential candidates.
• SQRO selects and invites Chairs.
• Chairs work with OSQR to develop a balanced,
  proposed list of panel members.
• SQRO reviews and approves final list of panelists.

8
What is an Excellent Plan?
Builds on past work (for continuing plans) Breaks
new ground Builds on new knowledge Displays
original ideas Risks breakthroughs
Presented in a logical, clear, easy-to-follow,
form. DONT tack varying bits from different
authors together DO write a consistent, smooth,
narrative from those pieces.
9
What do Reviewers Want?
Reviewers need to see a coherent story What are
the problems you are researching? Why are they
important? What do you hypothesize? (or why
a hypothesis is not appropriate) How will you
test your hypothesis? Why are you the best group
to do this? Dont let reviewers get another
perspective My reputation should be
enough This process isnt important Who are
these reviewers anyway! Whats the minimum I
can do to pass? Im not really taking this
process seriously
10
ARS Project Plans

• The foundation of research within ARS
• Treat this like a competitive grant
  requestreviewers do.
• Link to performance and impact of individual and
  team
• Congress and OMB use review scores to track
  overall success of the Agencyso your score makes
  a difference.
• Reflect project teams scientific expertise
• Reviewers often know youso they expect to see
  quality that matches your reputation.
• They cant score you on reputation.

11
The Goal of Peer Review

• Enhance research through Independent, expert
  examination of PROSPECTIVE plans for scientific
  and technical merit and relevance.

12
What is a Research Plan?
Tells the story of your research An integrated
and logical presentation A clearly presented
argument A Marketing brochure (and more!)

• A

Your plan should present your work in a clear,
logical fashion that will transmit enthusiasm for
the work and convince the reader that your team
is best suited to perform it.
13
Project Review Criteria

• Adequacy of Approach and Procedures
• Probability of Successfully Accomplishing the
  Projects Objectives
• Merit and Significance

14
Reviewers want to know

• What is the problem?
• Why is it important?
• Where are you going with it?
• How are you going to get there?
• And how will you know you have arrived?

Dont make them hunt for this! Poor organization
can wear their patienceand reduce your score.
15
Review Products

• Action Class Score
• Consensus review comments

16
Action Class Ratings

• No Revision
• Excellent project. No changes or additions are
  required.
• Minor Revision
• Approach sound. Some minor changes required
• Moderate Revision
• Some change to an approach required but project
  is generally feasible.

17
Where are the problems? (Science, Data, Writing)
The problem is writing and gaps in the narrative!
Little difference
18

• Now lets look at the nuts and bolts of a
  plan...
• Remember
• You are composing a narrative
• not completing a form!
• Your plan is not reviewed by bureaucrats.
• Reviewers are active scientists in your field.
• They often know you and your work.
• Reviewers devote, on average, 6 to 8 hours
  reviewing each plan BEFORE the panel meets.

19
Document Outline
Title and Investigators...page 1 Signature
Page...........page 2 Table of
Contents..page 3 Project summary (250
words)...page 4 Objectives....................
.page 5 Need for research (1-2 p) Scientific
Background (5-7 p) Approach Procedures (6-15
p) Prior Accomplishments (2 p) Literature
Cited Milestone Table (1-3 p) Past
Accomplishments of Project Team Members Issues of
Concern statements Appendices (letters plus other
material)
15 - 30 pages
These are not boxesthey are guides to your
narrative flow.
20
Your Plan is a Marketing Tooland the reviewers
are your customers

• By page 7 reviewers should know
• The subject of your research
• Why it is important
• What it will produce
• How you are going to get there
• The rest of the document will flesh out this
  but if the reviewers dont know it by page 7,
  they probably wont get it easily from the rest
  of the document.

21
Title and Investigators...page 1 Signature
Page...........page 2 Table of
Contents..page 3 Project summary (250
words)...page 4 Objectives....................
.page 5 Need for research (1-2 p) Scientific
Background (5-7 p) Approach Procedures (6-15
p) Prior Accomplishments (2 p) Literature
Cited Milestone Table (1-3 p) Past
Accomplishments of Project Team Members Issues of
Concern statements Appendices (letters plus other
material)
22
Project Plan Components
Project Summary 250 words The Appetizer!
Write this in active voice. State the essential
problem and why it is important. What have you
done to date (1-2 sentences)? How will you
address the issue? Why is this important? This
is where you capture the interest of the
reviewer. Make it compelling!
23
Project Plan Components
NEED FOR RESEARCH 1-2 pages Where are you going?
Express need scientifically AND in the context of
NP Action Plan. Be concise in statement of
research purpose. Discuss potential benefits and
anticipated products. Identify relevant customers
and stakeholders. Briefly note the principle
methods you will utilize (e.g., using microarray
technologies we will elucidate Build upon,
dont repeat, the overview!
24
Project Plan Components
Objectives 1-2 pages How does all this fit?
Why are all these pieces here? How do they
relate? Are there closely allied plans that bear
on this work? A figure can help! Should reflect
your plan and be a guide. - Objectives and
sub-objectives - Personnel - Outcomes -
Related projects Your plan can include 4 total
pages of figures. Use them!
25
Project Plan Components
Scientific Background 5-7 pages Why are you
going there?
Highlight knowledge gaps. Literature demonstrates
understanding and gaps. Not an exhaustive. Show
a rationale for the objectives. How will this
fill knowledge gaps? Limit to 1/3 of project
plan length Note similar projects within and
outside ARS and how your past work prepares for
or leads to this (provide details in the Prior
Accomplishments sectionbut say enough to
convince reviewers you know the area). Cite
preliminary data from your projects, if available
26
Project Plan Components
Prior Accomplishments 1-2 pages What have you
done before?
Highlighted briefly in the Background. Name prior
project terminated within two years Major
objectives and accomplishments Prior project
investigators Impact of prior work (science,
technology, users) Pertinent publications A table
or chart of past data can be very helpful.
27
Project Plan Components
Approaches Procedures 6-12 pages How are you
going to get there?
Set out your Experimental design. Formulate
REAL, testable, hypotheses! Describe approaches
and methods any why they are appropriate.
Discuss advantages and limitations (important
if methods are risky). Illustrate how
objectives can be achieved. Who will do
what, how, and when (including collaborators and
SCAs!) Describe nature and extent of
collaborations, including SCAs Letters in
Appendix need to confirm what you say! For
SCAs, a copy of the agreement is sufficient.
Include management, evaluation and
contingencies. What is your path to success?
How will you monitor it?
28
Project Plan Components
MILESTONES AND OUTCOMES
Summarizes the project
Dynamic over the project lifecycle
29
Some Recent Criticisms
Real Hypotheses Lack of connection between
pieces Uneven presentation between
objectives Lack of statistically sound
experimental design Context of planparticularly
when there is another, similar project. What will
project team (and collaborators)
contribute? Vagueness that prevents real
analysisif the information is confidential say
why you cant tell them but say enough to allow
some level of analysis. Risk without
justification Data accumulation without analysis
30
(No Transcript)
31
Comment Sampling No Revision
studies will fill rather substantial knowledge
gaps. well written and comprehensive well
written project plan well organized, cogently
prioritized and comprehensive research plan
32
Comment Sampling Minor Revision
studies will fill rather substantial knowledge
gaps. well written and comprehensive well
written project plan well organized, cogently
prioritized and comprehensive research
plan well written, well organized, and easy to
follow.
33
Comment Sampling Moderate Revision
one of the better written excellent discussion
of technology large research project plan but
poorly organized. concerns that some of the
objectives can be completed by this team. work
and ability described as "adequate" missing
current information hypotheses and plan well
conceived, approaches appropriate
34
Comment Sampling Major Revision
a heroic course thatwill not achieve the stated
objectives serious flaws in experimental
designs General Lack of focus. Plan is so broad
that it leaves out important detailsnot clear
data will be relevant or interpretable. lacks
cohesion and clear direction writing style,
quality, and organization significantly
detract Short on details. Not well planned. Need
to take process seriously.
35
Comment Sampling Not Feasible
approaches highly flawed, lack detail, many
experiments duplicative poorly writtendifficult
to discern what the authors want to do.
proposed research contradicts intention of the
proposal. exaggerations, incomplete reviews,
and misleading comments, as well as poor grammar
and organization. lack of knowledgeincomplete
understanding fundamental misunderstanding Work
does not address stated objectives.
36
Conclusions

• The reviewers need to see the logical thread
  through your work.
• Dont make readers search for what you are
  doing!
• Be clear, accurate, and correct.
• Dont assume reviewers know you and your work(a
  poor plan may not be saved even if they do!)
• A NONTECHNICAL reader should be able to
  understand the general goals of your research.
• A TECHNICAL reader should be confident you
  understand the problem and can accomplish the
  work.

37
Some hints to success

• Proofread Your Plan
• Ask a nontechnical person to read your plan
• Ask someone who hasnt seen it to read and
  proofread your plan
• Ask a highly critical colleague to read it
  thoroughly.
• Are collaborations documented appropriately?
• Check hypotheses
• Treat this the same care you would
• a competitive proposal. The reviewers will!

38
What if I have several SYs?

• Everyone has a contribution to make
• but dont follow this recipe for failure
• Ask for a portion from each
• Assemble those in order
• Consider the plan done(its just the start!)

Use the information to write a flowing, logical,
and consistent narrative. Settle scientific
differencesthen write! You may not like a
colleaguedont let the panel know that! You may
not like some of the workdont let the panel see
that! Alternatively, you may wish to write an
initial draft and edit in others additionskeep
the writing consistent throughout.
39
Hypotheses

• Most plans should have hypotheses.
• Hypotheses must be falsifiable and testable.
• Hypotheses are not restatements of objectives.
• A hypothesis may not be appropriate for work like
  breeding or germplasm characterizationbut
  explain that!
• Ask your Area statistician to review your
  hypotheses

40
What Happens After Review?
No, Minor or Moderate Revision Lead Scientist
responds to comments. Scientific Quality Review
Officer certifies compliance with
recommendations. Major Revision or Not Feasible
Lead Scientist revises and responds to comments.
Panel performs a second review assessing
response to their comments and assigns a new
Action Class Score. If still Major or Not
Feasible, project is returned for administrative
action. No further review. Projects are reviewed
no more than two times (There are no page limits
for revised plans)
41
What Happens After Review?(Researcher)
The Impact of Vertically Striped Voles (VSV) on
Wheat, Rye, and Egg Production
R. U. Kidding
1321-38000-123-00D
1/5/2006
Frontiers of Vole Biology and Relativity Theory
42
Can I disagree with the panel?

• This is a dialogue
• If you really disagreeput it away for a few
  days!
• Then
• Honestly consider panel opinions.
• Be polite but if you disagree say why
• DONT skip changes to plan
• DONT insult or impugn panelists
• DO provide justification for your alternative
  view
• Panels are NOT perfectthey are fellow scientists

43
Last Words

• Proofread
• Seek Review
• then
• proofread and seek more review
• And lastly
• Proofread and Seek Review
• However

44
Correct Grammar and Spelling are Important --
but not enough
Be sure your plan presents a clear, logical, path
to successat the outset and through the
document. Good grammar doesnt guarantee clear
writing.

• Twas brillig, and the slithy toves  Did gyre
  and gimble in the wabeAll mimsy were the
  borogoves,  And the mome raths outgrabe.
• Lewis Carroll

45
Peer Reviewer
46
Input
Assessment
Input
Planning
Review
Implementation