Strength Through Science

1
Strength Through Science
Program Assessment Rating Tool Long Term
Performance Measures And BERAC Christine
Chalk Office of Budget and Planning Office of
Science Department of Energy July 10, 2006
2
Federal programs should receive taxpayer dollars
only when they prove they achieve results.

• OMB created the Program Assessment Rating Tool
  for FY04 Budget to assess and improve program
  performance and to hold agencies accountable
  for accomplishing results.
• Seven separate PART tools for each of the seven
  types of government Programs SC PART is for RD
  programs with select questions added from Capital
  Assets Grants.
• Four PART sections for all programs Program
  Structure and Organization, Program Planning,
  Management, Results (half of the points are in
  Results).
• Without meaningful annual and long-term
  measures in place programs are automatically
  rated Results not Demonstrated. Impossible to
  score above 50 without measures in place.
• For RD programs PART assessments include the key
  elements of the RD Investment Criteria.

3
Critical Elements of PART

• Independently Validated Management Processes
• Excellence - Peer Review (project, portfolio
  processes (COVs))
• Relevance Planning Community and Program
  that is Transparent Realistic and Tied to
  Budgets
• Performance
• Long-term and Annual Performance Measures that
  are meaningful (to Programs, Community, OMB,
  Congress) quantitative and trendable (when
  possible) tied to budgets and plans included in
  performance ratings of senior managers.
• Project and Facility Management

4
Office of Science and BER Experience with PART

• For FY03, programs given red, yellow, or green
  light score for program performance based on OMB
  analyst assessment.
• For FY04, PART was developed to improve
  consistency in ratings and to help agencies
  improve.
• For FY04, all Office of Science Programs were
  rated Results not Demonstrated due to lack of
  meaningful performance measures.
• For FY05, three Office of Science Programs were
  rated Effective (including BER) and three were
  rated Moderately Effective.

5
Significant PART Improvements

• For the FY05 PART, Science developed new
  long-term and annual performance measures.
• These measures are the only measures in the
  Office of Science.
• Developed a website to help explain the measures
  and report progress. (www.sc.doe.gov/measures)
• Most annual measures are quantitative and
  trendable. They are included in budget, tracked
  Quarterly by the department and audited annually
  by independent accounting firm.
• Progress toward the qualitative long-term
  measures is evaluated, by independent scientific
  advisory committees, every 3 years.
• Developed roadmaps for the measure.
• Developed definitions for excellent, good, fair
  poor progress.
• Committees of Visitors were established to
  provide independent validation of peer review and
  other management processes.

6
A Roadmap for Biological and Environmental
Research Objectives and Performance Targets
2006
2008
2010
2012
2014
2016
Characterize the multi protein complexes (or the
lack thereof) involving a scientifically
significant fraction of a microbes proteins.
Develop computational models to direct the use
and design of microbial communities to clean up
waste, sequester carbon, or produce hydrogen.
(2015)
Mathematical model for microbial community that
detoxifies uranium (2007)
New strategies for CO2 capture (2012)
AG
Enhanced biobased sources of fuel and electricity
(2012)
AG
Life Sciences
Photosynthetic microbe for continuous hydrogen
production (2008)
Discussion DRAFT
New capabilities for using microbial communities
to solve complex energy challenges (2010)
Artificial chromosome (2006)
New knowledge base for cost-effective cleanup
(2012)
EM
Deliver new measurements of clouds where
observations are missing (2006)
Measure ecosystem responses to climate change
(2008)
Develop/validate new models predicting effect of
aerosols on climate forcing (2010)
Deliver improved climate data models for policy
makers to determine safe levels of greenhouse
gases (2015). Reduce differences between
observed temperature model simulations at
subcontinental scales using several decades of
recent data (2013).
Climate Change Research
NOAA
Provide a climate model that links the Earth
climate system with Earths biological systems
(2010)
Include realistic cloud simulations in a climate
model (2007)
NOAA
NOAA
Alternative cesium separations process for High
Level Waste (2006)
Develop science-based solutions for cleanup and
long-term monitoring of DOE contaminated sites
(2015). A significant fraction of DOEs long-term
stewardship sites will employ advanced
biology-based clean up solutions and
science-based monitors (2013).
New technologies for in situ characterization
(2008)
EM
EM
Environmental Remediation
Technical basis for leaving in place
radionuclides beneath tank farm at the Hanford
Site (2007)
Validate new long-term monitoring tools at
multiple field sites that are inexpensive and
fast (2010)
EM
EM
Suite of field characterization techniques for
long-term monitoring of closed sites (2012)
Bioremediation of metals and rads validated in
the field (2008)
EM
EM
Microfabrication of 1000-electrode prototype
artificial retina (2006)
60-electrode retinal devices implanted in humans
(2005)
In vitro testing of 1000-electrode artificial
retina devices in dogs (2007)
Department-sponsored research into viewing the
makeup of genes in living cells, tissues, and
organisms is used by clinicians as a new,
sensitive tool for diagnosing disease and for
monitoring the efficacy of disease therapies that
target the products of specific genes. (2013)
Nuclear medicine centers graduate first class
of new radio tracer chemists to support U.S.
nuclear medicine (2010)
NIH
NIH
Medical Sciences
NIH
Preclinical tests of radiolabeled probes for
imaging defective genes (2006)
Protein Production and Tags Facility
construction begins (2006)
Future Facilities (Cross cut and support multiple
objectives and targets)
Protein Facility operation begins (2011)
Analysis and Modeling of Cellular Systems
Facility construction begins (2006)
Cellular Systems Facility operation begins
(2010)
Whole Proteome Analysis Facility construction
begins (2006)
Proteome Facility operation begins (2010)
Characterization and Imaging of Molecular
Machines Facility construction begins (2006)
Characterization and Imaging Facility operation
begins (2010)
Key Intermediate Objective from DOE Strategic
Plan Long Term Success Measure from PART
with EM and between two cleanup timeline
elements, as noted.
Interdependencies (Descriptions)
Broadly with ASCR on computational developments,
both hardware and software, affecting all facets
of basic research and advanced instrumentation.
EM
NIH
with National Institutes of Health
Future facilities directly support performance
targets in the life sciences, and through this,
subsequent targets in climate change and
environmental remediation.
This timeline is for planning purposes only and
does not constitute financial or contractual
commitments by the Federal Government.
with Agriculture and EPA.
AG
with Commerce and other contributing agencies
NOAA
7
Improvements Were Noted

• Performance Measures.--The Committee commends
  the Office of Science for its efforts to develop
  quantifiable performance measures for its
  research activities. Some of the measures (e.g.,
  inverse picobarns) are less comprehensible to
  Congress than others, but the overall approach to
  quantitative performance measurement is
  worthwhile.
• House Report 108-554
• Energy and Water Development Appropriations Bill
  2005

8
BERAC and PART

• BERAC reviewed the long-term and annual PART
  measures developed for FY05 PART.
• Found goals generally meaningful, ambitious but
  achievable, and measurable with a few suggested
  changes that have been incorporated.
• First review of progress toward the long-term
  measures is current charge.
• Assess progress - four point scale and roadmap
  milestones
• Comment on whether goals are still ambitious but
  achievable, meaningful, and measurable.
• Suggest improvements
• BER will need a complete report before next PART
  re-assessment (scheduled for Spring 2007)