Scientific Data Stewardship

1
Scientific Data Stewardship

• Mitch Goldberg and John Bates
• National Environmental Satellite Data and
  Information Service
• National Oceanic and Atmospheric Administration
• Presentation to NPP Science Team

November 5, 2003
2
Scientific Data Stewardship Generic Guiding
Principles

• Careful monitoring of observing system
  performance for climate applications
• Generation of climate data records through
  validation of the calibration process,
  reprocessing, product generation and the blending
  of in situ and satellite measurements
• Provide state of the environment information for
  decision makers and place the current state in
  its historical context
• Archive and access to fundamental measurements,
  products and metadata (supported by CLASS)
• Data archaeology (data rescue)

3
What is a CDR?

• A climate data record is a collection of data
  that can be used to construct a high quality time
  series with quantified error characteristics.
• The time series should be free of instrument
  artifacts and changes in algorithms.
• CDRs provide information to
• monitor change (climate variability and trends)
  of the Earths climate.
• predict change especially SI forecasts
• input to model re-analyses (note reanalysis is
  also a CDR)
• validate climate prediction models and model
  reanalyses
• understand processes ( water vapor-cloud-radiatio
  n feedback)
• Goal - Provide credible information to users
• Generation of CDRs cuts across all other NOAA
  Climate Program Components (Forcings, Prediction
  and Projections, Climate and Ecosystems, Climate
  Information for Decisions)

4
The NOAA Climate Program has revised its program
structure to be consistent with the goal
structure of the Climate Change Science Program
(CCSP).

• NOAA Climate Program Components
• 1. Climate Observations and Analysis
• 2. Climate Forcing
• 3. Climate Predictions and Projections
• 4. Climate and Ecosystems

• CCSP GOALS
• 1. Extend knowledge of the Earths past and
  present climate and environment, including its
  natural variability, and improve understanding of
  the causes of observed changes.
• 2. Improve understanding of the forces bringing
  about changes in the Earths climate and related
  systems.
• 3. Reduce uncertainty in projections of how the
  Earths climate and environmental systems may
  change in the future.
• 4. Understand the sensitivity and adaptability of
  different natural and managed systems to climate
  and associated global changes.
• 5. Explore the uses and identify the limits of
  evolving knowledge to manage risks and
  opportunities related to climate variability and
  change.

5
Adjustment in FY05 for Scientific Data Stewardship

• Goal Ensure that satellite observations are
  processed into Climate Data Records, archived,
  and distributed to users in a manner that is
  scientifically defensible for monitoring,
  diagnosing, understanding, predicting, modeling,
  and assessing climate variation and change
• Funding is for 1) Observing System Performance
  Monitoring and 2) Generation of Climate Data
  Records
• Generation of CDRs will be a combination of
  internal and external activities (via grants and
  AOs, interagency partnerships )

6
Developing a Plan

• Need a comprehensive plan to generate CDRs from
  satellite observations.
• NOAA is particularly concerned that the plan for
  CDR generation include processes to promote
  science and user community participation and
  consensus.
• Commissioned an NRC study to provide
  advice/recommendations

7
Motivation for Developing A Plan

• Increasing demand for credible climate
  information to monitor and predict climate change
  and variability.
• A key CCSP objective is the scientific
  stewardship of the production of climate data
  records
• NOAAs operational satellites are a major source
  of sustained long term observations of the
  Earths climate
• NOAA is not fully exploiting these satellites to
  meet the needs of climate change investigators
• NASAs advanced observations from EOS are
  transitioning to NPOESS.
• There is a need to develop and implement a plan
  to optimize the creation of Climate Data Records
  that satisfy the requirements of the climate user
  community -AND we want this plan to be
  implemented for NPOESS

8
NPOESS can provide many of the key products
needed to monitor climate

• Atmosphere
• Temperature soundings
• Moisture soundings
• Clouds
• Aerosols
• Earth Radiation Budget
• Precipitation
• Ozone

• Ocean
• Surface temperature
• Ice cover
• Surface winds
• Color
• Sea level
• Land
• Vegetation condition
• Snow cover
• Other land characteristics (e,g., albedo, skin
  temperature, insolation,
• soil moisture)
• Biomass burning

9
Academy StudyProvide NOAA with advice in
developing the Plan

• The NRC involvement consists of two phases
• Phase 1 -- Information gathering workshop,
  including review of NOAA white paper on
  generation of CDRs (August 2003), and an interim
  report to NOAA (December 2003)
• Phase 2 Review of NOAA draft plan (March 2004)

10
NRC CDR workshop

• Over 50 participants
• We presented a NOAA white paper on CDRs
• Keynote presentations by Trenberth, Stephens,
  Kalnay, Rossow

11
Academy Study

• NOAA is seeking advice from the Committee on the
  following
• How does a CDR become a community standard?
• How can NOAA ensure that CDRs are responsive to
  user needs?
• What are the key attributes of successful CDR
  generation programs?
• What are the advantages and disadvantages of
  different models or strategies for producing
  CDRs, such as using partnerships among
  government, academia, and the private sector,
  different blends of space-based and in-situ data,
  or other approaches?
• How can NOAA learn from present and past efforts
  such as the NOAA/NASA Pathfinders and EOSDIS?
  What are the successes and failures, and how do
  we emulate the success or avoid the pitfalls?

12
Academy Study

• What approach should NOAA consider for obtaining
  science guidance and oversight for its end-to-end
  CDR program?
• What are some realistic options for engaging NASA
  in an end-to-end system for generating CDRs from
  the operational satellites?
• What are the optimal mechanisms for entraining
  the user community in the use, test, and
  evaluation of CDRs and for obtaining feedback?
• What should be the role of NOAA in international
  data projects? What functions should NOAA
  perform?
• What are some viable approaches for algorithm
  selection? Once an algorithm is implemented, what
  procedures should be used to determine when the
  algorithm should be updated or replaced? How
  should replacement algorithms be selected?

13
Purpose of NOAA White Paper

• Provide Committee with background information,
  issues, and initial NOAA thoughts on CDR
  creation
• Motivation to establish end-to-end system for
  creating climate data records (CDRs) from
  operational satellites
• Review of current programs to create CDRs (NOAA
  and external)
• Summary of lessons learned in the use of
  satellite observations for climate monitoring
• Conceptual framework for moving ahead
• Additional issues for Committee to consider
• Serves as a draft background section of the NOAA
  Plan for Creating CDRs from Operational
  Satellites

14
Where are We Now? Climate Products Produced from
Operational Satellites in Different Ways

• Operational products gridded to climate scale by
  NOAA or external investigators (Several DMSP
  SSM/I products ERB, SST, vegetation and aerosols
  from AVHRR)
• NOAA/NASA collaborations (Ozone from SBUV)
• Produced by external investigators (Atmospheric
  temperatures from MSU, snow cover from AVHRR)
• One-time projects such as NOAA/NASA Pathfinder
• International climate projects (ISCCP clouds and
  GPCP precipitation)
• NOAA/NESDIS satellite data resources
• 25 years of providing operational data
• NPOESS, NPP CLASS

15
Where are We Now? Examples of Climate Data
Records Based on Operational Satellite
Observations
Since
Produced by
Satellite Instrument
Climate Product
NESDIS
1978
POES/AVHRR
Earth Radiation Budget (ERB) Outgoing long-wave
radiation (OLR) Absorbed solar radiation (ASR)
NESDIS/NASA
1985
POES/SBUV/2 POES/ATOVS/HIRS
Ozone
Blended Sea Surface Temperature (SST)
NESDIS/NWS
1981
POES/AVHRR
DMSP SSM/I Climate Products (rainfall, rain
frequency, snow cover, sea ice cover, clouds,
water vapor, and oceanic wind speed)
NESDIS
1987
DMSP SSM/I
NESDIS
1982
POES/AVHRR
Vegetation (NDVI and drought index)
Univ. of Alabama
1979
POES/MSU
Atmospheric Temperature
Rutgers Univ. Climate Laboratory
POES/AVHRR, GOES, Meteosat, GMS Visible imagery,
DMSP/SSM/I
1966
Snow Cover
WCRP/International Satellite Cloud Climatology
Project
POES/AVHRR, GOES, Meteosat, and GMS Visible/IR
imagery
1983
Clouds
WCRP/Global Precipitation Climatology Project
POES/AVHRR, GOES Meteosat and GMS Visible IR
imagery, DMSP SSM/I
1986
Precipitation
16
Where are We Now? CDRs Produced by NESDIS by
Gridding Operational Products to Climate Scale
OLR 7-03
Rainfall 7-03
Vegetation Health 8-10-03 and 8-10-02
17
Where are We Now? Ozone A True NOAA/NASA CDR

• Ozone is adjusted to NOAA-9
• Validated against Dobson Stations
• Reprocessed when new algorithms are developed
• Compared with models

18
Where are We Now? Examples of CDRs Produced by
External Groups
U. Alabama Tropospheric Temp
Snow Cover Anomalies
ISCCP Global Cloud Amount
GPCP Global Precipitation
19
Where are We Now? The NOAA/NASA Pathfinder
Project A One-Time Reprocessing Project

• Initiated in the early 1990s to generate climate
  quality data sets from the archived observations
  of the U.S. operational environmental satellites
• AVHRR (since 1981), TOVS (since 1979), SSM/I
  (since 1987), GOES (since 1978)
• Guided by scientific working groups
• Transcribed the data from hundreds of thousands
  of magnetic tapes to more accessible media for
  processing
• Developed improved instrument calibration
  histories
• Selected community consensus algorithms for
  generating geophysical products
• Processed, validated, and provided access to the
  data sets
• Produced CDRs temperature and moisture
  profiles, cloudiness, precipitation, earth
  radiation budget, and aerosols

20
Where are We Now? The NOAA/NASA Pathfinder
Project A One-Time Reprocessing Project (cont)
Mean Global Maps Based on 20 Years of AVHRR
Observations
Cloud Amount
Reflected Solar Radiation
OLR
Aerosol Amount
21
Lessons LearnedImportant Questions Spawned via
Production of Climate Records Using Operational
Data

• What are the priority climate products that an
  operational center should produce and how?
• What methods and good practices should be used to
  facilitate access to a variety of products,
  metadata, and raw data?
• What in situ baseline observations are required
  to use for long-term calibration and validation
  of space-based observations?
• What critical performance measures should be
  monitored in real-time to avoid single-point
  failures in long-term records of critical climate
  variables?
• How do we reconcile different observations and
  analysis techniques to achieve the best and/or
  consensus climate data records?

22
Where are We Going?Creating Quality Climate Data
Records Requires

• Lowest level of data (level 1) be preserved with
  complete documentation and metadata, includes
  data that records the satellite and instrument
  performance
• Observing system performance monitoring to
  minimize spatial and temporal biases
• Tools to detect and account for changes in the
  observing system
• Science team guidance and participation
• Near Real-Time CDR Generation
• Tight connection between algorithm developer and
  CDR generator (may be same group)
• Strong calibration/validation program
• Research with the data set as part of the program
• Collaboration with user communities (e.g.,
  diagnosticians, modelers) to obtain feedback

23
Where are We Going?Creating Quality Climate Data
Records Requires (cont)

• Re-processing
• An improved algorithm is developed
• New information on an instrument
• An error is discovered in the processing system
• Research and Application
• Development of climate quality algorithms
• Analysis of time series to detect emerging trends
• Joint studies with climate modeling community
• Production of periodic assessments for decision
  makers, other climate researchers and the public
• Data Requirements
• End-to-end data management
• Near real-time access to data (including raw
  radiances)
• Development of community consensus algorithms and
  data standards
• Complete archiving data, meta data, source code,
  ancillary data, etc.
• Free and open sharing and exchange of climate
  data
• Nationally and internationally

24
Where Are We Going?Functional Areas

• An end-to-end process for creating CDRs could
  consist of 5 functional areas
• Observing System Performance Monitoring
• Detect problems early
• Production of near real-time CDRs
• Monitor current state of climate system and short
  -term variations
• Reprocessing of CDRs for long-term records
• Consistent, seamless, high quality time series
  with minimized bias
• Climate research and applications
• Joint activities with external community
• Archive and distribution
• Includes output of above activities, metadata,
  and timely distribution
• Above are guided by climate science teams
  experts in instrument characterization,
  algorithms, validation, data management,
  applications, and observing system performance
  monitoring

25
Where Are We Going?
Conceptual Framework
CCSP Observations Management Structure
Climate Data Science Teams
International Data Programs
Experts in Instrument Characterization,
Algorithms, Validation, Data Management,
Applications, and Observing System Performance
Monitoring
Improved CDR Products
Leveraging Resources/ Collaboration
Instrument Monitoring
Production of Near Real-Time CDRs
Processing of CDRs for Long Term Records
Climate Research/ Applications/Monitoring
International Space Agencies
Design for Future Systems
Archive and Distribution
Interactions with Users
NASA
Modeling
Monitoring
Prediction
Research
26
Concluding remarks

• Many recommendations from NRC reports for
  observing system performance monitoring,
  generation of climate data records, archive and
  distribution
• NOAA is taking the lead to turn these
  recommendations into an operational program.
• To be successful, NOAA needs to have interagency
  partners.

27
Other NPOESS activities

28
16 NESDIS/ORA Scientists are funded by IPO
Internal Govt. Studies
29
AIRS/CrIS Risk Reduction Studies

• Early demonstration of operational processing of
  high spectral resolution infrared sounder data
  prior to CrIS, IASI and GOES-R
• Validation of EDR performance
• Early opportunity for forecast centers to learn
  how to assimilate advanced IR data
• Demonstration of positive impact for NWP
• Extended study to CDR risk-reduction

30
AIRS/CrIS CDR study

• Optimal Data Compression for archiving
• Gridded data sets for fast reprocessing.

31
NPOESS Data Exploitation
NPOESS Data Exploitation (NDE) provides
environmental product processing and
dissemination services
CUSTOMERS NWS, FAA, DOA, Universities, etc.
NPOESS Ground Systems .
NPOESS Data Exploitation (NDE)
Environmental Products
Archive CLASS
Environmental Products
National Polar-orbiting Operational
Environmental Satellite System
32
NDE Objectives
Mission To ensure that the user community is able
to exploit NPOESS products

• System Development Objectives
• Develop algorithms that exploit NPOESS products
• Process data records provided by the NPOESS
  Integrated Data Processing Segment (IDPS) into
  products that satisfy the unique requirements of
  NOAAs civilian users
• Format the products for users (BUFR,
  gridded/mapped, etc.)
• Assist the Integrated Program Office (IPO) to
  Monitor/Validate NPOESS data records
• Develop an integrated, maintainable system to
  generate and disseminate all NDE products in near
  real-time

33
Examples of NPP NOAA Unique Products

• Compressed radiance data from CriS/ATMS (Spatial,
  Spectral and EOF data reductions)
• Sounder products for National Weather Service
  field units
• Radiation budget parameters (e.g. OLR) from CrIS
• Blended Snow Products from VIIRS, ATMS and CMIS
  and GOES
• Blended Ozone products from CrIS and OMPS
• Vegetation deliver weekly or biweekly global
  map of green vegetation fraction and leaf area
  index, drought index, vegetation health
• Hazard Geographic Information System (GIS)
  products Smoke, Fire, Aerosols, Flash Flood,
  Precipitation
• Coastwatch Products of Ocean Color, Coral
  Bleaching and SST
• Microwave-only products from ATMS (temperature,
  moisture, cloud liquid water, precipitation,
  rainfall rates, surface emissivity, snow/ice)
• Daily global, regional maps (gridded data) of all
  EDRs and SDRs for the science community and for
  validation

34
Backup Slides

35
(No Transcript)
36
Trenberths recommendation to NRC panel

• Climate monitoring requires a long-term
• commitment to quality and stability.
• Many of the climate-related signals are small,
• obscured by natural variability.
• There must be an active program of research
• and analysis utilizing climate data sets to
• ensure the data are state-of-the-art and meet
• requirements.
• Climate research and monitoring requires an
• integrated strategy of land/ocean/atmosphere
• observations, including both in situ and
  remote
• sensing platforms, and modeling and analysis.

37
(Trenberth)
Stewardship of the data is needed establish an
organization with responsibilities for
operational climate monitoring and prediction.
Essential infrastructure has to be established to
ensure the integrity and continuity of the
observations, their analysis into products, and
links to modeling and research activities.
Needed a central facility with oversight of
the health of the observing system and resources
to build and sustain a climate observing system
operating under the guideline principles. It
would have a new management structure, authority
and infrastructure and should be responsible for
a line of products for use in all aspects of
climate, and oversight of management, access and
archival of the data. Trenberth et al 2002
BAMS
38
Stephens recommendations
Producing CDRs is a multi-dimensional effort,
and has to engage the stakeholder science
communities (pullers of data rather than pushes
of data) . The model for such engagement cannot
simply follow the science team model of EDRs.
It is not enough to produce any given CDR in
isolation from other CDRs key applications of
CDRs require correlations between them -
implying that requirements on one map to
requirements (or even) performance on another.
One approach would be to develop CDRs around
particular (science) themes, or questions (ie
make the organizing principle less
product-centric).
39
WindSat TDR to SSM/I Brightness Temperature 37
GHz Horizontal Polarization
WindSat / Coriolis NPOESS CMIS Risk Reduction
The higher spatial resolution of WindSat is
apparent in these images.
40
Real-Time Operational Demonstrations
NPP (FY06) CrIS/ATMS VIIRS
Coriolis (FY03) CMIS
Aqua (FY02) AIRS/AMSU/HSB MODIS
METOP (FY06) IASI/AMSU/MHS AVHRR
NPOESS (FY10) CrIS/ATMS, VIIRS, CMIS, OMPS
Use of Advanced Sounder Data for Improved Weather
Forecasting/Numerical Weather Prediction
41
NDE Organization