START-08/pre-HIPPO FIELD CATALOG AND DATA MANAGEMENT

1
START-08/pre-HIPPO FIELD CATALOG AND DATA
MANAGEMENT
Steve Williams NCAR Earth Observing Laboratory
(EOL) Boulder, Colorado START-08/pre-HIPPO
Planning Workshop Boulder, Colorado 8-9 January
2008
2
Project Data Management Strategy

• Early involvement in project planning
• Close coordination with PIs to develop data
  management strategy (e.g., plan, policy, format,
  special collection and processing)
• Consistent implementation of strategy for
  lifetime of project and beyond
• Reliable and efficient archive and distribution
  system
• Easy and efficient long-term access to datasets
  by broader community including stakeholders,
  educators, and students

3
Project Data Management Considerations

• Develop Data Management Plan
• Data Types
• Data Formats and Documentation
• Data Collection
• Real-time Data Requirements
• Data Quality Control
• Data Archival
• Data Distribution
• Coordination with other Programs

4
NAME Data Management Timeline
2003
2005
2004
Complete Operations Summary (WWW)
Finalize Operations Plan
Operational Data Staged to CODIAC
Finalize Data Management Plan
Data Analysis Workshop
8 9 10 11 12 1 2 3 4 5
6 7 8 9 10 11 12 1 2
3 4 5 6 7 8 9 10 11 12
1
Final Data Staged
Preliminary data staged
Integrated Data Staged
Draft DM Plan
Field Ops and Data Collection
Final Specs from PIs
QCd data from PIs
OPS Planning Meeting
Planning Meeting
On-line Catalog Operational (CODIAC)
5
TYPICAL DATA POLICY SUMMARY

• All investigators must agree to promptly submit
  their data to the archive

• All data shall be provided to other Project
  Investigators upon request

• During the initial 1-year data analysis period,
  data may be provided to a third party only with
  the permission of the investigator(s) who
  collected the data

• All data will be considered public domain not
  more than 1-year following the end of the field
  phase

• Any use of the data will, at a minimum, include
  acknowledgment. Co-authorship TBD with the
  investigator(s) who collected the data

6
EOL FIELD CATALOG TOOL

• In-field tool to ingest and display operational
  and preliminary research data and project
  documentation for making real-time decisions and
  evaluating project progress
• Features
• Daily Mission Reports
• Operations Summary
• Facility Status Reports
• Data Analysis Products
• Authoring Tools
• Web-based access

7
FIELD CATALOG SAMPLE PRODUCTS
8
UCAR Office of ProgramsUniversity Corporation
for Atmospheric Research
9
T-REX Field Catalog Statistics
http//catalog.eol.ucar.edu/trex/

• Reports/Summaries (Status, Mission, and
  Operations)
• 340 documents and 1555 image files (0.26 GB)
• Research Platform Products (Aircraft, Surface,
  Lidar, Upper Air)
• 124,150 image files (8.01 GB)
• Operational Products (Satellite, Surface, Radar,
  Upper Air)
• 85,357 image files (7.81 GB)
• Model Output Imagery (Analysis and Forecast
  Fields)
• 143,252 image files (10.22 GB)
• TOTALS 354,654 Files (26.30 GB)

10
PROPOSED START08 FIELD CATALOG PRODUCTS
Daily reports HIAPER flight summary HIAPER
status Ops plan of the day Weather
summary Observational products (archive imagery
in FC) Satellite products GOES East and
West imagery (VIS, IR, WV) - hourly Radar
products - hourly U.S. national
radar mosaic Surface products North
American surface analysis Operational model
analyses and forecasts - 00Z and 12Z NCEP GFS
00Z and 12Z forecasts (graphics at 12-hour
intervals out to 2 days) 200, 300, 500,
700 hPa PCPN, MSLP, 850 hPa Z and wind
- PCPN, MSLP, 850 hPa T -
MSLP and thickness Tropopause
Height - START08 team will generate these
figures Pressure - START08 team will
generate these figures PV
Selected PV surfaces - START08 team will generate
these figures Cross sections of T,
wind, PV, TROP - START08 team will generate these
figures Research products WRF CLaMS
11
NAME Data Flow
Delayed Operational Data

• Operational
• Data Ingest
• GTS
• Satellite
• Radar
• Model
• Special
• Research
• Data Ingest
• Aircraft
• Ships
• Profiler
• ISS

NAME PI Data and Products

• Distributed
• Data Centers
• CICESE
• IMTA
• NASA/DAACs
• NASA/GSFC
• NASA/JPL
• NASA/LaRC
• NOAA/ETL
• NOAA/NCDC
• NOAA/NODC
• NOAA/NSSL
• NSIDC
• SMN
• Univ. of AZ

Preliminary Data Sets
Preliminary Products

NAME Data Archive Center NCAR (EOL)
Catalog Information
NAME Online Field Catalog
Final PI Research Data

SMEX-04 Data Center (NSIDC)

12
(No Transcript)
13
(No Transcript)