Research Data at NCAR

1
Research Data at NCAR

• 1 August, 2002
• Steven Worley
• Scientific Computing Division
• Data Support Section

2
Outline

• Archive building Basic ingredients
• Proactive content development
• Support for NCAR/UCAR/Universities
• Access to the data

3
Archive Building

• Collect, maintain, and archive data
• Data types
• Observations
• Analyses
• Reanalyses
• Model output data
• Staff data stewards
• SEs educated in meteorology and oceanography
• Activities
• Quality check the information
• Write discovery metadata
• Provide human based consulting

4
Proactive Content Development

• NCAR has a privileged position
• NOT restricted by Agency mission demands
• e.g. You must archive Level 1 MODIS data!
• Remain flexible to meet research demands
• Choose efficient and cost effective methods

5
Proactive Content Development

• Example of being proactive
• NCARs in situ observations collections are the
  most complete in the world
• Long term dedication 35 years
• Involved many special data exchanges
• (national and international)
• Add key data not available in the standard
  archives
• Achievement widely recognized
• US Academy of Science
• NCEP and ECMWF

6

• Challenges
• detect and repair, or identify erroneous data
• rectify historical format differences to create
  a user friendly best composite collection
• To meet the challenges
• have the knowledgeable staff
• need staff time to develop the graphics and
  analysis software

7
Proactive Content Development

• Many other examples
• Atmospheric upper air data
• Marine surface data
• Etc.

8
Proactive Content Development

• Importance of these collections
• Basis for research data products, e.g.
• NCEP/NCAR Reanalysis
• NCEP DOE Reanalysis II
• ERA40 (processing now at ECMWF)
• NCEP Regional Reanalysis (new)

9

• Countries that have received NCEP/NCAR Reanalysis
  from SCD
• CD-ROMS
• network downloads
• tape media copies

10
Proactive Content Development

• Challenges of Reanalyses Products
• Product Size
• Current sets, O( 2 TB)
• New sets, O( 20 TB)
• Many users, O( 100s/year)
• Not a problem in NCAR computing environment
• External access is the greatest challenge
• (more later)

11
Support for Users

• The Users are
• NCAR/UCAR scientists
• UCAR universities scientists and students
• International scientists and students
• Gov. agencies and private corporations
• Access through
• SCD MSS
• Customized requests
• Online data servers (ftp, and browser)
• Community Data Portal (CDP)

12

• CDP
• Features
• Real time data access
• 4-D dataset subsets
• multiple output formats
• data analysis
• Challenges
• Develop access to large (TB) datasets
• Sequential / parallel data processing
• Linkages to SANS
• Integration with other extant data systems
• To meet the challenge
• Continue to improve our servers, storage
  networks, and network connections
• Staff with expertise and dedicated time

13
(No Transcript)
14
Support for the UsersSearch and discover data

• How? ? Web based Information Server
• Features
• 5K html pages (metadata)
• All datasets are described
• Access options
• Higher level information
• Catalogs
• Project specific descriptions
• All information is current
• Based on text files and change control system
• Automatic re-build of affected .html pages

15
Support for the UsersSearch and discover data

• Next Step for Data Discovery
• Need a metadata descriptions in a standard
  format
• Enable organization wide discovery
• (eventually nation wide or global)
• UCAR has DMWG on metadata
• Likely scenario
• ltxmlgt based meta standard
• Well poised to based on .html experience

16
Take Home Key Points

• Long term involvement in archive building

• stewardship (collecting, maintaining, improving)

• Proactive development

• stay in tune with scientific needs
• remain flexible so we can accommodate the needs
• promote data exchanges that enhance the archive

• National and International collaborations

• distribute data at minimal or no cost
• provide data to large projects, e.g. NN
  Reanalysis

• Focus on scientific data content, documentation,
  and access

• have good data and metadata systems now
• more technology and software to enhance the
  access