TIGGE Archive and Access at NCAR

1
TIGGE Archive and Access at NCAR

• 4th Meeting, GIF-TIGGE Working Group
• Chinese Meteorological Administration
• Beijing, China
• March 21-22, 2007
• Steven Worley, NCAR
• Archive Team
• Douglas Schuster
• Dave Stepaniak
• Portal Team
• Nathan Wilhelmi
• Luca Cinquini

2
Outline

• Introduction
• Archive Structure and Content
• Access Portal
• Analysis Tools
• Futures
• Summary

3
Introduction

• Archive Centers
• NCAR
• ECMWF
• CMA
• Data Collection Mechanism
• Unidatas Internet Data Distribution/Local Data
  Manager system (IDD/LDM)
• Proven and scalable method
• 172 GB/day and will accommodate more

4
Archive Structure and Content
Center Start Date
ECMWF 10/2006
UKMO 10/2006
JMA 10/2006
NCEP 11/2006

• Future Data Providers
• Australia
• China (also an archive center)
• Canada
• Brazil
• Korea
• France

5
Archive Structure and Content

• Organization of TIGGE Parameters
• Four Level Types
• Grouping of parameters into data file
• (SL) Single Level
• includes surface
• (PL) Pressure Level
• 1000, 925, 850, 700, 500, 300, 250, 200, 50 hPa
• (IL) Isentropic Level
• 320K
• (PV) Potential Vorticity Level
• 2 PVU

6
Archive Structure and Content

• Storage Strategy
• Level Types separated by
• Provider
• Forecast time
• Each file contains all ensemble members and
  parameters for a Level Type
• Long-term archive at NCAR on Mass Store System
• Dataset Research Data Archive (RDA)
• Near real-time, 2-3 weeks data online
• Access via TIGGE Portal
• Note File strategy is quite different than ECMWF
  and CMA
• MARS System - enables access to individual 2D
  global grids

7
Archive Structure and Content

• Summary of Data Providers

8
Archive Structure and Content

• Type Pressure Level

9
Archive Structure and Content

• Type Isentropic Level

Type Potential Vorticity Level
10

• Type Single Level

11

• Type Single Level Parameters cont.

12
Access Portal

• TIGGE Web Portal Features (http//tigge.ucar.edu)
  
• Registration
• Search, discover, and download files
• Select data by
• initialization date/time
• data provider
• level type, i.e. file type
• forecast time
• By agreement, 48 hour delay for access
• Real-time by IPO determination

13
Access Portal
14
(No Transcript)
15
Access Portal

• TIGGE Web Portal (http//tigge.ucar.edu)
• Steps to finish a request
• Form to review and verify request
• Includes data volume
• If acceptable
• Download files one-by-one via browser
• Receive a download script to run on your machine
• Uses wget utility
• Each wget command has an encrypted key
• Running the script - stream multi-center,
  multi-file data to your system
• If not acceptable - refine your request, repeat,
  or stop

16
Access Portal

• TIGGE Web Portal Metrics
• Portal Users
• Total guests 1745
• User registrations 36
• Data Movement
• Data downloaded 140 GB
• Files downloaded 656
• Unique download users 21
• Expect Data Movement to increase drastically
  with new interface

17
Access Portal, long-term data provision

• NCAR Research Data Archive
• http//dss.ucar.edu/datasets/ds330.0/ -2006
• http//dss.ucar.edu/datasets/ds330.1/ -2007
• The RDA dataset enables
• Access to the long-term TIGGE archive.
• Easy use to use on NCAR CISL computers data
  coming directly from the Mass Store System
• NCAR computing accounts available upon request
• Support staff to handle requests for offline data
• Forecast file structure identical to online files

18
Analysis Tools

• Tools for Basic Data Manipulation
• WMO GRIB2 is relatively new, - tools are immature
• Forecasts with ensemble members add another
  dimension
• Improvements underway in
• NCAR Command Language (NCL)/Python
• New Release end of March 2007
• GEMPACK, GRIB-Java (Unidata)
• NOAA tools (wgrib2, GRIB-2 software libraries,
  etc)
• ECMWF GRIB API
• http//tigge.ucar.edu/data-tools.htm

19
Futures

• New providers and provider updates, e.g. NCEP is
  going to increase ensemble members next week
• Interface to receive subset data requests (e.g.
  parameter selection), the delayed mode processing
• Provide user selected grid interpolation across
  multiple models
• Add spatial sub-setting functionality
• Include subscription services for recurring
  requests
• Continue improvement on analysis tools
• Provide web services for automated requests
• Common interface for NCAR, ECMWF, and CMA
• Accomplishments are funding resource dependent

20
Summary

• NCAR now archiving data from 4 providers (ECMWF,
  NCEP, UKMO, JMA)
• Other data providers plan to begin in 2007
• TIGGE archive and access systems designed to
  accommodate high diversity in data provided
• More access improvements are needed
• Excellent collaboration with the other archive
  centers

21

• Questions
• Contact Information
• http//tigge.ucar.edu
• worley_at_ucar.edu