Introduction to netCDF

1
Introduction to netCDF

• Juli Rew
• CISL
• SOARS/SIParCS, June 6, 2008

2
Concepts to be Covered

• Overview of netCDF
• netCDF file format
• netCDF library and programming model
• Useful commands and tools
• Examples

3
Who Invented netCDF?
Unidata
Unidata - helps universities acquire, display,
and analyze Earth-system data UCAR University
Corporation for Atmospheric Research - a
nonprofit consortium of universities Thanks to
Russ Rew of Unidata and Jianwei Li of
Northwestern University for information in this
presentation Help support_at_unidata.ucar.edu
4
What is netCDF?

• An effective way to store and retrieve scientific
  data
• Portable - A platform-independent binary data
  format
• Self-describing - A way of making the data show
  you what it contains
• Interface - A library of routines for
  reading/writing the data

5
Why Use netCDF?

• Facilitate the use of common datasets by distinct
  applications
• Permit datasets to be transported between or
  shared by dissimilar computers transparently
  (without translation)
• Reduce programming effort usually spent
  interpreting formats
• Reduce errors from misinterpreting data and
  ancillary data
• Output data from one application into another
• Establish an interface standard which simplifies
  the inclusion of new software into existing
  system.

6
netCDF Dataset Jargon

• Dimensions
• name, length
• Variables named arrays
• name, type, shape, attributes, values
• Fixed sized variables array of fixed dimensions
• Record variables array with its most-significant
  dimension UNLIMITED
• Coordinate variables 1-D array with the same
  name as its dimension
• Attributes - metadata about the data
• name, type, values, length
• Variable attributes
• Global attributes

7
More netCDF Terms

• Coordinate Variable a 1D variable with the
  same name as a dimension, which stores values for
  each dimension value (for example, latitude)
• Unlimited Dimension a dimension which has no
  maximum size. Data can always be extended along
  the unlimited dimension. Recommended when
  creating new netCDF dataset that may be extended
  later.

8
netCDF Example

• Suppose you want to store temperature and
  pressure values on a 2D latitude/longitude grid.
• In addition to the data, you want to store
  information about the lat/lon grid.
• You may have additional data to store, for
  example, the units of the data values.

9
netCDF Example
Variables
Dimensions
Attributes
temperature
latitude
units celsius
pressure
longitude
units mbar
Coordinate Variables
latitude
longitude
10
netCDF File Format - binary

• Header
• Number of record variables
• Dimension list
• Global attribute list
• Variable list
• Data (row-major)
• Fixed-sized data
• data for each variable is stored contiguously
• Record data
• a variable number of fixed-size records, each of
  which contains one record for each of the record
  variables in order.
• Both use extended XDR

11
Accessing netCDF Data

• Access is direct - efficient
• Can access elements of arrays
• Can stride through arrays to pick up portions of
  data

12
netCDF Library Programming Model
Routines available for many languages Modes
definition mode, data mode IDs dataset ID,
dimension ID, variable ID, attribute number
13
Example library functions (Fortran)

• Typical order of calls to create a netCDF file

14
Library functions, cont.

• There are also inquiry functions
• nf_inq_dimid
• nf_inq_varid
• nf_get_att
• nf_get_var

15
Supported Data Types

• Different library functions for each data type
• Automatic conversion to or from external type
• Conversion from one numeric type to another

16
netCDF 4 - new features

• Parallel write support
• Support for multiple unlimited dimensions
• Support for efficient structure changes
• Support for ragged arrays and other data
  structures
• String type
• Unicode names

17
CDL - Common Data Language

• Little language defined for displaying
  information about netCDF files
• Can be used to create files without programming
• Can be used to create writing program in Fortran
  or C
• Used by ncgen/ncdump utilities

18
Recall Our Example - How Can We Write a Fortran
Program?
19
CDL Example
netcdf foo // example netCDF specification in
CDL dimensions lat 10, lon 5, time
unlimited variables int lat(lat),
lon(lon), time(time) float z(time,lat,lon),
t(time,lat,lon) double p(time,lat,lon) int
rh(time,lat,lon) latunits
"degrees_north" lonunits "degrees_east"
data lat 0, 10, 20, 30, 40, 50, 60, 70,
80, 90 lon -140, -118, -96, -84, -52
20
Useful Commands - ncgen

• ncgen uses CDL file as template and generates
  either
• 1. Code with library calls that will produce
  a netCDF file
• Code can be generated in Fortran or C
• Fortran ncgen -f mydata.cdl gt mydata.f
• C ncgen -c mydata.cdl gt mydata.c
• 2. A netCDF file
• ncgen -o mydata.nc mydata.cdl

21
Exercise

• cd to /SOARS/NETCDF
• Generate a Fortran program from the CDL file
  called start.cdl
• Compile and run the program. Example
• f77 -I /usr/local/include \ dataprog.f
  -L/usr/local/lib \ -l netcdf./a.out
• What is the resulting netCDF data file called?

22
Useful commands - ncdump

• Ncdump useful for quick glimpse of what kind of
  data file contains - looks at files metadata
• ncdump -h myfile.nc
• shows the header information in the file
• ncdump -c myfile.nc
• shows the header info plus values of coordinate
  variables (variables that are also dimensions)

23
Exercise

• Look at structure of an existing file using
  ncdump
• Compare -c and -h options
• What is the difference?

24
More Examples and Exercises

• 1. Open a netCDF file using NCL and do a contour
  diagram of the data
• cd to SCRIPTS directory
• Look at gsun02n.ncl - this is an NCL script that
  opens the netCDF file contour.cdf and plots it
• To run it, typencl gsun02n.ncl

25
Examples and Exercises

• 2. Browse the data in a netCDF file using ncview
  browser
• Ncview was written by David Pierce at Scripps
  Institution of Oceanography
• To use ncview (from /SOARS/NETCDF/SCRIPTS
  directory), typesetenv XAPPLRESDIR
  /SOARS/NETCDF/ncview-1.92cncview contour.cdf
• Select a variable, such as Temperature, to view