Geoscience%20Data%20Analysis%20and%20Visualization%20Tools%20from%20NCAR

1
Geoscience Data Analysis and Visualization
Tools from NCAR

• HDF/HDF-EOS Workshop XI
• Nov 6-8, 2007
• David Brown

2
Visualization and Enabling Technologies
SectionNational Center for Atmospheric
ResearchPhotos Lynda Lester
Mesa Lab Boulder, Colorado
3
Topics

• Two interfaces the same capabilities (mostly)
• NCL - a self-contained scripting language
• PyNGL and PyNIO - Python modules
• Quick overview visualization and analysis
• Data model
• State of our I/O component development
• Enabling access to HDF 5, HDFEOS 5, and NetCDF 4

4
Brief History NCAR Graphics HLUs NCL NIO PyNGL PyN
IO
NIO 2005

5
NCAR Command Language (NCL)A scripting language
designed forscientific data analysis and
visualization

1. Simple, robust file input and output
2. Hundreds of analysis functions
3. Visualizations (2D) are world class and highly
   customizable

http//www.ncl.ucar.edu/
6
NCL Data analysis

• Array-based syntax and operators
• Hundreds of functions
• Spherical harmonics
• Scalar and vector regridding
• Vertical interpolation
• EOFs
• Many tailored to geosciences
• Most handle missing data
• Can call C and Fortran routines

7
NCL Visualization

• High-quality and customizable visualizations
• Contours, XY, vectors, streamlines
• Maps with most common map projections
• Handles data on regular and irregular grids,
  triangular meshes
• Specialized scripts for skew-T, wind roses,
  histograms, Taylor diagrams, panels, bar charts
• GSUN interface simplifies visualization
• Over 1,400 visualization options available

8
(No Transcript)
9
PyNGL - Python module

• Python NCL Graphics Library
• Python version of popular GSUN interface
• Same publication-quality graphics as NCL
• Utilizes existing Python modules and development
  tools (swig, Numeric, NumPy)
• Contains some climate-specific data analysis
  functions
• Extensive and updated documentation

http//www.pyngl.ucar.edu/
10
NCL Data Model

• Based on netCDF 3
• Language variables can have
• Named dimensions
• Attributes
• Coordinates variables
• A convention in NetCDF but a language feature in
  NCL
• 1d array with the same name as a dimension
• Contains coordinate values for dimension elements

11
Supported file format I/O

• One function reads all supported data formats
• NetCDF (now including NetCDF4 classic), HDF4,
  HDF-EOS 2, GRIB 1 and 2
• Writes NetCDF and HDF4
• All file formats massaged into same model file
  variables have basically the same features as
  internal variables
• Flat name space HDF group names are appended to
  variable names

12
PyNIO - Python module

• Same I/O library as NCL (libnio)
• Reads and writes same formats as NCL
• Same NetCDF-like view of all formats
• But interfaces with NumPy for data access
• Modeled on the Scientific NetCDF module
• NioVariable type is a reference to the variable
  in the file (metadata is attached to variable)
• Dereferencing it results in a NumPy variable with
  no metadata
• Numpy is a Python array processing module

http//www.pyngl.ucar.edu/Nio.shtml
13
2D Coordinates

• Traditional NetCDF coordinate variables are 1D
  vectors
• However, satellite data and model data on modern
  grids require 2D coordinate grids
• 2D coordinate variables created on the fly for
  GRIB grids and HDFEOS Grid-type data
• Added value variables that allow data to be
  plotted on any map projection

14
Enabling NetCDF 4 classic

• Recompile with NetCDF 4 and HDF 5 1.8 beta (we
  also included szip)
• Add options for specifying format and compression
  level
• Tests using ncl_convert2nc on GRIB files show
  reduction in file size by 1/2 over GRIB.
• Caveats
• Beta version NetCDF 4 not yet supported on some
  architectures (64 bit mainly)
• Not supported for OPeNDAP NetCDF client library

15
Path to HDF, HDFEOS 5 and full NetCDF 4 support

• size_t for dimension sizes to enable large
  variables on 64-bit hosts
• Add support for more atomic types (int64)
• Support for groups in the file context
• User control of chunking
• Compound and variable length data types
• Components to read HDF 5 and HDFEOS 5
• Current NetCDF module extended to handle v4
• More support for aggregation

16
Current status

• NCL 5.0.0 released this week
• NCSA-style Open Source license (finally)
• Binaries available under slightly more
  restrictive license (because of included 3rd
  party software)
• PyNGL/PyNIO 1.2.0 available
• Supports NumPy 1.0.x and Python 2.5
• Binaries (easy installation) available for
• Various flavors of Linux, Mac OSX, Cygwin
• Other Unix systems, 32 and 64 bit

17
Distribution of NCL sites
Distribution of PyNGL/PyNIO sites
18
Support

• Websites with extensive documentation
• Tutorials
• Hundreds of examples with downloadable scripts
• Active email lists
• ncl-talk_at_ucar.edu, pyngl-talk_at_ucar.edu
• Hands-on training workshops

19
Download locations

• NCL
• PyNGL and PyNIO

http//www.ncl.ucar.edu/download.html
http//www.pyngl.ucar.edu/download.html
20
Questions? Email me dbrown_at_ucar.edu
21
What kind of research(from our ESG download
website)
monsoons
education
atmospheric research
rainfall
cloud modeling

• astronomy

climate change
dust
weather forecasting
data conversion
paleoclimatology
salmon abundance
lightning
utility forecasting
glaciers
medical
oceanography
flood risk mgmt
pollution
hydrology
marine studies
electric power
acoustics
air quality
fire modeling
storms
physics
planetary atmospheres
aerosol light absorption
agriculture
microbiology
22
Sample GRIB outputfrom PyNIO
filename ruc2.T00Z.bgrbanl path
/Users/dbrown/grib/ruc2.T00Z.bgrbanl.grb file
global attributes dimensions gridx_236
113 gridy_236 151 lv_HYBL2 40
lv_DBLL3 5 variables float
PRES_236_TRO ( gridx_236, gridy_236 )
center US National Weather Service - NCEP
(WMC) long_name Pressure
units Pa _FillValue -999
coordinates gridlat_236 gridlon_236
level_indicator 7 grid_number
236 parameter_number 1
model RUC Model from FSL (isentropic
scale 20km at 40N) forecast_time
0 initial_time 05/11/2000
(0000) float gridlat_236 ( gridx_236,
gridy_236 ) corners ltARRAYgt
long_name latitude GridType
Lambert Conformal Secant or Tangent, Conical or
bipolar units degrees_north
23
Resources (a.k.a. attributes)

• Same control mechanism for NCL and PyNGL
• Detailed control of the appearance of a
  visualization
• Example using POP grid ocean current data

24
(No Transcript)
25
(No Transcript)
26
(No Transcript)
27
(No Transcript)
28
(No Transcript)
29
NCL and PyNGL visualization examples

• scatter and line plots, contours, vectors,
  skew-t, meteograms, maps, wind barbs, bar charts,
  streamlines, trajectories, filled polygons,
  paneled visualizations

30
(No Transcript)
31
(No Transcript)
32
Skew-T graphic Courtesy of Dennis Shea
A Skew-T plot is used by meteorologists to
analyze data from a balloon sounding.
33
(No Transcript)
34
(No Transcript)
35
(No Transcript)
36
Based on a visualization of Adam Phillips
37
Based on a visualization of Joel Norris
(Scripps) using dummy data
38
Data from Climate Analysis Section Christian
Guillemot
39
GRIB 2 TIGGE (THORPEX Interactive Grand Global
Ensemble) data Japan Meteorological Agency
THORPEX - an international research and
development program responding to the weather
related challenges of the 21st century to
accelerate improvements in the accuracy of 1-day
to 2- week high impact weather forecasts for the
benefit of society, the economy and the
environment.
40
Courtesy Jeff Yin NCAR CGD
41
(No Transcript)
42
Image courtesy of Nan Rosenbloom, CGD
43
First two map databases built-in high-resolution
available as simple download
44
This grid could be described as a tripole grid
that is further modified by the arbitrary
displacement of some portions of the grid to
achieve finer resolution over areas of interest
(typically, ocean areas).
Christophe Cassou (CNRS/CERFACS)
45
Courtesy Mark Stevens, NCAR CGD
46
Brett D. Estrada Naval Research Laboratory
47
Grid from Tom Gross NOAA/NOS/CSDL/MMAP
48
Special grids you can contourSoon to include
vectors and streamlines

• Grids already on a triangular mesh
• Grids with cell boundary information
• Grids represented by one-dimensional arrays of x,
  y, and z values

49
http//isccp.giss.nasa.gov/
50
http//isccp.giss.nasa.gov/
51
ARPEGE GRID (used by Meteo-France) Christophe
Cassou (CNRS/CERFACS)
Similar to ISCCP grid, but with somewhat finer
resolution. Grid is rotated with respect to the
globe so as to put its poles somewhere other than
at the North and South Pole.
52
Data from Dave Randall, Todd Ringler, Ross Heikes
of CSU
Most geodesic grids appear to be formed by
elaborating an icosahedron each of the 20 faces
of the icosahedron is subdivided into smaller
triangles in a more or less obvious way.
53
Data from Dave Randall, Todd Ringler, Ross Heikes
of CSU
54
Station data contouring on a triangular mesh
55
(No Transcript)
56
Triangular mesh from Brett D. Estrada, Naval
Research Laboratory
Grid from Brett D. Estrada, Naval Research
Laboratory
57
Triangular mesh from Tom Gross NOAA/NOS/CSDL/MMAP
58
Samples from VASPACKTVectors and streamlines on
a triangular mesh

• Developed by Dave Kennison
• Simple vectors
• Curly vectors
• Streamlines
• Examples use dummy data

59
Simple vectors
60
Curly vectors
61
Streamlines