Unidata Overview

1
Unidata Overview

• Tom Yoksas
• yoksas_at_unidata.ucar.edu
• Unidata Program Center
• University Corp for Atmospheric Research
• Boulder, Colorado, USA
• http/my.unidata.ucar.edu

2
Topics Covered

• Unidata as Community
• Current Tools and Services
• Data Analysis and Display Software
• Data Distribution Systems
• Support
• Future Developments
• THREDDS
• DLESE/NSDL Participation

3
Unidata Is

• A Program Enabling Universities to Acquire and
  to Use Meteorological Related Data
• Program Center Credo undertake no function that
  can be performed better by the universities
  themselves
• Operated by the University Corporation for
  Atmospheric Research In Boulder, Colorado
  http//www.ucar.edu
• Sponsored by the National Science Foundationof
  the U.S. Government http//www.nsf.gov

4
University Roots

• 1983 Grassroots Madison workshop defines goals
• Weather Service changing data access system
• Personal computers/workstations proliferating
• 1984 Working groups from academia and technology
  define initial strategy
• 1986 Policy committee begins meetings
• 1987 Initial user committee meets, satellite
  data broadcast initiated
• 1988
• First user workshops held
• Local data manager (LDM) tested at universities
• User applications packages deployed

5
Unidata Community Today

• More than160 institutions
• Includes over 100 academic institutions plus
  government agencies and private sector research
  groups
• Does not count separate installations, e.g.
  Spanish weather service implementation of IDD
• Interdisciplinary from the outset
• over 2/3 of institutions have some uses outside
  meteorology (oceanography, hydrology,
  climatology, civil engineering, environmental
  science)
• 97 percent satisfied or very satisfied

6
Unidata User Impacts

• Over 11,000 college students per year use Unidata
  tools and data in classrooms and labs
• At least 500 graduate students use Unidata tools
• More than 400 faculty and research staff use
  Unidata tools
• Over 40,000 K-12 students involved through
  university programs
• Nearly 4,000 women/minority students
• Nearly 900 teacher training participants
• Informal education in excess of 400,000 hits at
  Unidata-based university web sites per day

7
Community Governance

• Policy Committee (3 meetings/year)
• Users Committee (2 meetings/year plus user
  workshops)
• MetApps Task Force (online use cases, testing,
  input for applications development)
• Advanced Technical Advisory Committee (ATAC) ad
  hoc meetings as needed

8
Principal Activities of the Unidata Program
Center

• Facilitating Data Access to a broad spectrum of
  observations forecasts (in near real time)
• Providing Tools to visualize, analyze, organize,
  receive, share data
• Supporting Faculty who use Unidata systems at
  colleges universities (most in the U.S.)
• Building and Advocating for a Community where
  data, tools, best practices in
  education/research are shared

9
IDD The Community in Action

• The Internet-based system by which universities
  acquire huge quantities of weather data in
  near-real time (i.e. ASAP) typifies Unidatas
  community orientation.
• The system has no data center -- all tasks are
  performed on the participants own (typically
  small) computers.
• Currently the most used advanced application on
  the Abilene network (2-3 in terms of packets and
  bytes transferred)

10
Types of Data

• Weather station observations
• Satellite imagery
• Output from supercomputer forecast models
• Radar observations
• Lightning strike data
• Profiler data
• Data from commercial aircraft instruments

11
Internet Data Distribution (IDD)

• Currently Unidatas primary method for supplying
  real-time meteorological data to over 160
  departments and research organizations
• Uses TCP/IP (i.e., Internet) for data transport
• Uses SMTP (i.e., email) for monitoring
• Delivers data products in seconds or minutes, and
  keeps hourly statistics on its own performance

12
Local Data Manager (LDM)

• Software for Internet Data Distribution (IDD)
• Handles multiple simultaneous data streams
• Compatible with most Unix environments
• Handles data reception and data relaying
• Manages local decoding and storage
• Freely available, without restriction
• Robust used 24 hours/day by some delivering tens
  of products per second

13
Sharing Data from Multiple Sources via
Cooperating LDMs
14
IDD Fanout from Source
15
IDD Roles for Providers

• Segmenting real-time data into natural products
• Typically over 100 bytes and less than 5 Mbytes
• Accompanied with IDs (WMO headers, e.g.) that
  recipients employ to control selection,
  processing, and routing (via pattern matches)
• Injecting products--with IDs--into a TCP/IP
  network, using LDM protocols ( software)
• Reporting injection statistics

16
IDD Roles for Recipients

• Running LDM software
• Configuring LDMs (using ID patterns), per local
  needs
• To acquire specified data from others
• To run event-driven processes (decoders, e.g.)
• To create local data holdings
• Relaying data to others (according to central
  guidelines)
• Reporting performance statistics
• Products received
• Latencies
• Managing local data holdings

17
IDD Coordinator Roles

• Defining relay configurations (i.e., the IDD
  topology)
• Processing statistics messages fromproviders and
  recipients
• Monitoring enhancing performance
• Troubleshooting
• Adjusting the topology
• Publishing overall performance stats (on a Web
  server, e.g.)

18
IDD Characteristics

• All participants run LDM software
• Participant's LDMs interact to disseminate
  real-time data
• LDMs produce local holdings of raw or decoded
  data and report statistics (volumes latencies)
  to Unidata
• Data are selected organized per local needs
• E.g., time series raw data for a selected station
• E.g., event-driven decoding into netCDF files
• Participants network limitations are
  accommodated by subscribing to subsets of the
  available data
• Any site may be a source or sink or both

19
Demonstrated IDD Capacity

• Ingesters (run at several sites) take in nearly
  1GB per hour
• Queues processing 40K products/hour (10
  products/second)
• Additional ingesters (such as for the lightning
  strike data feed at SUNY/Albany) can be added
  without limit
• IDD currently delivers 200 gigabytes per day
  aggregate
• IDD can grow indefinitely because no link carries
  traffic proportional to the number of receivers
• IDD is being extended in various ways
• More environmental data (e.g. surface ozone)
• NWS is testing experimental model outputs
• International sites receiving data
• NEXRAD data from 130 radars

20
Real-time Data for Local Weather, Flood,
Dispersion, Models
Satellite imagery
Weather station observations
High ResolutionLocal Model
Radar data assimilation
Demographic data for decision makers
Stream gauge measurements
21
THREDDSTHematic Realtime Environmental Data
Distributed Services

• NSDL Collection
• Funding via NSF Division of Undergraduate
  Education
• Integrates environmental data and tools into
  digital libraries (metadata technology)
• Infrastructure for embedding pointers to data and
  interactive analysis tools into online
  educational materials and scientific publications
• 20 major partners data providers, visualization
  tool builders, metadata experts.

22
Unidata Metadata and Distributed Data
Infrastructure

• Metadata services for THREDDS client/server
  software for dataset access by time, location,
  quantity
• Dataset aggregation server serving virtual
  datasets from collections of other datasets
• THREDDS catalogs XML-based structures providing
  application access to automatically generated
  metadata
• Remote access to data slices by client
  applications, implemented for the Unidata
  Integrated Data Viewer
• Challenges balance between simplicity for data
  providers and richness for data consumers,
  integration with web-services infrastructure

23
Typical Data Service Approaches
User Downloads Data for Analysis with Local Tools
User Browses Server Catalog and Analyzes Data
with Server Tools
Application specific protocols
Web protocols for discovery and visualization
Web protocols for discovery
Downloaded data converted to application specific
formats
.
Datasets on Server...
FTP
24
Traditional Data Handlingat a Unidata Site
Unidata user running McIDAS, GEMPAK, MetApps
Unidata user running McIDAS, GEMPAK, MetApps
Weather station observations
Application specific protocols
Decoders
Satellite imagery
Decoders
IDD
Local data decoded into application specific
formats
Radar data
Decoders
Decoders
Lightning, aircraft, GPSmet, etc.
Decoders
25
Client Server Data Access (User has to know
where the datasets reside)
DODS/ADDE clientE.g., McIDAS, MetApps/VisAD,
IDL, MatLab...
DODS, ADDE protocols
Satellite Images, e.g.
Satellite Images, e.g.
Satellite Images, e.g.
Weather Observations
Model Output
Satellite Imagery...
26
Thematic Data Servers Concept

• Some portion of the Unidata community pulls
  data
• To limit network loads reduce storage needs
• To simplify system care (less software less
  computer expertise)
• To examine utilize new or unfamiliar types of
  data
• Universities or centers with powerful systems
  themes
• Receive push data among themselves in
  real-time, via IDD
• Offer current retrospective data to the larger
  community
• Specialized along disciplines, instruments, time
  series, processing, etc.
• Direct, remote access via the DODS or ADDE
  protocols, thus providing efficient use of
  subsets also, access via FTP
• Facilitate data discovery
• Search or browse via DLESE (the Digital Library
  for Earth-System Education)
• Comprehensive contextual metadata usage
  metadata

27
Thematic Data Servers (combining IDD push with
several forms of pull and DL discovery)
User applications e.g., LAS, McIDAS,
MetApps/VisAD, IDL, MatLab...
DLESE Digital Library for Earth-System Education
Discovery
DODS, ADDE, FTP protocols
DLinterchangeprotocol
Satellite Images, e.g.
Satellite Images, e.g.
Satellite Images, e.g.
Hydrology Data, e.g.
Geophysical Data, e.g.
Satellite Imagery...
IDD
IDD
IDD
IDD
IDD
28
Key Components

• Real time delivery and updates (LDM/IDD)
• Remote access programming interfaces (ADDE and
  DODS)
• Discovery metadata
• Work with DLESE, GCMD and others on content
• Usage metadata (application accessible/usable)
  metadata
• Note the work of Lamont Doherty
• XML conversion tools
• Metadata exchange
• SDLIP and DIENST for accessing discovery metadata
• LDM/IDD for replicating information on servers

29
Publishable Inventories and CatalogsPICats

• XML form
• HTTP delivery
• Refer to DODS or ADDE datasets
• Accessible from browsers and applications
• Harvesting via digital libraries (e.g., DLESE)
• Enable virtual aggregations
• Can exist separate from data server
• Foundation of data web with references in
• Educational materials
• Scientific publications
• Search systems

30
Schematic of Server Components
Analysis and display applications
Exchange among servers
Population of central catalogs
Browsing tools, such as LAS
Clients
Metadata
Data
Protocols
ADDE
DODS
Catalog Server
Server
31
Targeted Initial Sites
32
Metadata Collaborations

• The Earth System Markup Language (ESML,
  University of Alabama-Huntsville)
• The DIstributed MEtadata System (DIMES, George
  Mason University)
• The aggregation data catalog that is part of the
  Distributed Oceanographic Data System (DODS,
  University of Rhode Island)
• GDLIP, General Digital Library Interchange
  Protocol (Alexandria Digital Library)
• Digital Library for Earth System Education
  (DLESE) and
• Web Mapping Testbed (OGC, Digital Earth).

33
Client Developer Collaborations

• VisAD infrastructure from SSEC (Bill Hibbard, the
  University of Wisconsin-Madison)
• LAS - Live Access Server (Steve Hankin, PMEL)
• WXWise applets (Tom Whittaker, the University of
  Wisconsin-Madison)
• The Virtual Exploratorium (the University of
  Illinois, West Chester State, DLESE, and NCAR,
  Don Middleton)
• EDMI (Earth Data Multimedia Instrument, Bruce
  Caron, New Media Studio) and
• WorldWatcher from Northwestern University (Mike
  Taber, the University of Northern Colorado).

34
Tools Are Central

• Unidata offers software developed by others
• Meteorological display and analysis tools from U
  of Wi, National Weather Service, others
• Distributed Oceanographic Data System (U of RI)
• Unidata develops software internally
• Widely used tools for managing scientific data
  e.g., netCDF, LDM, UDUNITS
• IDV (Java-based tools) for 2D/3D visualization
  and next-generation, collaborative data analysis
• Unidata supports software by offering training,
  consultation, fixes, and upgrades free of charge

35
1Km Radar Image
36
Integrated Data Visualization Client

• 3D radar reflectivity from NCAR server via DODS
  protocol
• Visible 1K satellite image from Wisconsin SSEC
  via ADDE protocol
• Balloon sounding temperature profile from local
  disk delivered automatically in real-time via IDD
• Different sources, protocols, resolutions,
  time-scales

37
National 1K NEXRAD Mosaic
38
(No Transcript)
39
(No Transcript)
40
For More Information

• Unidata http//my.unidata.ucar.edu
• Unidata LDM http//my.unidata.ucar.edu/software/l
  dm/index.php
• Unidata IDD http//my.unidata.ucar.edu/software/i
  dd/index.php
• GEMPAK http//my.unidata.ucar.edu/software/gempak
  /indx.php
• IDV http//my.unidata.ucar.edu/software/metapps/
  index.php
• McIDAS http//my.unidata.ucar.edu/software/mcidas
  /index.php
• THREDDS http//www.unidata.ucar.edu/projects/THRE
  DDS/Overview.htm