SURA IT Committee Meeting March 22, 2005

1
SURA IT Committee MeetingMarch 22, 2005
SCOOP Status

• Sara J. Graves, Ph.D.
• Director, Information Technology and Systems
  Center
• University Professor, Computer Science Department
• University of Alabama in Huntsville
• Director, Information Technology Research Center
• National Space Science and Technology Center
• 256-824-6064
• sgraves_at_itsc.uah.edu

2

• Whereas, the Southeastern Universities Research
  Association has proposed the creation of an
  open-access network of distributed sensors,
  linked via an ultra-fast network to
  state-of-the-art computing systems that track and
  model the southeastern coastal zone in real time
  and provide components of a more comprehensive
  coastal security infrastructure known as
  Southeastern Coastal Ocean Observing Program
  (SCOOP) now, therefore, be it
• Resolved, That the Southern Governors
  Association supports SURAs Southeastern Coastal
  Ocean Observing Program to bring more effective
  protection of life
• and property to the increasingly developed
  coastal zone, to offer a vehicle for bringing
• the extensive and widely dispersed intellectual
  talent of the ocean sciences community to
• address program of homeland security via an
  integrated and spatially distributed program,
• and to aid in addressing the ecological and
  environmental concerns endangering health
• and safety of inhabitants and marine resources.

3
SURAs Southeastern Coastal Ocean Observing
Program (SCOOP) will facilitate the assimilation
of observational data into community models and
provide a distributed data ingestion and support
grid with broad band connectivity. This is
expected to become a coastal counterpart to the
Global Ocean Data Assimilation Experiment (GODAE)
with emphasis on the southeast.
4
Board of Trustees Meeting Nov 2002

• Data fusion is critical
• Modeled and observed fields must have equal
  representation
• Use GODAE (Global Data Assimilation Experiment)
  as a guide for CODAE (Coastal Ocean Data
  Assimilation Experiment)
• SURA is a strong brand (we should use it)
• Focused sub-regional efforts with specified
  deliverables which would be new and exciting
• Broad SURA effort targeted on building a culture
  supporting region-wide collaboration in shared
  scientific goals

5
Integrated Ocean Observing System (IOOS)

• Serves national needs for
• Detecting and forecasting oceanic components of
  climate variability
• Facilitating safe and efficient marine operations
  
• Ensuring national security
• Managing resources for sustainable use
• Preserving and restoring healthy marine
  ecosystems
• Mitigating natural hazards
• Ensuring public health

6
National Federation of Regional Systems

• National Backbone
• Satellite remote sensing
• In situ sensing
• reference sentinel
• station-network
• Link to global ocean
• component
• Data standards
• exchange protocols

• Regional Systems
• Regional priorities
• Effects of climate change
• Effects of land-based
• sources
• ? Resolution,
• ? Variables

7
Overarching Principles for Coastal Observing
Programs

• 1. A national coastal observing program will
  necessarily consist of regional and sub-regional
  components.
• National, regional and sub-regional observing
  systems must consist of three interconnected
  aspects (i) spatially distributed sensor arrays
  (ii) data management and dissemination hubs and
  (iii) nowcasting and forecasting models that are
  fused with assimilated observational data.
• 3. The creation and long-term viability of
  nested integrated and sustained coastal observing
  systems will depend on a high level of
  interagency coordination.

8
SCOOP Vision Statement
The SURA Coastal Ocean Observing and Prediction
(SCOOP) program is an initiative to create a
open-access, distributed national laboratory for
scientific research and coastal operations.
SCOOP is designed to complement the efforts of
both Ocean.US - the organization responsible for
implementing the national Integrated Ocean
Observing System (IOOS)- and the coastal
component of NSFs Ocean Research Interactive
Observatory Networks (ORION) project. The SCOOP
emphasis is on interoperability in order to
create a real-time observations system for both
monitoring and prediction. Through SURA
Universities, SCOOP will provide the expertise
and IT infrastructure to integrate observing
systems that currently exist, and incorporate
emerging systems. This will promote the
effective and rapid fusion of observed data with
numerical models, and facilitate the rapid
dissemination of information to operational,
scientific, and public and private users.
9
Overarching Goals for SCOOP

• System of Systems
• Ocean Observing IOOS OOI ORION
• Coastal Ocean Component of the Global Earth
  Observing System of Systems (GEOSS)
• Components (i) Sensor arrays, (ii) Data
  management communication, (iii) Predictive
  models
• Distributed National Lab for Research
  Applications
• IT Glue...Bricks Mortar
• Research to Operations
• Academic Federal Agency Industry partnership
• IT Enabling Big Science
• Environmental prediction
• Standards enable innovation
• Interoperable community solving the really big
  problems

10
Planned Capabilities

• Validate accurate and timely short and long-term
  predictions
• Simultaneous measurements of winds, waves,
  currents, water density, nutrients, water
  quality, biological indices, and fish stocks
  under all conditions
• Focus on storm surge, wind waves, and surface
  currents, with special attention to predicting
  and visualizing phenomena that cause damage and
  inundation of coastal regions during severe
  storms, hurricanes and possibly tsunamis
• Bridge the gap between scientific research and
  coastal operations

11
SCOOP Science Goals

• Assess and predict the coastal response to
  extreme atmospheric events focus on storm
  surge, flooding waves
• Modular modeling tools for regional issues (wave
  coupling, sediment suspension, etc.)
• Standardized interfaces for data and (coupled)
  model interoperability
• Ensemble prediction forecasts based on many
  independent models runs

12
SCOOP Research Goals

• Measure, understand and predict environmental
  conditions
• Provide RD support for operational agencies
  including NOAA, the U.S. Navy, and others
• Include outreach and education components that
  assure relevance of their observing activities

13
SCOOP Objectives

• Develop and deploy standards and protocols for
  data management, exchange, translation and
  transport
• Implementation of existing standards and
  protocols (e.g. FGDC, OGC, web services, etc.)
• Application of Grid Technologies
• Deployment of the communications infrastructure
  to link ocean sensors operating in extreme
  environmental conditions to people who need
  timely information
• Cultivation of industry partners

14
Coordination is Key

• Ocean.US - National Office for Integrated and
  Sustained Ocean Observations coordinates
  development of an operational, integrated and
  sustained Ocean Observing System (created by
  NOPP) http//www.ocean.us/
• Integrated Ocean Observation System (IOOS) a
  national effort to create an Integrated Ocean
  Observing System http//www.openioos.org/
• National Oceanographic Partnership Program (NOPP)
  15 federal agencies providing leadership and
  coordination of national research and education
  programs http//www.nopp.org/
• National Federation of Regional Associations
  provide a framework for orchestrating regional
  collaborations http//www.usnfra.org/
• NSF Ocean Research Interactive Observatory
  Networks (ORION) an emerging network of
  science-driven ocean observing systems
  http//www.orionprogram.org/default.html

15
Interoperability is Key

• Ocean.US Data Management and Communications
  (DMAC) Plan provides the framework for
  interoperability http//dmac.ocean.us/dacsc/imp_pl
  an.jsp
• Open Geographic Information Systems (GIS)
  Consortium (OGC) an open consortium of industry,
  government, and academia developing interface
  specifications to support interoperability
  http//www.opengis.org
• Marine Metadata Interoperability a community
  effort to make marine metadata easier to find,
  access and use http//www.marinemetadata.org/

16
Interoperability Demonstration
NOAA and ONR grant recipients collaboration
www.openioos.org
17
SCOOP System Development

• Funding provided by ONR, NOAA
• 2004 List of SCOOP Partners

• Consortium for Oceanographic Research and
  Education
• Gulf of Maine Ocean Observation System (GoMOOS)
• Louisiana State University, Center for
  Computation Technology
• Louisiana State University, Coastal Studies
  Institute
• Southeast Atlantic Coastal Ocean Observing
  System (SEACOOS)
• Southeast Coastal Ocean Observations Regional
  Association (SECOORA)
• Texas AM University Gulf Coast Ocean
  Observing System (GCOOS)
• University of Alabama in Huntsville
• University of Delaware (Mid-Atlantic Regional
  Association (MACOORA)
• University of Florida
• University of Miami, Center for Southeastern
  Tropical Advanced Remote Sensing
• University of North Carolina
• Virginia Institute of Marine Science

18
SCOOP Program Elements

• Data Standards
• Metadata standards compliant with existing and
  emerging standards
• Standard data models to facilitate aggregation
• Data Grid
• OGC Web services for distributed maps and data
• Augmenting with new data, e.g., surface currents
• Model Grid
• Storm surge wave prediction
• Modular, standardized prediction system

19
SCOOP Data Architecture (high level)
Transport Mediums
HTTP HTML
NOAA
SCOOP Modeling Partners
Regional Data Provider 1
TBD???
Regional Association Data Center (Archive)
Other Regional Association Data Centers
Regional Data Provider 2
LDM???
Web Browsers
HTTP HTML
Regional Data Provider N
OGC Protocols
GIS Clients
NDBC MODEM
NDBC
SCOOP Modeling Partners
TBD???
20
SCOOP Prediction SystemAll Versions

1. Standard naming conventions Adopt existing
   community standards where appropriate (e.g., CF
   or NCEP) and add our own conventions only when
   necessary.
2. Mechanisms for tracking metadata, e.g.,
   provenance, forcing, source of OBCs, forcing used
   to create OBCs, etc.
3. Portals entry point for access to models
   model output. Deals with authentication
   authorization.

21
SCOOP Prediction System, Version 1.0

• Modular wind forcing
• Modular embedded regional models
• Coupled models for existing groups
• Using existing computational resources
• Verification real time model-data comparisons
• Model-GIS interface OGC Web services
• Web mapping with roads, etc.
• Web mapping with time sequences (WMS)
• Standardized time-series verification
• Openioos.org for displaying results
• Other?

22
SCOOP Operational Prediction System Version 1.0
Standardize Transport/Encapsulation
XML, FTP, LDM, OPeNDAP?
Large Scale Response
Forcing
Operational Wave Predictions (BIO/GoMOOS)
Regional Response
NOAA/NCEP (ETA)
Regional Model Center 1
Operational Tide/Surge Predictions (SABLAM)
NOAA/NCEP/UNC? (EDAS) Archive
Regional Model Center 2
Enhanced Winds (Miami)
Coupled Wave-Surge Predictions (Miami)
Standardize model interfaces
23
SCOOP Verification Visualization
Prediction System
OGC, RSS?
Modeling Center 1 (Regional or otherwise)
Information Providers
Regional Web Server 1
Modeling Center 2 (ditto)
Data System
Regional Web Server 2
Regional Data Center 1
Regional Data Center 2
Standardize model-GIS interfaces
Standardize verification tools data
24
Prediction System Task Elements for Version 1.0
Task Lead Partner Data standards TAMU Data
transport UAH Data translation
mgmt UAH Coupled modeling Miami Nested
Modeling VIMS Customized configuration TAMU Vis
ualization Services LSU Verification
validation Miami Computing storage
resources LSU Security TAMU Grid management
middleware LSU Web Mapping Demonstration GoMOOS
25
SCOOP Data ArchitectureHigh level Services
Users, Modeling Partners, other Data Centers,
etc.
Regional Data Providers
Observation Data
Regional Association Data Centers
GeoSpatial OneStop / FGDC Clearinghouse
User Interface
Data Provider
data
Data and Metadata
Metadata Services
Model
SCOOP Catalog
Data Translation Services
Archive/Repository Broker
data
Metadata only
data
Modeler / Data Provider
Data Access Services
Data Access Services
Data Provider
data
Model or Application
26
SCOOP Data Architecture SpecificsData
Acquisition example technologies to support
dynamic transport and metadata cataloging
Data Transport Metadata Cataloging
Regional Association Data Centers
Regional Data Providers
Observation Data
Data Provider
e.g., XML, Metadata Harvest
data
Metadata and Data
Metadata Services
Model
SCOOP Catalog
e.g., LDM
data
Metadata only
Archive/Repository Broker
Modeler / Data Provider
data
LDM
Data Access Services
Data Access Services
Data Provider
data
27
SCOOP Data Architecture SpecificsData Discovery
Access example technologies to support
dynamic transport, analysis and visualization
Users, Modeling Partners, other Data Centers,
etc.
GeoSpatial OneStop / FGDC Clearinghouse
Data Discovery
User Interface
XML
Z39.50
SOAP
Metadata Query Services
IOOS Interoperability Demo
SCOOP Catalog
Archive/Repository Broker
data
Data Access
HTTP
data
Modeler / Data Provider
Model or Application
ESML
FTP OPeNDAP OGC
FTP OPeNDAP OGC
OGC WMS
Data Translation Services
FTP
Regional Association Data Centers
Regional Data Providers
28
SCOOP Information ArchitectureExample metadata
exchange technologies to support Data Discovery
Metadata Population
GeoSpatial OneStop
User
SCORE
Z39.50
Manual Updates
FGDC Records
XML
Ingest Svcs
Query Svcs
Metadata Harvest
Metadata Services
Data Provider
XML ?
data
IOOS Interoperability Demo
SCOOP Catalog
OGC
HTTP
Metadata Harvest
WMS data list metadata
Get Capabilities
Data Discovery
SOAP
Local Metadata
SOAP
SCOOP Data Dictionary
SCOOP Interactive Search U/I
Metadata Harvest
SOAP
Model or Application
data
Automated Data Discovery
SOAP
Modeler / Data Provider
Regional Association Data/Service Centers
Regional Data Providers
Users, Modeling Partners, other Data Centers,
etc.
29
SCORE Accomplishments Plans

• SCORE is the catalogs and services infrastructure
  for SCOOP data management
• Data Model Survey provided initial snapshot of
  partners data (observations and model results)
• Developed database schema for SCORE to support
• Strawman SCOOP Catalog requesting input on
  improved capabilities
• IOOS demo working with GoMOOS team to integrate
  catalog with demo
• FGDC Clearinghouse to support Geospatial
  One-Stop plan to create FGDC metadata records
  from SCORE
• Issues
• What data management functionality is needed
  within SCOOP?
• Metadata services for data collections, data
  files/streams, general model information,
  information on specific model runs,
• How to coordinate metadata and data management
  across sites?
• How to automate population of SCORE?

30
Science Goals for Version 2.0

• Environmental Prediction
• Prediction systems fuse models observations
• Nonlinear dynamics limits predictability
  Lorentzs seagull
• Probability and statistics ensemble modeling
• Hurricane Surge Waves
• Biggest uncertainty in the winds
• Ensemble of winds different models or different
  simulations
• New paradigm new metrics for skill assessment
• Research to Operations
• Improving upon SLOSH a good idea 30 years ago
• GIS compatibility enabling application
  visualization
• OpenIOOS.org is the high visibility front end

31
Version 2.0
Wind Forcing
Wave and/or Surge Models
Result Dissemination
Select region and time range
Archive
ADCirc
Verification
ElCirc
Transform and transport data
or
Visualization
WAM/SWAN
Regional Archives
OpenIOOS
or
Ensemble of models run across distributed
resources
Synthetic Wind Ensemble
Analysis, storage and cataloguing of output data
Ensemble wind fields from varied and distributed
sources
32
SCOOP Data-to-Model (D2M) RealtimeTransport and
Translation (Nested Model) Scenario
NCEP (NAM)Wind Forecasts
Atmospheric Models
Regional Oceanic/Coastal Models
TAMU
LDM-push
(FTP-pull)
UNC
POC Gerry Creager
(1)
MM5
POC Rick Luettich, Brian Blanton

• Translation Services
• subset
• subsample
• re-format
• re-grid

Translated Winds and fluxes
Alternate
LDM-push
ADCIRC
WRF (future)
LDM-push
(1a)
High-Res Wind Forecasts
(2)
LDM-push
ESML
Water levels
D2M Node
(3)
POC Matt Smith, Ken Keiser (UAH)

• Translation Services
• subset
• subsample
• re-format
• re-grid

1. Atmospheric Model products are translated
   through D2M to the form requested by the client
   model. Currently, using ftp-pull, all NAM grids
   0-84h for the 4 runs (00, 06, 12, 18 UTC) of
   AWIP12 and AWIP32 are sent to a D2M node and
   translated.
2. Via LDM, UNC, TAMU, UF have access to the raw
   and translated model data.
3. Partners use translated ob/model data in their
   models. Then push their results to a D2M node.
   Currently, ADCIRC output files (text and netCDF)
   are being pushed to a D2M node (for translation)
   and other modeling partners via LDM.
4. Resulting translated data products area pushed to
   a client models site and made available for
   other transport vehicles (FTP, OPeNDAP, OGC, etc)
   for use in retrospective studies and other
   applications. Likewise the output of other
   models can be processed through D2M for
   translation steps requested by other client
   models.

ESML
LDM, OPeNDAP, FTP Push/pull
D2M Node
TranslatedWater levels
LDM-push
(4)
VIMS
TAMU, UF, Others?
ELCIRC
Model X
POC Harry Wang
Localized Models, Users and Archives
33
Data Management Goals

• Version 1
• Provided a high-level data catalog for SCOOP data
  discovery, providing descriptions of partner data
  holdings and pointers to partner data access
  points (web, ftp, OPeNDAP, etc.)
• Based initial catalog on Data Model Survey
  results
• Coordinated with Data Transport (Task 2) to
  develop initial LDM network to exchange data in
  near real time among SCOOP partners.
• Coordinated with Data Standards (Task 1) on
  development of metadata keywords for SCOOP
• Version 2
• Expand SCOOP data discovery capabilities based on
  evolving data management practices of SCOOP
  partners.
• Support IOOS Demo
• Field an FGDC Clearinghouse node for SCOOP
• Monitor Marine Metadata Interoperability
  activities and their potential interaction with
  SCOOP
• Assist SCOOP partners in developing standard
  metadata to describe their data collections
• Continue coordination with all partners on data
  management issues

34
Distributed Data Integration
Merged data product for on-demand visualization
Countries
Cyclone Events
AMSU-A Channel 01
MCS Events
Coastlines
Knowledge Base
AMSU-A
ITSC
GLOBE
AMSU-A data overlaid with MCS and Cyclone events,
merged with world boundaries from GLOBE.
35
Heterogeneity Leads to Data Usability Problems

• Science Data Characteristics
• Many different formats, types and structures (18
  and counting for atmospheric science alone!)
• Different states of processing (raw, calibrated,
  derived, modeled or interpreted)
• Enormous volumes

36
Interoperability Accessing Heterogeneous Data
The Problem
The Solution
DATA FORMAT 3
DATA FORMAT 1
DATA FORMAT 2
DATA FORMAT 1
DATA FORMAT 2
DATA FORMAT 3
ESML FILE
ESML FILE
ESML FILE
FORMAT CONVERTER
ESML LIBRARY
READER 1
READER 2
APPLICATION
APPLICATION

• One approach Enforce a standard data format,
  but
• Difficult to implement and enforce
• Cant anticipate all needs
• Some data cant be modeled or is lost in
  translation
• Converting legacy data is costly

• A better approach Interchange Technologies
• Earth Science Markup Language

37
What is ESML?

• It is a specialized markup language for Earth
  Science metadata based on XML - NOT another data
  format.
• It is a machine-readable and -interpretable
  representation of the structure, semantics and
  content of any data file, regardless of data
  format
• ESML description files contain external metadata
  that can be generated by either data producer or
  data consumer (at collection, data set, and/or
  granule level)
• ESML provides the benefits of a standard,
  self-describing data format (like HDF, HDF-EOS,
  netCDF, geoTIFF, ) without the cost of data
  conversion
• ESML is the basis for core Interchange Technology
  that allows data/application interoperability
• ESML complements and extends data catalogs such
  as FGDC and GCMD by providing the use/access
  information those directories lack.
• http//esml.itsc.uah.edu

38
ESML IN ACTION Ingest surface skin temperature
data in Numerical Models

• Purpose
• Use ESML to incorporate observational data into
  the numerical models for simulation
• Skin temperatures come in a variety of data
  formats
• GOES McIDAS
• Reanalysis Data - GRIB
• MM5 Model - Binary
• AVHRR HDF
• MODIS - EOS-HDF

Reanalysis GRIB files
MM5
GOES
ESML file
ESML file
ESML file

• Scientists can
• Select remote files across the network
• Select different observational data to increase
  the model prediction accuracy

http//vortex.nsstc.uah.edu/sud/web/default.htm