The Earth System Grid II

1
The Earth System Grid II
UCRL-PRES-148698

• Presented by
• Dean N. Williams
• LLNL/PCMDI
• Representing the work of the
• Earth System Grid Team

2
Overview

• I. The Science
• II. ESG Pervasive Grid Access to Distributed
  Resources
• III. Software Leveraging Partnerships
• IV. Portals Space-time Superglue to Put it
  All Together
• V. What's Next? Interfacing with other Grids?

3
Part I

• The Science

4
Fingerprint studies using vertical profiles of
temperature change
Model Changes CO2 Only
Height (kilometers)
Pressure (hPa)
Observed Changes
Temperature changes in oC
Height (kilometers)
Pressure (hPa)
5
Fingerprint studies using vertical profiles of
temperature change
Model Changes CO2 Aerosols Stratospheric
Ozone
Pressure (hPa)
Height (kilometers)
Observed Changes
Height (kilometers)
Temperature changes in oC
Pressure (hPa)
6
Can natural variability explain recent tropopause
height changes?
7
Multidisciplinary Scientific Communities

• Climate study is fundamentally multidisciplinary.
  As we strive to understand its complexity,
  researchers from different fields and different
  locations must become engaged in large
  multinational teams to tackle these Grand
  Challenge problems
• Need a software infrastructure to support this
  multidisciplinary Virtual Organization (VO)
• Community code (open/modular/shared simulation
  codes)
• Tools that support collaboration and data sharing
• Location-independent equal-access to shared
  resources (data, visualization, supercomputers,
  experiments, whiteboard, etc..)

8
Part II

• ESG Pervasive Grid Access to Distributed
  Resources

9
Earth System Grid (ESG) Overview

• Funded by the Scientific Discovery through
  Advanced Computing (SciDAC), this program seeks a
  new paradigm in the climate change community
  evolving from centralized data sharing to
  distributed data-sharing.
• Enabling geographically distributed teams of
  researchers to effectively and rapidly acquire
  knowledge and understanding of massive amounts of
  climate data holdings.
• Multiple interfaces to ESG will allow researchers
  to focus on science and not issues with data
  receipt, format, and data set manipulation.

10
Role of The Earth System Grid (ESG)?

• Allow access to retrospective climate data (input
  and output) needed to enable a feedback mechanism
  to tie researchers directly back to quality
  control and diagnostics of models.
• Allow researchers access to format independent
  climate and observational data for case-study
  training.
• In the U.S., climate simulation can be viewed as
  a systems problem, allow a team of multi-agencies
  and institutions working together in
  collaboration (i.e., Virtual Organization (VO))

11
ESG U.S. Collaborations Development
ANL Computational grids, grid-based
applications
LBNL Climate storage facility
LLNL Model diagnostics inter-comparison
USC/ISI Computational grids, grid-based
applications
ORNL Climate storage computational resources
LANL Next generation coupled models computing
NCAR Climate change predication and scenarios
12
ESG Requirements Priority Matrix
13
ESG U.S. Department of Energy (DOE) Next
Generation Internet (NGI) Project

• ESG-I (past)
• Focused on developing techniques for the
  high-speed data movement between sites and users
  (e.g., the secure highly efficient File Transfer
  service, called GridFTP, developed by ANL (i.e.,
  Globus))
• Developed replica catalogs for keeping track of
  data locations
• Developed request manages for coordinating
  multiple transfers
• Developed a grid-enabled version of LLNLs data
  analysis package

14
ESG ESG-I Architecture
15
ESG ESG-I Team Presented their work at
Supercomputing 2001
RAID
CLOUD
LDAP/Sever Metadata Catalog LLNL
TERRAIN
U V
LDAP/Sever Metadata Catalog LBNL
parallel disk system
16
ESG DOE SciDAC Project

• ESG-II (present)
• Building upon the substantial work of ESG-I
• Grid-wide services supporting authentication,
  authorization, data discovery, and user specified
  analysis
• Metadata services supporting remote data
  browsing, querying, accessing, displaying, etc.
• Filtering services performing intelligent model
  specific analysis before delivering the results
  to the user
• Integrate next-generation data analysis and
  visualization applications (such as ongoing work
  at LLNL and NCAR), web-based data portals and
  other thin clients supporting the Distributed
  Oceanographic Data System (DODS), and
  collaborative problem-solving environments.

17
ESG ESG-II Architecture
18
ESG Metadata Services
19
ESG Collaboration Network
Grid and Network
Infrastructure
20
Part III

• Software Leveraging Partnerships
• Community Code

21
Leveraging Software

• PMEL
• NOAAs Pacific Marine Environmental Laboratory
  Live Access Server (LAS) and Ferret.
• COLA
• The Center for Ocean-Land-Atmospheric Studies
  GrADS-DODS Server (GDS) and GrADS.
• UCAR/Unidata and DODS
• Unidata THematic Real-time Environmental
  Distributed Data Services (THREDDS)
• URI/MIT Distributed Oceanographic Data System
  (DODS)

22
Leveraging Software (cont.)

• NCAR
• NCL Partnering with NCARs Community Data Portal
  (CDP) project
• LLNL
• Climate Data Analysis Tools (CDAT) LAS-CDAT
• NASA Global Change Master Directory
• Through the GCMD, NASA is a partner in the
  distributed data sharing framework in and is
  developing a XML database search engine to be
  hosted at PMEL.

23
Part IV

• Portals Space-time Superglue to make these
  components work together for the Virtual
  Organization

24
Portal Client Layers

• Thin Client Slow interaction, but you know its
  going to work!
• Delivery HTML to any web-browser
• Users No time investment
• Slender Client Faster interaction, but primary
  work on remote server.
• Delivery Modules, signed applications, tiny
  binaries
• Users Some time investment in acquiring modules
• Fat Clients Portal merely a data broker between
  distributed resources and your helper
  application.
• Delivery Standalone applications of any sort
• Users More significant time investment to
  install helper application (i.e., CDAT, Ferret,
  NCL)

25
CDAT Example of an ESG Script Access

• The next-generation language, Python, is used to
  access the Earth System Grid (ESG) at LLNL

26
CDAT Example of an ESG GUI Client Access
27
LAS/CDAT Example of a Web-based Data Portal

• Technology Web Based (end user requirements)
• LAS, DODS, ESG (i.e., Globus), Ferret, CDAT
• Portal should hide/simplify the Grid for users
• Single access, locates resources, builds/finds
  executables, central management of parameter
  files/job output, submit jobs to local batch
  queues, tracks active jobs. Submission/management
  of distributed runs
• Accesses the ESG Grid Testbed

28
The Big Picture
Any Viz Client
HTTP
Remote Viz data
XML
Vtk
netCDF
Remote Viz data
29
Part V

• What's Next? Interfacing with other Grids?

30
Building An Integrated Grid Architecture to
Enable Dynamic Grid Applications
U.S. Users

• Large Grid Projects in collaboration with ESG
• e-Science ClimatePrediction .com, NERC DataGrid
• NOAA Operational Model Archive and Distribution
  System (NOMADS)
• Committee on Earth Observation Satellites
  (CEOS)
• Grid Application Toolkit for application
  developers and infrastructure (APIs/Tools)
• Look at www.globus.org for details

U.K. Users
Climate Community
University Users
Community Outreach
CDAT Users
Ferret Users
Commercial Users
Sponsors
Networks
U.K. NERC DataGrid
ESG Grid
CEOS Grid
Other Grids
31
Concluding Statements

• ESG is a highly collaborative effort and will
  allow users to quickly access data storage
  facilities storing petabytes of raw or processed
  data in an application independent manner.
• Payoffs of this distributed collaborative
  infrastructure, would include
• Distributed data-sharing
• Simplified data discovery of climate data
• Large-scale climate data processing and analysis
• Increased collaboration among climate research
  scientists
• Aid in climate assessments and estimates of
  future climate variability and trends
• For more information on ESG, visit our websites
  at
• http//www.earthsystemgrid.org