Grids and Grid Technologies

1
Grids and Grid Technologies

• Ian Foster
• Mathematics and Computer Science Division
• Argonne National Laboratory
• and
• Department of Computer Science
• The University of Chicago
• Member, Executive Committee, Global Grid Forum
• http//www.mcs.anl.gov/foster

2
Overview

• The Grid problem and concept
• Review of major Grid projects
• State of the art in Grid technologies
• Global Grid Forum
• Opportunities for integration

3
Grid Concept

• Enable communities (virtual organizations)
  to share geographically distributed resources as
  they pursue common goalsin the absence of
  central control, omniscience, trust relationships

4
Universal Nature of the Grid Problem

• Original motivation, and support, from high-end
  science and engineering
• But sharing fundamental in many settings
• Application Service Providers, Storage Service
  Providers, etc. Peer-to-peer computing
  Distributed computing B2B
• Not adequately addressed by existing tech
• Sharing at a deep level, across broad ranges of
  resources and in a general way
• E.g., user provides ASP with controlled access to
  their data on an SSP how??

5
A Little History(U.S. Perspective)

• Early 90s
• Gigabit testbeds, metacomputing
• Mid to late 90s
• Early experiments (e.g., I-WAY), software
  projects (e.g., Globus), application experiments
• 2001
• Major application communities emerging
• Major infrastructure deployments are underway
• Rich technology base has been constructed
• Global Grid Forum gt1000 people on mailing lists,
  192 orgs at last meeting, 28 countries

6
Major Application Communities are Emerging

• Intellectual buy-in, commitment
• Earthquake engineering NEESgrid
• Exp. Physics, etc. GriPhyN, PPDG
• Simulation Earth System Grid, Astrophysical Sim.
  Collaboratory
• Emerging, e.g.
• Natl. Ecological Obs. Network
• Bioinformatics Grids
• Fusion collaboratory
• National Virtual Observatory

7
Grid Communities ApplicationsData Grids for
High Energy Physics
Image courtesy Harvey Newman, Caltech
8
Grid Communities and ApplicationsNetwork for
Earthquake Eng. Simulation

• NEESgrid national infrastructure to couple
  earthquake engineers with experimental
  facilities, databases, computers, each other
• On-demand access to experiments, data streams,
  computing, archives, collaboration

NEESgrid Argonne, Michigan, NCSA, UIUC, USC
9
Major Infrastructure Deployments are Underway

• Projects well under way
• NSF National Technology Grid
• NASA Information Power Grid
• DOE ASCI DISCOM Grid
• Starting/proposed
• DOE Science Grid
• NSF Dist. Terascale Facility
• DOD MOD Grid
• GRIDS Center National Middleware Infrastructure

10
A Rich Technology Basehas been Constructed

• 6 years of RD have produced a substantial code
  base based on open architecture principles esp.
  the Globus Toolkit, including
• Grid Security Infrastructure
• Resource directory and discovery services
• Secure remote resource access
• Data Grid protocols, services, and tools
• Essentially all projects have adopted this as a
  common suite of protocols services
• Enabling wide range of higher-level services

11
Large U.S. Grid Projects

• DOE ASCI DISCOM
• DOE Particle Physics Data Grid
• DOE Earth Systems Grid
• DOE Science Grid
• DOE Fusion Collaboratory
• NASA Information Power Grid
• NSF National Technology Grid
• NSF Network for Earthquake Eng Simulation
• NSF Grid Application Development Software
• NSF Grid Physics Network
• smaller computer science research portals,
  security, resource management, discovery,
  monitoring, programming models, etc., etc.

And in Europe EU DataGrid EuroGrid
GridLab AstroGrid GridPP Etc.
12
Grid Challenges and Technologies

• New applications enabled by the coordinated use
  of geographically distributed resources
• E.g., distributed collaboration, data access and
  analysis, distributed computing
• Persistent infrastructure for large-scale
  resource sharing systems
• E.g., authentication, authorization, policy
  protocols for resource discovery/access, etc.
• Developing programs programming
• Abstractions, tools

13
Layered Grid Architecture(By Analogy to Internet
Architecture)
14
Grid Services ArchitectureConnectivity Layer
Protocols Services

• Communication
• Internet protocols IP, DNS, routing, etc.
• Security Grid Security Infrastructure (GSI)
• Uniform authentication authorization mechanisms
  in multi-institutional setting
• Standards SSL/TLS, X.509 CA, GSS-API
• Extensions for single sign-on, delegation
• Credential management Login, logout, etc.
  smartcards online credential repositories for
  Web portal login and delegation K5cert for
  automatic X.509 certificate creation

GSI www.globus.org
15
GSI Working Group Documents(GGF -gt IETF)

• Grid Security Infrastructure (GSI) Roadmap
• Informational draft overview of working group
  activities and documents
• Grid Security Protocols Syntax
• X.509 Proxy Certificates
• X.509 Proxy Delegation Protocol
• The GSI GSS-API Mechanism
• Grid Security APIs
• GSS-API Extensions for the Grid
• GSI Shell API

16
Current and Future Work

• Ease of use
• CA operation, credential mgt, account mgt
• Authorization
• Policy languages, community authorization
• Protection (despite compromised resources)
• Restricted delegation, smartcards
• Flexible communication support
• GSS-API extensions
• Independent Data Units (UDP, IP multicast)
• Apply GSI to new areas
• Personal Area Networks, PDAs, wireless, etc.

17
Grid Services ArchitectureResource Layer
Protocols Services

• Grid Resource Access Mgmt (GRAM)
• Remote allocation, reservation, monitoring,
  control of compute resources
• GridFTP protocol (FTP extensions)
• High-performance data access transport
• Grid Information Service protocols
• Registration, information access/monitoring
• Network reservation, monitoring, control
• All integrated with GSI authentication,
  authorization, policy, delegation

GRAM, GridFTP, GRIS www.globus.org
18
Resource Management Problem

• Enabling secure, controlled remote access to
  computational resources and management of remote
  computation
• Authentication and authorization
• Resource discovery characterization
• Reservation and allocation
• Computation monitoring and control
• Addressed by new protocols services
• GRAM protocol as a basic building block
• Resource brokering co-allocation services
• GSI for security, MDS for discovery

19
GRAM Protocol

• Simple HTTP-based RPC
• Job request
• Returns a job contact Opaque string that can
  be passed between clients, for access to job
• Job cancel
• Job status
• Job signal
• Event notification (callbacks) for state changes
• Pending, active, done, failed, suspended
• Moving to SOAP (more later)

20
GridFTP Basic Approach

• FTP is defined by several IETF RFCs
• Start with most commonly used subset
• Standard FTP get/put etc., 3rd-party transfer
• Implement standard but often unused features
• GSS binding, extended directory listing, simple
  restart
• Extend in various ways, while preserving
  interoperability with existing servers
• Parameter set/negotiate, parallel transfers
  (multiple TCP streams), striped transfers
  (multiple hosts), partial file transfers,
  automatic manual TCP buffer setting, progress
  monitoring, extended restart (via plug-ins)

21
Grid Information Service

• Provide access to static and dynamic information
  regarding system components
• Large numbers of sensors, in resources,
  services, applications, etc.
• A basis for configuration and adaptation in
  heterogeneous, dynamic environments
• Requirements and characteristics
• Uniform, flexible access to information
• Scalable, efficient access to dynamic data
• Access to multiple information sources
• Decentralized maintenance

22
The GIS Problem Many Information Sources, Many
Views
23
Globus MDSRegistration and Inquiry Protocols

• Registration (GRRP) and inquiry (GRIP) protocols
• Both currently based on LDAP protocols models
• Support creation of aggregate directories
  providing application-specific views of resources
  sets

24
Grid Services ArchitectureCollective Layer
Protocols Services

• Index servers aka metadirectory services
• Custom views on dynamic resource collections
  assembled by a community
• Resource brokers (e.g., Condor Matchmaker)
• Resource discovery and allocation
• Replica catalogs
• Co-reservation, co-allocation (choreography)
• Etc., etc.

Metadirectory www.globus.org Condor
www.cs.wisc.edu/condor
25
Grid Protocols in Action

• Compute server
• GRAM reservation, job submit/monitor/control
• MDS discovery/monitoring of availability
• GridFTP staging data in/out
• Storage server
• GRAM disk space and bandwidth reservations
• MDS discovery of available space, utilization
• GridFTP client-server, server-to-server
  transfers
• Bandwidth broker
• GRAM bandwidth reservation and allocation
• MDS discovery/monitoring of availability
• Online instrument
• GRAM reservation MDS discover use and features
• GridFTP for transferring data from the device

26
The Programming Problem

• Protocols and APIs are necessary but not
  sufficient for application development
• Programming needs abstractions and tools that
  reduce complexity, enable reuse, facilitate
  difficult and/or common tasks
• Presumably can occur at multiple levels
• Low-level programming models and tools
• Higher-level domain-specific libraries
• Workflow, GUI tools
• No reason to expect one right answer

27
Examples ofProgramming Technologies

• MPICH-G2 Grid-enabled message passing
• CoG Kits, GridPort Portal construction, based on
  N-tier architectures
• GDMP, Data Grid Tools, SRB replica management,
  collection management
• Condor-G simple workflow management
• Legion object models for Grid computing
• Cactus Grid-aware numerical solver framework

28
Grid Forum Initial Motivation (1998)

• Diverse efforts building Grids
• 12 multi-institutional projects from most major
  US Federal Agencies (NASA, NSF, DOE, DOD, etc.)
• Several dozen Grid frameworks or components
• HypothesisSufficient common interest that
  coordination will be useful to
• Discuss common interests, problems, solutions
• Identify/define standards to promote code
  sharing/interoperability
• Promote Grid Technology
• No obvious fit with existing groups (e.g. IETF,
  ISOC, W3C, OMG, etc.)

29
Global Grid Forum History
1999
2000
1988
GF BOF (Orlando)
GGF-1
GF1 (San Jose NASA Ames)
GF2 (Chicago iCAIR)
eGrid and GF BOFs (Portland)
GF3 (San Diego SDSC)
Global GF 1 (Amsterdam)
eGrid1(Posnan PSNC)
GF4 (Redmond Microsoft)
Asia-Pacific GF Planning (Yokohama)
eGrid2 (Munich at Europar)
GF5 (Boston Sun)
Global GF BOF (Dallas)
www.gridforum.org Chair
Charlie Catlett, Argonne
30
Global Grid Forum 1 (March 2000)U.S. and
European GFs Merge

• 325 Participants
• Capped by facilities
• 60 late-registrants turned away
• 192 Organizations
• Previous high 110 at GF-5
• 28 Countries
• Previous high 11 at GF-5
• 85 Document Authors

Next meeting GGF2, Washington DC, July 2001
www.gridforum.org
31
Grid Forum Areas and Selected Groups

• Grid Information Services
• Modeling, Schema, JINI
• Scheduling and Resource Management
• Advance reservation, resource management, NPI
• Security
• Security Infrastructure, CA Policy
• Grid Performance
• Performance Monitoring Architecture
• Applications
• Various
• Architectures and Frameworks
• GridFTP, Data Arch., Protocol Arch., Economic
  Models
• Programming and Run Time Environments
• Computing Envs., Prog. Models, Collaborative Envs.

Working groups Research groups
32
Relationships

• Grid technologies are complementary to other
  distributed computing technologies
• Additive, not competitive
• To date, have addressed primarily systems issues
  of interoperability and sharing
• Need to integrate with tools that address
  programming, workflow, modeling issues
• Ideally, also integrate with other systems
  technologies
• Integration with other technologies critical

33
Future Directions Grid Services?

• Core Grid protocols predate Web Services
• Various substrates HTTP, LDAP, FTP
• We are currently exploring retargeting to Web
  Services, as part of ongoing redesign
• SOAP for all protocols, WSDL, UDDI
• Grid Services layer provides soft-state, secure
  management of remote context
• Other Grid protocols build on this substrate
• Grid Services enhance Web Services with key Grid
  (peer-to-peer) capabilities

34
Summary

• Grids are being developed by a substantial
  community facing challenging problems
• Considerable experience with large-scale Grids,
  some useful technology
• Emerging standards are being adopted within
  community, also starting in IETF
• We see commonality of interest, and perhaps
  opportunities for joint work, with W3C and OMG

35
For More Information

• Book (Morgan Kaufman)
• www.mkp.com/grids
• Globus
• www.globus.org
• The Anatomy of the Grid Enabling Scalable
  Virtual Organizations
• Global Grid Forum
• www.gridforum.org
• GriPhyN
• www.griphyn.org