Developing Grid and WebBased Applications

1
Developing Grid and Web-Based Applications
Opportunities Challenges
Shannon Whitmore San Diego Supercomputer
Center swhitmor_at_sdsc.edu
2
Introduction Seti_at_Home

• Basic ideas
• distribute small work units to many compute
  resources
• utilize idle compute cycles
• build redundancy into the system
• Simple client application
• available to the public
• ported to a variety of platforms

3
Intro Seti_at_Home--Results

• Over a 12 month period
• 1.7e21 floating point operations
• 221 million recorded work units
• Average throughput 27.36 teraflops
• Participants in 200 countries
• Client running on 140 platforms
• 1.7 million CPU hours total
• 1.5 million on x86 platform

4
Intro Grid Motivation

• Extend computing power
• Faster time to results
• More sophisticated analysis
• Lower computing costs
• Maximize use of existing resources
• Separate data acquisition from computation
• Decrease dependence on one system
• Facilitate collaboration

5
Why isnt grid computing as easy as SETI appears
to be?
6
Interactive

• How many of you have opinions on the grid?
• How many of you have run applications on the
  grid?
• If you have, then you know that running
  applications is not always easy

7
People

• Resource site staff
• Middleware developers
• Grid applications developers
• Users

8
Grid Layers

Hardware Software (Compute, Memory, Disk,
Application, Network)
Resources
Virtual Operating System (Authentication
Security, Access, etc)
Middleware
Applications
High Level Software
Grid Consumer
9
Grid Environments

• Geographically dispersed resources
• Different rules for access
• establishing accounts
• tracking usage
• managing security
• Heterogeneous systems
• hardware
• operating system
• middleware
• configuration

10
Grid Layers

Hardware Software (Compute, Memory, Disk,
Application, Network)
Resources
Virtual Operating System (Authentication
Security, Access, etc)
Middleware
Applications
High Level Software
Grid Consumer
11
Middleware

• Rapidly changing standards
• Heterogeneous systems interoperability
• Resources come and go
• Consists of large code base
• Not robust or hardened
• Incomplete
• Developed by different organizations

12
Why the Complexity (contd)

• Many components exist
• What do they do?
• Which ones to use?

Development Tools/Libraries GridPort
(Pending) MPICH-G2 LaPACK for Clusters
Distributed Data Management Datacutter Storage
Resource Broker Globus (GridFTP/GASS)
Job Management Globus (GRAM) APST Condor-G
Monitoring/Prediction Network Weather
Service Ganglia Globus (MDS)
Authentication GSI-OpenSSH MyProxy Kx.509
Configuration Management GridConfig
13
Grid Layers

Hardware Software (Compute, Memory, Disk,
Application, Network)
Resources
Virtual Operating System (Authentication
Security, Access, etc)
Middleware
Applications
High Level Software
Grid Consumer
14
Grid Applications

• Middleware does not guarantee a minimum
  environment
• Environment variables
• Existence/Location of software
• Grid services may not be available when needed
  sometimes software is down
• Applications are difficult to debug
• Pinpointing problems across distributed resources
• Some software does not expose debugging
  information

15
Grid Layers

Hardware Software (Compute, Memory, Disk,
Application, Network)
Resources
Virtual Operating System (Authentication
Security, Access, etc)
Middleware
Applications
High Level Software
Grid Consumer
16
Grid Consumers

• Access to a grid is not always easy
• Obtain an accounts at all sites
• Obtain certificate
• Certificate propagation involves multiple steps
• Grid applications are complex
• Setup/configuration steps
• Complexity from lower levels is not hidden
• Most users need to be grid experts

17
Challenges Summary

• Its a complex system in its infancy
• Middleware wont immediately stabilize
• Standards will take a while to settle
• Tremendous potential
• Burden at this time is on the grid application
  developer

18
People

• Resource site staff
• Middleware developers
• Grid applications developers
• Users

19
What can be done?
20
Opportunities

• Hide complexities of lower layers
• Users should only have to know enough
• to run their application
• Make software fault tolerant
• Develop reliable and easy to use interfaces
• Provide documentation and user support

21
Opportunities (Contd)

• Automate as much as possible
• Grid accounts and certificates
• Develop Web Portals/Services
• Single point of access to the grid
• Simplifies workflows
• Spend time developing glue code
• middleware integration
• missing functionality
• error checking

22
Opportunities (contd)

• Work on narrower problems
• Make a smaller number of tools work well
• Develop relationships
• With users, resource sites, and middleware
  developers
• Get more involved in standardization process

23
Case Study
24
Scalable Visualization Portal
An application for rendering frames of raw
volumes across heterogeneous grid resources
Your Job
SRB
Volume Archive
25
VisPortal

• Hardware
• NPACI Grid
• Horizon, Longhorn, Morpheus
• Teragrid
• SDSC
• Software
• NPACKage
• APST, GridPort, SRB, Globus GSI-OpenSSH
• Scalable Visualization Toolkit
• Vista (renderer)

26
NPACI Grid
Collectively, the NPACI Grid provides 1766 CPUs
and 4190 peak GFlops of computing power.
Longhorn
Hypnos
Morpheus
Blue Horizon
27
Grid middleware
28
Under the Hood
Grid Resources
SRB Server
Job Submission Host
Portal
Raw volumes Rendered images
Longhorn Blue Horizon Morpheus Hypnos
Teragrid APST SRB Client Vista
WebServer VisPort interface GridPort Globus GSI
Open SSH
Griddle APST Globus GSI Open SSH
User Workstation
Web Browser SRB client
29
Job Execution
30
VisPortal Live
Demo
31
Acknowledgments
32

• Additional Information
• http//npackage.npaci.edu
• http//npacigrid.npaci.edu
• http//vis.sdsc.edu
• http//vistools.npaci.edu
• http//visservices.npaci.edu

swhitmor_at_sdsc.edu
grid-consult_at_npaci.edu
33
SCEC Visualization

• Rupture dynamics simulation
• Rupture event between 2 faults at the
  hypocenter, located underground
• Velocity wave front shown at the north-south
  strike-slip fault
• 3000 time steps
• every 5th step saved
• 600 frames
• 1 GB sized files