Lectures on Grid Computing

1
Lectures on Grid Computing
Tugba Taskaya-Temizel Prof. K. Ahmad January 2005
2
Grid ComputingEverywhere

• Business Sectors like financial services,
  industrial manufacturing, energy

Humanitarian works
Research Health, Aerospace, Astronomy,
Finance
Government
3
Grid Computing

• The internet took 20 years to be taken seriously
  by business. By comparison the grid is happening
  far more rapidly. Tom Hawk, IBM
• Insight Research says the worldwide market for
  grid technology and services is doubling every
  year and will reach 5 billion by 2008.
• Grid computing is just one of the technologies
  the UK government says, in its latest report,
  should receive more support and funding.
  (December 17,2003)

4
Grid Computing

• "We really do believe that grid computing is
  real," CEO of Hewlett-Packard Carly Fiorina said.
  "It is driving the RD in our industry. For the
  first time our energy is focused on something
  else than building a killer app or a hot box. We
  are more focused on making system that combines
  the best of IT and business. Imagine what is
  possible." (September 11, 2003)
• "The Grid will be the major new direction for
  IT," said Geoff Brown, technical director for ATS
  Core Technologies at Oracle. (October 28, 2002)

5
DEFINITIONS Grid?

• ELECTRICITY GRID
• A network of high-voltage transmission lines and
  connections that supply electricity from a number
  of generating stations to various distribution
  centres in a country or a region, so that no
  consumer is dependent on a single station.
• UTILITY GRID
• (Term) used of any network that serves a
  similar purpose for other services.

www.oed.com
6
DEFINITIONS Grid?

• GRID
• The Grid is envisaged to be the computing and
  data management infrastructure that will provide
  the electronic underpinning for a global society
  in business, government, science and
  entertainment
• Berman, Fox and Hey (20039)

7
DEFINITIONS Grid?

• GRID
• A virtual information processing environment
  where the user has the illusion of a seamless
  single-source computing power which is actually
  distributed.

8
Why should you care?

• Ian Foster explains why we should care Grids in
  three points

9
Why should you care?

• Grid is a disruptive technology Vision
• It ushers in a virtualized, collaborative,
  distributed world.
• Two interrelated opportunities
• 1) Enhance economy, flexibility, access by
  virtualizing computing resources
• 2) Deliver entirely new capabilities by
  integrating distributed resources

Source Ian Foster s presentation on The Grid
, COMDEX 2003, Las Vegas, Nevada USA, November
18, 2003
10
Why should you care? Virtualization
Source The Grid Blueprint for a New Computing
Infrastructure (2nd Edition), 2004
11
Why should you care? Distributed System
Integration
UK e-Science Centres
Source http//www.nesc.ac.uk/centres/
12
Why should you care?
The real and specific problem that underlies the
Grid concept is coordinated resource sharing and
problem solving in dynamic, multi-institutional
virtual organizations.
Source The Anatomy of the Grid, Foster,
Kesselman, Tuecke, 2001
13
Why should you care?Terminology

• Grid has strong links with Utility Computing,
  Autonomic Computing and Service Oriented
  Architecture.

14
Why should you care?

• Grid addresses pain points now Reality
• Grids are built not bought, but are delivering
  real benefits in commercial settings
• Low utilization of enterprise resources
• High cost of provisioning for peak demand
• Inadequate resources prevent use of advanced
  applications
• Lack of information integration

Source Ian Foster s presentation on The Grid
, COMDEX 2003, Las Vegas, Nevada USA, November
18, 2003
15
Why should you care?Early Commercial Applications

• Leading adopters (Oct 2003)
• Financial services 31
• Life sciences 26
• Manufacturing 18

Grid Services Market Opportunity 2005
Sources IDC, 2000 and Bear Stearns- Internet 3.0
- 5/01 Analysis by SAI
Source Ian Foster s presentation on The Grid
, COMDEX 2003, Las Vegas, Nevada USA, November
18, 2003
Grids 2004 From Rocket Science To Business
Service, The 451 Group
16
Why should you care?Grid Deployment Strategies

• A range of excellent commercial open source
  products for resource federation
• Federate enterprise computing resources
• Federate enterprise information resources
• Globus Toolkit inter-enterprise sharing
• But, Grids are built, not bought
• Integration with other enterprise systems is
  needed to deliver complete solution
• Start small with well-defined ROI case
• Grow based on experience

Source Ian Foster s presentation on The Grid
, COMDEX 2003, Las Vegas, Nevada USA, November
18, 2003
17

Data Grids for High Energy Physics
Fastest particle accelarator Large Hadron
Collider When completed in 2005, CERN's Large
Hadron Collider will send protons and ions from
hydrogen nuclei rushing through a 17-mile
circular tunnel at speeds of up to 52,200,000
miles per hour.
Image courtesy Christian Richters SourceWired
News
18

Data Grids for High Energy Physics
Image courtesy Harvey Newman, Caltech
19
Mathematicians Solve NUG30Quadratic Assignment
Problem

• The distances are
• d(1,2) 22,
• d(1,3) 53,
• d(2,3) 40,
• d(3,4) 55.

Location 1
Location 2
Location 4

• The required flows between facilities are
• f(2,4) 1,
• f(1,4) 2,
• f(1,2) 3,
• f(3,4) 4.

Location 3
The permutation p corresponding to this graphical
solution is ( 2, 1, 4, 3 ).
MetaNEOS Argonne, Iowa, Northwestern,
Wisconsin SourceShawn McKee The GridThe Future
of High Energy Physics Computing? January 7,2002
20
Mathematicians Solve NUG30

• Looking for the solution to the NUG30 quadratic
  assignment problem
• An informal collaboration of mathematicians and
  computer scientists
• Condor-G delivered 3.46E8 CPU seconds in 7 days
  (peak 1009 processors) in U.S. and Italy (8 sites)

NUG30 Solution 14,5,28,24,1,3,16,15, 10,9,21,2,4,
29,25,22, 13,26,17,30,6,20,19, 8,18,7,27,12,11,23
MetaNEOS Argonne, Iowa, Northwestern,
Wisconsin SourceShawn McKee The GridThe Future
of High Energy Physics Computing? January 7,2002
21
Network for Earthquake Engineering Simulation

• NEESgrid national infrastructure to couple
  earthquake engineers with experimental
  facilities, databases, computers, each other
• On-demand access to experiments, data streams,
  computing, archives, collaboration

NEESgrid Argonne, Michigan, NCSA, UIUC, USC
22
The 13.6 TF TeraGrid Computing at 40 Gb/s
Site Resources
Site Resources
26
HPSS
HPSS
4
24
External Networks
External Networks
8
5
Caltech
Argonne
External Networks
External Networks
NCSA/PACI 8 TF 240 TB
SDSC 4.1 TF 225 TB
Site Resources
Site Resources
HPSS
UniTree
TeraGrid/DTF NCSA, SDSC, Caltech, Argonne
www.teragrid.org
23
iVDGLInternational Virtual Data Grid Laboratory

• Sloan Digital Sky Survey is the most ambitious
  astronomical survey project ever undertaken.
• The survey will map in detail one-quarter of the
  entire sky, determining the positions and
  absolute brightnesses of more than 100 million
  celestial objects.
• It will also measure the distances to more than a
  million galaxies and quasars

U.S. PIs Avery, Foster, Gardner, Newman, Szalay
www.ivdgl.org Image courtesy of
http//www.sdss.org/news/releases/20050111.yardsti
ck.html
24
iVDGLInternational Virtual Data Grid Laboratory
U.S. PIs Avery, Foster, Gardner, Newman, Szalay
www.ivdgl.org
25
Why should you care?

• An open Grid is to your advantage Future
• Standards are being defined now that will
  determine the future of this technology

Source Ian Foster s presentation on The Grid
, COMDEX 2003, Las Vegas, Nevada USA, November
18, 2003
26
Grid Vision, Marketing, and Reality

• Vision
• Computing data resources can be shared like
  content on the Wb
• Marketing
• Have we got a Data, compute, knowledge,
  information, desktop, PC, enterprise, cluster,
  Grid for you!
• Reality
• Commercial products mostly noninteroperable
• Open source tools offer de facto standards, but
  are also far from a complete solution

Source Ian Foster s presentation on The Grid
, COMDEX 2003, Las Vegas, Nevada USA, November
18, 2003
27
Standards Matter!

• Open, standard protocols
• Enable interoperability
• Avoid product/vendor lock-in
• Enable innovation/competition on end points
• Enable ubiquity
• In Grid space, must address how we
• Describe, discover, access resources
• Monitor, manage, coordinate, resources
• Account charge for resources
• For many different types of resource

Source Ian Foster s presentation on The Grid
, COMDEX 2003, Las Vegas, Nevada USA, November
18, 2003
28
Open Grid Services Architecture

• Define a service-oriented architecture
• the key to effective virtualization
• that addresses vital Grid requirements
• AKA utility, on-demand, system management,
  collaborative computing
• in particular, distributed service management
• building on Web services standards
• extending those standards where needed

The Physiology of the Grid An Open Grid
Services Architecture for Distributed Systems
Integration, Foster, Kesselman, Nick, Tuecke,
2002
29
Latest Step ForwardWS-Resource Framework

• A family of six Web services specifications
• A design pattern to specify how to use Web
  services to access stateful components
• Message-based publish-subscribe to Web services

Source Ian Foster s presentation on The Grid
, COMDEX 2003, Las Vegas, Nevada USA, November
18, 2003
30
WS-Resource Framework Completes Grid-WS
Convergence
Grid
Web
The definition of WSRF means that Grid and Web
communities can move forward on a common base
Source Ian Foster s presentation on The Grid
, COMDEX 2003, Las Vegas, Nevada USA, November
18, 2003
31
The Evolution of the GRID
32
The Evolution of the GRID

• Currently there are (clusters) of very powerful
  computing/ communications systems
• (i) Systems for acquiring digital data and
  processing data (Amazon.com or Oracle clusters)
• (ii) Systems for analysing and visualising
  information (CERNs large hadron collider,
  Protein Synthesis systems)
• (iii) Systems for imaging, analysis and
  visualisation for distributed data (weather
  prediction, satellite based military civilian
  systems)
• (iv) Systems that can link Sensors and predict
  on real-time information (military systems, video
  surveillance)

33
The Evolution of the GRID

• Developments in networking technologies,
  operating systems, clustered data bases,
  application services and device technologies have
  enabled developers to build systems with
  literally distributed millions of nodes for
  providing
• Web-based services personal commercial
  transactions
• Content delivery networks that can cache
  web-pages seamlessly
• Wireless networks have spawned ad-hoc
  distributed systems that when linked to
  wide-area networks lead to a complex distributed
  system.
• Problems of efficiency, reliability,
  accessibility and security are not addressed in
  global terms.

34
The Evolution of the GRID
Source www.gridbus.org
35
The Evolution of the GRID
Source www.gridbus.org
PERFORMANCE Q o S
Administrative Barriers

• Individual
• Group
• Department
• Campus
• State
• National
• Globe
• Inter Planet
• Universe

Personal Device
SMPs or SuperComputers
Local Cluster
Global Grid
Enterprise Cluster/Grid
36
The Evolution of the GRID
Grid is being developed not only to make
distributed resources available to end-user not
also to co-ordinate such usage ? for sharing and
aggregation of resources.
37
The Evolution of the GRID

• Moores law improvements in computing produce
  highly functional end-systems
• The internet and burgeoning wired and wireless
  provide wide-spread connectivity
• Changing modes of working and problem solving
  emphasise teamwork, computation
• Network growth produce dramatic changes in
  topology and geography

38
The Evolution of the GRID

• The first generation involved proprietary
  solutions for sharing high-performance computing
  resources
• The second generation introduced middleware to
  cope with scale and heterogeneity
• The third generation introduced a
  service-oriented approach leading to commercial
  projects in addition to the scientific projects
  now collectively known as e-Science

39
The Evolution of the GRID

• The first generation
• FAFNER, I-WAY
• The second generation
• Technologies Globus, Legion
• Distributed object systems (Jini and RMI, The
  common component architecture form)
• Grid resource brokers and schedulers
• Grid portals
• Integrated systems
• Peer-to-Peer computing
• The third generation
• Service-oriented architecture (web services,
  OGSA, Agents)
• Information aspects relation with the World Wide
  Web
• Live information systems

40
Source Ian Foster s presentation on The First
50 Years , British Computer Society, Lovelace
Medal Award Presentation, May, 2003
41
Building blocks of the Grid

• Networks
• Computational nodes on the Grid
• Pulling it all together
• Common infrastructure standards

42
GRID Key Issues
43
GRID Key Issues ? Sharing

• A biochemist will be able to exploit 10,000
  computers to screen 100,000 compounds in an hour
• 1,000 physicists worldwide will be able to pool
  resources for petop analyses of petabytes of data
• A multidisciplinary analysis in aerospace couples
  code and data in geographically distributed
  organisations may be possible
• Civil engineers colloborate to design, execute,
  and analyse shake table experiments
• Climate scientists will be able to visualise,
  annotate, and analyse terabyte simulation datasets

44
GRID Key Issues ? Sharing Online Access to
Scientific Instruments
Advanced Photon Source
wide-area dissemination
desktop VR clients with shared controls
real-time collection
archival storage
tomographic reconstruction
DOE X-ray grand challenge ANL, USC/ISI, NIST,
U.Chicago
45
MORE DEFINTIONS

• Resource
• Network protocol
• Network enabled service
• Application Programming Interface(API)
• Software Development Kit (SDK)
• Syntax

46
MORE DEFINTIONS Resource

• An entity that is to be shared
• E.g., computers, storage, data, software
• Does not have to be physical entity
• E.g., Condor pool, distributed file system,
• Defined in terms of interfaces, not devices
• E.g. scheduler such as LSF and PBS define a
  compute resource
• Open/close/read/write define access to a
  distributed file system, e.g NFS, AFS, DFS

47
MORE DEFINTIONS Network protocol

• A formal description of message formats and a set
  of rules for message exchange
• Rules may define sequence of message exchanges
• Protocol may define state-change in endpoint,
  e.g. file system state change
• Good protocols designed to do one thing
• Protocols can be layered
• Examples of protocols
• IP, TCP, TLS( was SSL), HTTP, Kerberos

48
MORE DEFINTIONS Network enabled services

• Implementation of a protocol that defines a set
  of capabilities
• Protocol defines interaction with service
• All services require protocols
• Not all protocols are used to provide services
  (e.g. IP, TLS)
• Examples FTP and Web servers

49
MORE DEFINTIONS Application Programming
Interface (API)

• A specification for a set of routines to
  facilitate application development
• Spec often language specific (or IDL)
• Routine name, number, order and type of
  arguments mapping to language constructs
• Behaviour or function of routine
• Examples
• GSS API(security), MPI (message passing)

50
MORE DEFINTIONS Software Development Kit (SDK)

• A particular instantiation of API
• SDK consists of libraries and tools
• Provides implementation of API specification
• Can have multiple SDKs for an API
• Examples of SDKs
• MPICH, Motif Widgets

51
MORE DEFINTIONS Syntax

• Rules for encoding information, e.g.
• XML, Condor ClassAds, Globus RSL
• Distinct from protocols
• One syntax may be used by many protocols
• Syntaxes may be layered
• E.g., Condor ClassAds -gt XML-gtASCII

52
References

• Berman F., Fox G., Hey T. (2003) Grid Computing
  Making the Global Infrastructure a Reality,
  Chichester, John Willey Sons Inc.
• http//www.computing.surrey.ac.uk/courses/csm23/li
  st.html

53
CSM23 Assessment and Weighting
54
CSM23 Timetable