Introduction to GRID computing

1
Introduction to GRID computing

• Introduction GRID Tutorial
• Jules Wolfrat
• SARA

2
Definition of Grid

• From an EU brochure
• It doesnt matter if your team is modeling the
  Earths atmosphere, designing cars, creating
  animated films or finding new medicines, the
  basic principle is the same your Grid supplies
  all the computing power, software, data and
  knowledge you need in one integrated package, and
  helps project teams work more closely together
• The analogy with the power grid
• Like you can plug in anywhere to the power grid
  without knowing where your energy is coming from
  you can plug into the grid without knowing where
  your (computing) resources are coming from.

3
History (1)

• From a news item in 1991
• Smarr describes the metacomputer as a network of
  heterogeneous, computational resources linked by
  software in such a way that they can be used as
  easily as a personal computer
• So the concept was introduced already in the
  early 90s, known as metacomputing.
• Motivation was the emergence of computer
  networks.

4
Example (1)

• Following is an example of the kind of
  initiatives started in those years from close by
• In 1996 a project was started in Amsterdam
• The Amsterdam Metacomputing project is an ongoing
  effort from the University of Amsterdam (UvA),
  the Free University (VU) and "Academic Computing
  Services Amsterdam" (SARA) to develop a
  Metacomputer environment on the Amsterdam campus.
• Important components of this environment will be
  automatic distribution and monitoring of jobs
  over a network of computer systems, uniform
  access to files of other users from each place to
  work and to each computer system incorporated in
  the environment, distributed storage of data on
  various fileservers, automatic backup, migration
  and archiving, general availability of both
  commercial and public domain software on software
  servers, and a minimum of system management
  tasks.
• In this way scientists will be able to devote all
  of their time to their actual task science.

5
Example (2)

• An extensive package of services will gradually
  be implemented and finally include the following
  components
• fileservers and distributed, transparent
  file-systems
• backup, migration and archiving services
• batch-queueing systems, designed for efficient
  use of local systems, and if desired, of
  computational servers supplied by SARA
• public domain and specialist (commercial)
  software servers.
• All components will be accessible from the
  scientist's desktop. A client-server architecture
  will play an important role. Combining components
  will be a relatively easy task, enhancing
  efficiency in terms of man-hours needed to
  accomplish a given task. These pages, as well as
  the Metacomputer are still in a development stage
  ..

6
Example (3)
Systems available at SARA in 1996
CRAY YMP Vector system
Parsytec CC 56 CPUs
IBM SP2 76 CPUs
7
Example (4)

• SARA news item on 16-6-1998
• Basis voor meta-omgeving gelegd.
• Sinds 4 mei maakt SARA's IBM RS/6000 SP
  parallelle supercomputer gebruik van de DCE/DFS
  omgeving, een filesysteem dat een transparante
  computeromgeving mogelijk maakt. Met het nieuwe
  filesysteem zijn bestanden van DCE/DFS gebruikers
  wereldwijd toegankelijk met andere
  computersystemen die beschikken over DCE/DFS,
  waarmee een belangrijke basis is gelegd voor de
  meta-omgeving.
• Gebruikers aan de VU science faculty hebben nu
  op een uniforme manier toegang tot hun bestanden,
  ongeacht of ze werken op de RS/6000 SP of een
  lokaal workstation. Hetzelfde geldt voor
  gebruikers van het Parsytec CC systeem bij SARA
  vanaf zowel de Parsytec als de RS/6000 SP zijn
  alle bestanden voor de gebruiker direct
  toegankelijk.

8
Example (5)

• A web interface was developed for submitting jobs
  to the metacomputing environment, also a meta job
  language was used.
• Also job migration between systems and mpi over
  two systems was investigated
• First time we heard about globus, one of the well
  known building blocks now for grid
  infrastructures.
• Network link between systems was a problem, only
  FE link, Gbit not available, HiPPI (800 Mbps) not
  available for Parsytec.

9
Today (1)

• So what is new today?
• Scale! Grid infrastructures operate worldwide
• International infrastructures - EGEE, DEISA,
  Nordugrid, OSG, TeraGrid
• National NAREGI (Japan), UK-eScience, D-Grid,
  NLGrid
• Interoperability availability of middleware
  Globus toolkit, UNICORE, NAREGI, schedulers

10
Today (2)

• Some basic requirements for a grid infrastructure
• Transparent user administration single sign on
  (single grid identity), authorisation and
  accounting based on grid identity AAA
  facilities
• Job scheduling which can handle different
  environments
• Global data access
• Global information services job information,
  data information, resource information
• Interoperability!
• Standards needed for federation of
  infrastructures GGF, IETF.

11
Networking (1)

• Developments in network connectivity (high
  bandwidths) and tools play an important role
• 10 Gbps WAN links available today, both shared
  links and dedicated lightpaths (based on lambda
  technology)
• 1 Gbps network adapters are commodity items on
  systems today and 10GE adapters available

12
Networking (2)

• GridFTP can use multiple streams in order to take
  full advantage of available bandwidth
• Parallel files systems can take full advantage of
  underlying high speed networks - throughput can
  be in the order of 100MByte/s and more
• Tuning of WAN TCP must get attention, e.g.
  latencies are in the order of milliseconds (20
  in Europe), defaults on systems mostly not suited
  for bulk data transports.

13
SURFnet6 DWDM on dark fiber
SURFnet 6 infrastructure
Muenster
14
NetherLight Lightpath connections to the
Netherlands
3rd quarter 2005
15
Global Lambda Integrated Facility (GLIF)World Map
Visualization courtesy of Bob Patterson,
NCSA/University of Illinois at Urbana-Champaign. D
ata compilation by Maxine Brown, University of
Illinois at Chicago. Earth texture from NASA.
www.glif.is
16
GEANT2 topology
17
The EGEE project

• EGEE
• 1 April 2004 31 March 2006
• 71 partners in 27 countries, federated in
  regional Grids
• Operation of a pan European production Grid
• EGEE-II
• 1 April 2006 31 March 2008
• Expanded consortium
• 91 partners
• 11 Joint Research Units
• Natural continuation of EGEE
• Emphasis on providing production-level
  infrastructure
• increased support for applications
• interoperation with other Grid infrastructures
• more involvement from Industry

18
The EGEE infrastructure

• Mission
• Manage and operate production e-Infrastructure
  open to all user communities and service
  providers
• Contribute to Grid standardisation and policy
  efforts
• Infrastructure operation
• Currently include 200 sites across 39 countries
• Continuous monitoring of Grid services in a
  distributed global infrastructure
• Automated site configuration/management
• Future
• Expand on interoperability with related
  infrastructures

19
EGEE-II Federations and Countries
20
Operational Organisation
21
User Support Activities
22
User support in NE region

• NE website http//www.egee-ne.org/operations
• User support contact user support at local site
  or mail to support_at_egee-ne.org
• NE uses a ticketing system monitored by different
  partners from our region. In NL NIKHEF, RC-RuG,
  SARA responsible.
• Tickets from GGUS are also imported in the NE
  system
• Application support NA4 activity. In NL RC-RuG,
  SARA

23
A Selection of Monitoring tools
1. GIIS Monitor
2. GIIS Monitor graphs
3. GOC Data Base
4. Scheduled Downtimes
5. GridIce VO view
6. Live Job Monitor
24
(No Transcript)
25
The DEISA project

• Objective To enable Europes terascale science
  by the integration of Europes most powerful
  supercomputing systems.
• DEISA is an European Supercomputing Service built
  on top of existing national services. This
  service is based on the deployment and operation
  of a persistent, production quality, distributed
  supercomputing environment with continental
  scope.

26
THE DEISA SUPERCOMPUTING GRID
AIX distributed super-cluster
Vector systems (NEC, )
GEANT
Linux systems (SGI, IBM, )
27
Participating Sites
BSC Barcelona Supercomputing Centre
Spain CINECA Consortio Interuniversitario
per il Calcolo Automatico Italy CSC
Finnish Information Technology Centre for
Science Finland EPCC/HPCx University of
Edinburgh and CCLRC
UK ECMWF European Centre for
Medium-Range Weather Forecast UK
(int) FZJ Research Centre
Juelich Germany HLRS High
Performance Computing Centre Stuttgart
Germany IDRIS Institut du
Développement et des Ressources
France en Informatique Scientifique -
CNRS LRZ Leibniz Rechenzentrum Munich
Germany RZG Rechenzentrum Garching of
the Max Planck Society Germany SARA
Dutch National High Performance Computing
The Netherlands and Networking centre

28
DEISA technologies

• GPFS parallel filesystem for transparent file
  access from all systems dedicated European
  network used for high throughput
• Loadleveler-MC for job submission on AIX systems
• UNICORE for job submission to all systems
• Common Programming Environment (CPE) on all
  systems for DEISA users
• Single username on all systems

29
Access

• Users can submit proposals for access to DEISA
  resources through DECI (DEISA Extreme Computing
  Initiative) calls
• Proposals are evaluated by national committees
  and depending on ranking get access to resources
• Most partners contribute about 10 of their
  resources for DEISA applications
• URL www.deisa.org