Grid Computing and the NorduGrid

1
Grid Computing and the NorduGrid

• Balázs Kónya
• LDC NETinfo-meeting
• 28 May 2002

2
What is the Grid?

• Grid is a technology to share and seamlessly
  access computing resources of the world
• The future infrastructure of computing and data
  management
• a kind of distributed computing
• Middleware
• World Wide Web access to information
• World Wide Grid access to computing capacity and
  ...

3
Grid The beginning

• Ian Foster, Karl Kesselman, July, 1998
• Blueprint for a new Computing Infrastructure
• The Computational Grid lt-gt Power Grid analogy was
  suggested
• Early Testbeds in the USA connected
  supercomputing centers
• The birth of the Globus toolkit

4
Grid Today

• The media is full of Grid announcements
• Grid is expected to become the NEXT big thing
  after the WEB
• Governments and Research Foundations rush to
  support Grid projects all over the word
• Europe seems to lag behind in the race
• Huge commercial interests
• IBM
• Grid-enable the companys entire product
  portfolio
• Close cooperation with the open source Globus
  development community and Global Grid Forum
  standards
• Platform Computing
• Recently announced the first commercial Grid
  software, the Platform Globus

5
what is behind?

• Powerful PCs are everywhere
• Clusters are commodity
• Network Storage Computing exponentials
• Networking speed grows faster than computing
  power
• Even data storage outperforms the CPUs

taken from Scientific American, Jan 2001
6
the future ...

• Cohen Communication Group
• Grid computing will be the driving force behind
  the 150 annual internet traffic expansion in
  2005
• in contrast to the 60 predicted growth rate
  accounted mainly for video streaming and video
  file transfer forecasted by McKinsey - JP Morgan
  
• IBM
• Grid is the next utility in the line of the
  water, drainage, gas and electricity systems
• people will pay their computing bills
• Ian Foster
• People used to overestimate the short term impact
  of change but underestimate the long-term effect

7
Grid Supercomputers

• The present day Supercomputers are the PC
  clusters
• Grid will provide a uniform access to all the
  resources
• The Supercomputing centers will be the power
  plants of the Grid

Cray T3E
PC cluster
The Grid
Sun E10000
8
Grid projects

• USA
• TeraGrid (53 million)
• 53 million from the NSF
• 13.6 teraflops of Linux clusters
• 450 terabytes of data storage
• 4 sites
• 40 Gbits/sec (later 50-80) network connections
• Globus based Grid toolkits
• Visualisation environment
• Nasa Information PowerGrid (IPG)
• Earth System Grid
• Asia
• Asia-Pacific Grid (ApGrid)
• Japanese Grid initiatives
• ThaiGrid
• KoreanGrid

9
Grid projects II

• Europe
• European Data Grid (EDG) (9.8M Euro)
• UK GridPP (26M Pounds)
• UK E-science project (98M Pounds)
• INFN Grid, Italy 10M Euro in 2002
• Finlands Grid research budget (connected to
  DataGrid) 0.5M Euro /year
• EU 6th Framework Programme Grid activities
  expect 40 MEuro/year for 5 years, plus
  30MEuro/year from Information Society Programme.
• Cross Grid projects in order to link together
  Grid projects
• for more projects see www.gridcomputing.com

10
Grid in Scandinavia (history)

• PDC participation in the early Globus and Condor
  tests development of tools
• March 2000 Lund workshop first discussion on
  using Grid for basic research in particle physics
• During 2000 DataGrid and NorduGrid projects were
  formed.

11
Grid in Scandinavia (DataGrid)

• Elementary Particle Physics LU the EU DataGrid
  Project
• (WP6) Testbed through the NorduGrid project
• (WP8) HEP Applications through the NorduGrid
  project
• more info on EDG www.eu-datagrid.org

12
Grid in Scandinavia The NorduGrid Project

• Nordic Testbed for
• Wide area Computing and Data Handling
• www.nordugrid.org
• The pioneering Grid project in Scandinavia

13
NorduGrid objectives

• Introduce the Grid to Scandinavia
• Pursue basic research on Grid Computing
• Develop Middleware Solutions
• Create a Grid infrastructure in Nordic countries
• Survey current Grid technologies
• Operate a functional Testbed
• Expose the infrastructure to end-users of
  different scientific communities

14
(No Transcript)
15
NorduGrid project details

• Start May 2001, end October 2002
• The project is financed via the Nordunet2
  programme.
• Budget 2.2 MDKK. Coordinator J. Renner Hansen,
  NBI.
• The project was initiated by the high energy
  physics community
• LHC data challenge of Cern (several Petabytes /
  year)
• Operates as part of the EU DataGrid Testbed
• Physicists are among the driving forces of the
  World Wide Grid developement
• A heighly acknowledged member of the Grid
  community
• Cern, where the web was born

16
NorduGrid resources

• Network Nordunet network
• Hardware dedicated test clusters at 5 sites (3-5
  machines), separate test machines at 2 sites
• 7 Grid-connected sites
• Manpower 5 postdoc researcher (3 of them
  financed by NorduGrid)
• The Nordic supercomputing centers are about to
  put their resources onto the Grid
• NSC clusters from Linköping
• Nordic Grid Consortium

17
NorduGrid organization

• Steering panel for management
• Technical working group
• To carry out the projects activities
• Fully distributed research project
• Researchers are scattered all over Scandinavia
• (A. Wäänänen Copenhagen, O. Smirnova B.Kónya
  Lund, M. Ellert Uppsala, A. Konstantinov Oslo)
• Project is succesfully run via the web, mailing
  lists phone conferences
• Bi-monthly technical meetings
• NorduGrid Workshops
• March 2001 Lund, November 2001 Oslo
• 22-23 May 2002 Helsinki First Presentation of
  the NorduGrid Toolkit

18
NorduGrid activities

• The fabric of the NorduGrid was laid down by June
  2001
• Setting up the dedicated clusters, machines
• NorduGrid Authentication System was put into
  operation in May 2001
• NorduGrid Certificate Authority in Copenhagen
• Issues certificates for Nordic Grid users
• Recognized by major Grid projects
• The first middleware was deployed and the sites
  were Grid-enabled by July 2001
• Early version (1.1.4) of Globus
• Part of Datagrid Testbed 0

19
NorduGrid activities cont.

• Active participation in Globus version 2 alpha
  testing (August-October)
• NorduGrid was among the first sites successfully
  running Globus version 2
• Globus v. 2 beta was officially released in
  November
• Installation and detailed investigation of the
  GridFTP service over the NorduGrid resources
  (October)
• http//arxiv.org/abs/cs.DC/0205023
• The Globus project showed interest in our results
• Further Grid services were put into operation
  (November-December)
• NorduGrid User Management System (Virtual
  Organization)
• NorduGrid Information System
• Grid Data Mirroring Package (GDMP)
• Data replication catalog

20
NorduGrid activities cont.

• Deployment evaluation of the first (Testbed 1)
  release of the EDG Middleware (December-January)
• Design and implementation of the NorduGrid
  Middleware Architecture
• information system, gridmanager, userinterface,
  broker, xrsl language
• http//arxiv.org/abs/cs.DC/0205021
• March 28, 2002 first REAL job is submitted from
  Lund to Oslo and successfully returned all the
  expected output
• May 23, a production level Grid System was
  demonstrated in Helsinki at the NorduGrid
  Workshop
• The Atlas Data Challenge 1 will use the NorduGrid
  Testbed
• Petabytes of data (107 events) will be generated,
  stored and analyzed in a distributed way

21
NorduGrid Summary

• The underlying structure of a Trans-Nordic
  Computational Grid is set up by the NorduGrid
  project
• all the presently available Grid-services are
  enabled
• the Testbed is ready for production
• the NorduGrid Testbed is about to expand ...
• NorduGrid carries out a pioneering role in
  building a Grid infrastructure in Scandinavia
• Pursues generic research on Grid
• Deploys operates the NorduGrid
• Continuously evaluates the rapidly changing Grid
  technologies
• Actively contributes to the Grid middleware
  development
• Strong ties with the Grid research community
• Contribute to the Testbed of EU DataGrid
• NorduGrid is acknowledged by and has a good
  working relation to the industry standard
  Globus project.

22
Future ...

• Strategic importance of Grid-competence has been
  emphasized by Research Council
• In a few years, it is foreseen that a Nordic
  Grid-Centre will be created
• The Nordic Centre will be a so-called Tier-1
  Centre with a capacity of about 1700 x (Dual
  1 GHz Intel PIII) plus fileservers and tape
  robots.
• Financial scope 150 MSEK (16MEuro).
• The Centre will serve all interested fields of
  science. N
• Meanwhile
• NorduGrid will be continued funding frame 0.5
  MEuro/year (4.5 MSEK), personnel and dedicated
  hardware.
• NorduGrid continues to operate and extend its
  Testbed

23
NorduGrid in action

• a brief demo of a Grid session
• job submission
• resource brokering
• Grid status monitoring
• job status information
• result retrieval (gridftp)

24
you are welcome

• if you happen to have a cluster that you want to
  put on the Grid
• if you are interested in Grid research and want
  to solve some of the outstanding questions
• if you have an application which is
  embarrassingly parallel and cries for the Grid
• or you simple would like to check out what the
  Grid can do for you
• NorduGrid is planning to organize Grid Tutorials
  and demonstrations