International Computing eInfrastructures: Past, Present and Future...

1
International Computing e-Infrastructures Past,
Present and Future...

• Fabrizio Gagliardi
• EMEA Director
• Technical Computing
• Microsoft Corporation

2
Outline of the talk

• Some personal introductory remarks
• Definition of e-Infrastructures
• Need for e-Infrastructures
• Recent past history
• Current situation, accomplishments and challenges
• Outlook for the future

3
Definition of e-Infrastructures

• Infrastructures to support wide geographically
  distributed communities which share problems and
  resources to work towards common goals
• Leveraging international network
  interconnectivity
• Based on safe AAA architecture
• Need persistent software middleware (S/W is
  integral part of the infrastructure)

4
Grids A Catch-All Marketing Term

• Grids mean many different things to many
  different people/companies
• P2P desktop cycle-stealing
• Linked Supercomputer Centers
• Managed virtual distributed clusters
• Internet access to giant, distributed
  repositories
• Virtualization of data center IT resources
• Out-sourcing to utility compute centers
• Sharing resources distributed among different
  administrative domains (Ian Foster)
• For Microsoft, Grids are about Data Management as
  much as Compute Cycles

5
Need for e-Infrastructures

• Science, industry and commerce are more and more
  digital, process vast amounts of data and need
  massive computing power
• We live in a flat world
• Science is more and more an international
  collaboration and often requires a
  multidisciplinary approach
• Need to use technology for the good cause
• Fight Digital/Divide
• Industrial uptake has become essential

6
Recent past history

• Meta-computing and distributed computing early
  examples in the 80 and 90 (CASA, I-Way,
  Unicore, Condor etc.)
• EU-US workshop in Annapolis in 1999 on large
  scientific data bases http//www.cacr.caltech.edu
  /euus/
• EU FP5 and US Trillium and national Grids
• EU FP6, US OSG, NAREGI/Japan

7
Chronology
IGO-----ETICS/EGO----
EDG start 2001
EGEE-II start 2006
CHEP2000
EGEE-XX 2008?
EGEE start 2004
We are here
8
ARCADE 2002 Barcelona, 14th February 2002
Unlimited bandwidth, breaking the frontiers of
computing the path in Europe from FP5 to FP6.
Antonella Karlson Research Networks Unit, DG
INFSO, EC antonella.karlson_at_cec.eu.int
"The views expressed in this presentation are
those of the author and do not necessarily
reflect the views of the European Commission"
9
GRIDs - IST projects (36m Euro) An integrated
approach
10
GRIDs Examples of large testbeds
11
GRIDs Examples of large testbeds
12
Current situation accomplishments and challenges

• Many Grids around the world, very few maintained
  as a persistent infrastructure (best example is
  the secret Google Grid)
• Need for public and open Grids (OSG, EGEE and
  related projects, NAREGI, and TERAGRID, DEISA
  good prototypes)
• Persistence, support, sustainability, long term
  funding, easy access are the major challenges

13
Projects in Europe (I)

• Access to IT-resources (connectivity, computing,
  data, instrumentation) for scientists
• Providing e-Infrastructure
• Géant2
• EGEE
• DEISA
• SEE-GRID
• Benefiting from e-Infrastructure
• DILIGENT
• SIMDAT
• GRIDCC
• CoreGRID
• GridLab
• Concertation GRIDSTART, GridCoord
• Grid mobility Akogrimo

14
Projects in Europe (II)

• Sample of National Grid projects
• Austrian Grid Initiative
• DutchGrid
• France
• e-Toile
• ACI Grid
• Germany
• D-Grid
• Unicore
• Grid Ireland
• Italy
• INFNGrid
• GRID.IT
• NorduGrid
• UK e-Science
• National Grid Service
• OMII
• GridPP project

15
Policy Forums

• The e-Infrastructures Reflection Group (eIRG)
• Mission study and promote policies for easy and
  cost-effective shared use of electronic resources
  in Europe
• 25 countries (government-appointed
  representatives), EU 2 members
• White Papers
• European Strategy Forum on Research
  Infrastructures (ESFRI)
• Role to support a coherent approach to
  policy-making on research infrastructures in
  Europe, and to act as an incubator for
  international negotiations about concrete
  initiatives
• representatives of the 25 EU Member States,
  appointed by Research Ministers and a
  representative of the European Commission
• ESFRI eIRG European roadmap for new research
  infrastructures of pan-European interest (10-20
  years)

16
Géant2

• GÉANT2 is the 7th generation of the pan-European
  research and education network, successor to the
  multi-gigabit research network GÉANT.
• Official start 1 September 2004, Duration 4
  years
• Funding EC, national research, education
  networks
• Managed by DANTE
• Goal
• To connect 34 countries through 30 national
  research and education networks (NRENs)
• using multiple 10Gbps wavelengths
• Status
• Equipment and services currently in operations
  (officially inaugurated by Commissioner Reding
  last June)
• Transition from GÉANT network to GÉANT2 gradually
  completing, started in the first quarter of 2005

17
EU Grid technology infrastructure (I)

• New Grid Research Projects in FP6
• EU Funding52 MILLION - Start SUMMER 2004

From a talk by Ulf Dahlsten, Den Haag, Nov 2004
18
EU Grid technology infrastructure (II)

• Building the European eInfrastructure for
  research

FP7
From a talk by Ulf Dahlsten, Den Haag, Nov 2004
19
EU Grid technology infrastructure (III)

• eInfrastructure
• Testbeds

Courtesy of K. Baxevanidis, EU
20
EU Grid technology infrastructure (V)

• eInfrastructure achievements

• Connectivity service
• Computing, storage service
• Testbeds
• International links

• GÉANT network 10Gbit/s, IPv6 enabled, 3900
  Research Centres connected
• EGEE production quality, 10000 CPUs, 5PB
  storage, training, coverage of 27 countries
• DEISA Supercomputer network, reaching 40
  Tflop/s
• Rich set of technologies tested/ verified (IPv6,
  Grids, Optical, End-to-End QoS, Security,
  Mobility) and communities involved (scientific,
  industry)
• USA, Russia, Mediterranean, Asia, Latin America...

From a talk by Ulf Dahlsten, Den Haag, Nov 2004
21
Outlook for the future

• Outlook for the future

22
Supercomputing Goes Personal
23
The Continuing Trend Towards Decentralized,
Networked Resources
Grids of personal departmental clusters
Personal workstations departmental servers
Minicomputers
Mainframes
24
Leverage IT Industrys Existing RD

• Digital experimentation
• Collaboration-enhanced Office productivity tools
• Structure experiment data and derived results in
  a manner appropriate for human reading/reasoning
  (as opposed to optimizing for query processing
  and/or storage efficiency)
• Enable collaboration among colleagues
• (Scientific) workflow environments
• Automated orchestration
• Visual scripting
• Provenance

• Parallel applications development
• High-productivity IDEs
• Integrated debugging/profiling/tracing/analysis
• Code designer wizards
• Concurrent programming frameworks
• Platform optimizations
• Dynamic, profile-guided optimization
• New programming abstractions
• Distributed systems issues
• Web Services HPC grids
• Security
• Interoperability
• Scalability
• Dynamic Systems Management
• Self (re)configuration tuning
• Reliability availability
• RDMS data mining
• Ease-of-use
• Advanced indexing query processing

25
Scientific Information WorkerPast and Future

• Past
• Buy lab equipment
• Keep lab notebook
• Run experiments by hand
• Assemble analyze data (using stat pkg)
• Collaborate by phone/email write up results with
  Latex
• Metaphor
• Physical experimentation
• Do it yourself
• Lots of disparate systems/pieces

• Future
• Buy hardware software
• Automatic provenance
• Workflow with 3rd party domain packages
• Excel Access/SQL-Server
• Office tool suite with collaboration support
• Metaphor
• Digital experimentation
• Turn-key desktop supercomputer
• Single integrated system

26
Where Grids will be in 5 years?

• Like in the past ES, AI, networking, OS they will
  disappear from the hot research (and hype) space
  and become mainstream technology
• Major Grids already work in production (EGEE
  18000 computers, Google 100000 computers?...)
• Major IT vendors will integrate Grid middleware
  in their standard products (industrial uptake)
• Computing and data resources will become
  commodities on the Internet
• ISPs will offer a wide range of services Grid
  based, a full mature market will develop for
  these services
• The result will be a tremendous computing and
  data processing power which will enable a new set
  of scientific applications and generate large
  revenues for business applications
• A potential leveler for a worldwide science and
  economy digital Divide could be moderated

27

• And time will tell how wrong we are in our
  predictions now
• See you back here next year!