The Future of EGEE and gLite

1
The Future of EGEE and gLite

• Dieter Kranzlmüller
• GUP Institute of Graphics and Parallel
  ProcessingJoh. Kepler Univ. Linz, Austria

CE EGEESEEGRID-2 Summer School on Grid Appl. 08
July 2006 Budapest, Hungary
2
GUP Institute of Graphics and Parallel
ProcessingJoh. Kepler University Linz
EGEE Enabling Grids for E-sciencECERN, Geneva,
Switzerland
Austrian Grid Initiative
GGF - Global Grid Forum
e-IRG - e-Infrastructure Reflection Group
3
Lost in Definitions?

• Defining the Grid
• Access to (high performance) computing power
• Distributed parallel computing
• Improved resource utilization through resource
  sharing
• Increased memory provision
• Controlled access to distributed memory
• Interconnection of arbitrary resources (sensors,
  instruments, )
• Collaboration between users/resources
• Higher abstraction layer above network services
• Corresponding security

4
Defining the Grid

• A Grid is the combination of networked resources
  and the corresponding Grid middleware, which
  provides Grid services for the user.
• This interconnection of users, resources, and
  services for jointly addressing dedicated tasks
  is called a virtual organization.
• Comparison between Grids and Networks
• Networks realize message exchange between
  endpoints
• Grids realize services for the users ? higher
  level of abstraction

5
Defining the Grid

• A Grid is the combination of networked resources
  and the corresponding Grid middleware, which
  provides Grid services for the user.

6
The EGEE Project

• EGEE
• 1 April 2004 31 March 2006
• 71 partners in 27 countries, federated in
  regional Grids
• EGEE-II
• 1 April 2006 31 March 2008
• Expanded consortium
• 91 partners
• 11 Joint Research Units (48 partners)
• Exploitation of EGEE results
• Emphasis on providing production-level
  infrastructure
• ? increased support for applications
• ? interoperation with other Grid
  infrastructures
• ? more involvement from Industry

7
Defining the Grid

• A Grid is the combination of networked resources
  and the corresponding Grid middleware, which
  provides Grid services for the user.
• Status of EGEE-II (as of July 8, 2006)

8
EGEE Infrastructure
Country participating in EGEE

• Scale (June 2006)
• 200 sites in 40 countries
• 20 000 CPUs
• gt 10 PB storage
• gt 20 000 concurrent jobs per day
• gt 60 Virtual Organizations

9
EGEE Infrastructures

• Production service
• Scaling up the infrastructure with resource
  centres around the globe
• Stable, well-supported infrastructure, running
  only well-tested and reliable middleware
• Pre-production service
• Run in parallel with the production service
  (restricted nr of sites)
• First deployment of new versions of the gLite
  middleware
• Test-bed for applications and other external
  functionality
• T-Infrastructure (TrainingEducation)
• Complete suite of Grid elements and application
  (Testbed, CA, VO, monitoring, )
• Everyone can register and use GILDA for training
  and testing

15 sites on 3 continents(all of them GÉANT sites)
10
Defining the Grid

• A Grid is the combination of networked resources
  and the corresponding Grid middleware, which
  provides Grid services for the user.
• Status of EGEE-II (as of July 8, 2006)

11
Production Grid Middleware

• Key factors in EGEE Grid Middleware Development
• Strict software process
• Use industry standard software engineering
  methods
• Software configuration management, version
  control, defect tracking, automatic build system,
  
• Conservative approach in what software to use
• Avoid cutting-edge software
• Deployment on over 100 sites cannot assume a
  homogenous environment middleware needs to work
  with many underlying software flavors
• Avoid evolving standards
• Evolving standards change quickly (and sometime
  significantly cf. OGSI vs. WSRF) impossible to
  keep pace on gt 100 sites

Long (and tedious) pathfrom prototypes to
production
12
EGEE Middleware gLite

• Exploit experience existing components
• VDT (Condor, Globus)
• EDG/LCG
• AliEn
• Develop a lightweight stack of EGEE generic
  middleware
• Dynamic deployment
• Pluggable components
• Focus is on re-engineering and hardening
• March 4, 2006 gLite 3.0

gLite 3.0
13
Developing

• gLite 3.0 now available on production
  infrastructure
• After gLite 3.0
• Continuous release of single components
• As needed by users and as made available by
  developers
• Major releases provide a check-point
• In general in coincidence with major application
  challenges
• Continuing development to
• Bring components not yet included in release to
  maturity
• Improve functionality
• Increase robustness
• Increase usability
• Improve the compliance to international standards

14
Grid Interoperability

• Leading role in building world-wide grids
• Incubator for new Gridprojects world-wide
• Interoperation efforts
• Bilateral EGEE/OSG, EGEE/NDGF, EGEE/NAREGI
• Multilateral Grid Interoperability Now (GIN)
• Experiences and requirements fed back into
  standardization process (GGF)
• Strengthening contacts with industry

15
Building Software for the Grid
Courtesy IBM
Platform Infrastructure
Unix
Windows
JVM
TCP/IP
MPI
.Net Runtime
VPN
SSH
Slide Courtesy David Abramson
16
Building Software for the Grid
Upper Middleware Tools
Lower Middleware
Courtesy IBM,
Bonds
Platform Infrastructure
Unix
Windows
JVM
TCP/IP
MPI
.Net Runtime
VPN
SSH
Slide Courtesy David Abramson
17
Defining the Grid

• A Grid is the combination of networked resources
  and the corresponding Grid middleware, which
  provides Grid services for the user.
• Status of EGEE-II (as of July 8, 2006)

18
EGEE Applications

• gt20 applications
• High Energy Physics
• Biomedicine
• Earth Sciences
• Computational Chemistry
• Astronomy
• Geo-Physics
• Financial Simulation
• Fusion
• Further applications in evaluation

Applications now moving from testing to routine
and daily usage
19
High Energy Physics

• Large Hadron Collider (LHC)
• One of the most powerful instruments ever built
  to investigate matter
• 4 Experiments ALICE, ATLAS, CMS, LHCb
• 27 km circumference tunnel
• Due to start up in 2007

20
Applications Example WISDOM

• Grid-enabled drug discovery process for neglected
  diseases
• In silico docking
• compute probability that potential drugs dock
  with target protein
• To speed up and reduce cost to develop new drugs
• WISDOM (World-wide In Silico Docking On Malaria)
• First biomedical data challenge
• 46 million ligands docked in 6 weeks
• Target proteins from malaria parasite
• Molecular docking applications Autodock and
  FlexX
• 1 million virtual ligands selected
• 1TB of data produced
• 1000 computers in 15 countries
• Equivalent to 80 CPU years
• Significant results
• Best hits to be re-ranked using Molecular
  Dynamics

New data challenge planned for autumn 2006
21
Example Avian flu

• Avian Flu H5N1
• H5 and N1 proteins on virus surface
• Biological goal of data challenge
• Study in silico the impact of selected point
  mutations on the efficiency of existing drugs
• Find new potential drugs
• Data challenge parameters
• 5 Grid projects Auvergrid, BioinfoGrid, EGEE,
  Embrace, TWGrid
• 1 docking software autodock
• 8 conformations of the target (N1)
• 300 000 selected compounds
• ? gt100 CPU years to dock all configurations on
  all compounds
• Timescale
• First contacts established 1 March 2006
• Data Challenge kick-off 1 April 2006
• Duration 4 weeks

22
Example Determining earthquake mechanisms

• Seismic software application determines
  epicentre, magnitude, mechanism
• Analysis of Indonesian earthquake (28 March
  2005)
• Seismic data within 12 hours after the earthquake
• Solution found within 30 hours after earthquake
  occurred
• 10 times faster on the Grid than on local
  computers
• Results
• Not an aftershock of December 2004 earthquake
• Different location (different part of fault line
  further south)
• Different mechanism
• ? Rapid analysis of earthquakes important for
  relief efforts

23
EGEE-II Overview

• Status of EGEE-II
• (as of July 5, 2006)
• Grid Resources
• Grid Middleware
• Grid Applications

BUT
24
EGEE-II Overview
25
EGEE and Sustainability

• BUT
• How does EGEE compare to other computing
  infrastructures?
• Number of infrastructure users?
• Number of application domains?
• Number of computing nodes?
• Number of years in service?
• What would happen, if EGEE is turned off?
• What happens after April 2008 (End of EGEE-II)?

26
The Future of Grids

• Increasing the number of infrastructure users by
  increasing awareness
• Dissemination and outreach
• Training and education
• Increasing the number of applications by
  improving application support and middleware
  functionality
• High level grid middleware extensions
• Increasing the grid infrastructure
• Incubating related projects
• Ensuring interoperability between projects
• Protecting user investments
• Towards a sustainable grid infrastructure

27
User Information Support

• More than 170 training events and summer schools
  across many countries
• gt3000 people trained
• induction application developer advanced
  retreats
• Material archive online with 250 presentations
• Public and technical websites
• Dissemination material
• ? constantly evolving to expand information and
  keep it up to date
• 4 conferences organized ( 460 _at_ Pisa)
• Next conference September 2006 in Geneva 600
  participants

28
Industry and EGEE-II

• Industry Task Force
• Group of industry partners in the project
• Links related industry projects (NESSI, BEinGRID,
  )
• Works with EGEEs Technical Coordination Group
• Collaboration with CERN openlab project
• IT industry partnerships for hardware and
  software
• development
• EGEE Business Associates (EBA)
• Companies sponsoring work on joint-interest
  subjects
• Industry Forum
• Led by Industry to improve Grid take-up in
  Industry
• Organises industry events and disseminates grid
  information

29
The Future of Grids

• Increasing the number of infrastructure users by
  increasing awareness
• Dissemination and outreach
• Training and education
• Increasing the number of applications by
  improving application support and middleware
  functionality
• High level grid middleware extensions
• Increasing the grid infrastructure
• Incubating related projects
• Ensuring interoperability between projects
• Protecting user investments
• Towards a sustainable grid infrastructure

30
Building Software for the Grid
Upper Middleware Tools
Lower Middleware
Courtesy IBM,
Bonds
Platform Infrastructure
Unix
Windows
JVM
TCP/IP
MPI
.Net Runtime
VPN
SSH
Slide Courtesy David Abramson
31
Portals on EGEE

• 
  P-Grade
• 
  Genius

32
High Level Middleware Extensions

• Example
• Understanding data through graphical
  representations ? Scientific Visualization
• Approach Grid Visualization Kernel (GVK)
• Interactive visualization service in the grid
• Integrated in existing visualization toolkits
• Optimized transportation and processing
• Research project
• Interactivity on the Grid ? glogin
• Batch-bypass technology
• Easy to use/install
• Secure

33
glogin - Interactive Tunneling
WorkerNode
Gatekeeper
Client
Point ofContact
glogin
WorkerNode
WorkerNode
glogin
Traffic Forwarding
socket
WorkerNode
WorkerNode
on the Grid
34
glogin Shell

• Interactive access to grid nodes
• Authentication via grid certificates
• Tunneling of arbitary traffic

35
Grid Visualization Kernel
Workernode
GridVisualization Kernel
Workernode
WORKING!
Workernode
glogin
Client
Workernode
glogin
on the Grid
36
Example Biomedicine

• Parallel simulationof blood flowon the Grid
• Onlinevisualizationof simulationresults on
  thedesktop
• Interactivesteering ofsimulation
• Grid isinvisible

Cooperation with University Amsterdam
37
Example Flooding Crisis Support

• Simulation of floodingon the Grid
• Onlinevisualizationof simulationresults in
  theCAVE
• Interactivesteering ofsimulation
• Grid isinvisible

Cooperation with Slowak Academy of Sciences
38
Scientific Visualization

• Use your favourite device to connect to the Grid
• Sony PSP PlayStation Portable

39
Building Software for the Grid
Upper Middleware Tools
Lower Middleware
Courtesy IBM,
Bonds
Platform Infrastructure
Unix
Windows
JVM
TCP/IP
MPI
.Net Runtime
VPN
SSH
Slide Courtesy David Abramson
40
The Future of Grids

• Increasing the number of infrastructure users by
  increasing awareness
• Dissemination and outreach
• Training and education
• Increasing the number of applications by
  improving application support and middleware
  functionality
• High level grid middleware extensions
• Increasing the grid infrastructure
• Incubating related projects
• Ensuring interoperability between projects
• Protecting user investments
• Towards a sustainable grid infrastructure

41
Projects related to EGEE
42
Related Infrastructures
43
The Future of Grids

• Increasing the number of infrastructure users by
  increasing awareness
• Dissemination and outreach
• Training and education
• Increasing the number of applications by
  improving application support and middleware
  functionality
• High level grid middleware extensions
• Increasing the grid infrastructure
• Incubating related projects
• Ensuring interoperability between projects
• Protecting user investments
• Towards a sustainable grid infrastructure

44
Sustainability Beyond EGEE-II

• Need to prepare for permanent Grid infrastructure
• Maintain Europes leading position in global
  science Grids
• Ensure a reliable and adaptive support for all
  sciences
• Independent of project funding cycles
• Modelled on success of GÉANT
• Infrastructure managed centrally in collaboration
  with national bodies (in EGEE-II JRUs)

45
Grids in Europe

• Great investment in developing Grid technology
• Sample of National Grid projects
• Austrian Grid Initiative
• DutchGrid
• France Grid5000
• Germany D-Grid Unicore
• Greece HellasGrid
• Grid Ireland
• Italy INFNGrid GRID.IT
• NorduGrid
• Swiss Grid
• UK e-Science National Grid Service OMII
  GridPP
• EGEE provides framework for national, regional
  and thematic Grids

46
Evolution
European e-InfrastructureCoordination
47
Summary

• Grids represent a powerful new tool for science
• ?Today we have a window of opportunity to move
  grids from research prototypes to permanent
  production systems (as networks did a few years
  ago)
• EGEE offers
• a mechanism for linking together people,
  resources and data of many scientific community
• a basic set of middleware for gridfying
  applications with documentation, training and
  support
• regular forums for linking with grid experts,
  other communities and industry

48
Summary

• Success will lead to the adoption of grids as the
  main computing infrastructure for science
• If we succeed then the potential return to
  international scientific communities will be
  enormous and open the path for commercial and
  industrial applications

49
EGEE06 Conference

• EGEE06 Capitalising on e-infrastructures
• Demos
• Related Projects
• Industry
• International community (UN organisations in
  Geneva etc.)
• 25-29 September 2006
• Geneva, Switzerland
• http//www.eu-egee.org/egee06