Enabling eScience Grid Technologies Today

1
Enabling eScienceGrid Technologies Today
Tomorrow

• Ian Foster
• Argonne National LaboratoryUniversity of
  ChicagoUniva Corporation

eScience n Large-scale science carried out
through distributed collaborationsoften
leveraging access to large-scale data computing
2
Overview

• The need for Grid technology
• Grid technology today
• Future challenges directions
• Summary

3
Science Today is a Team Sport
4
NEES Network for Earthquake Engineering
Simulation
Links instruments, data, computers, people
5
The Application-Infrastructure Gap

• Dynamicand/orDistributedApplications

6
Bridging the GapGrid Technology
Users

• Service-oriented applications
• Wrap applications asservices
• Compose applicationsinto workflows
• Service-orientedinfrastructure
• Provision physicalresources to support
  application workloads

Composition
Workflows
Invocation
ApplnService
ApplnService
Provisioning
7
Overview

• The need for Grid technology
• Grid technology today
• Future challenges directions
• Summary

8
Grid TechnologyService-Oriented Infrastructure
User Application
User Application
User Application
Database
Specialized resource
Computers
Storage
9
Web ServicesStandards, Tools, Interoperability
10
Authentication Authorization
Users
11
The Many Dimensions of Scaling
Disk-to-disk onTeraGrid
Local DB Update send (secs) Bloom filter (secs) Bloom filter (bits)
10K lt1 2 1 M
1 M 2 24 10 M
5 M 7 175 50 M
gt2 Gbyte/s
GridFTP High-performance, reliable data movement
Replica location LIGO 40M files, 10 sites
12
ScalingGrid2003Workflows
Genome sequence analysis
Sloan digital sky survey
Physics data analysis
13
Application Examples
Fusion Grid 1000s of jobs
14
GlobalCommunity
15

• 26776 U.S.
• 2753 China
• 1318 Japan
• 1017 India
• 750 U.K.
• 495 Italy
• 488 Germany
• 391 Brazil
• 328 S. Korea
• 306 Taiwan
• 268 France
• 241 Canada
• 211 Viet Nam
• 211 Spain
• 202 Russia
• 187 Pakistan
• 159 Australia
• 142 Singapore
• 131 Greece

68 Belgium 66 Venezuela 66 Romania 64
Indonesia 62 Mexico 61 Turkey 60 Malaysia 58
Portugal 57 Austria 54 Ireland 44 Hong Kong
40 Hungary 38 Egypt 38 Argentina 34 Uruguay
31 Ukraine 29 Slovakia 25 Israel 23
Yugoslavia 23 Iran 22 Bulgaria 22 Uzbekistan
22 Czech Rep. 22 N. Korea 21 Lithuania 21
Croatia
20 Finland 20 New Zealand 17 Nigeria 17 South
Africa 16 Jordan 16 Slovenia 16 Afghanistan
15 Denmark 15 Philippines 14 Vanuatu 14
Luxembourg 14 Tunisia 12 Virgin Is. (U.K.) 12
Peru 12 Yemen 11 Norway 11 Algeria 11 Armenia
10 Iceland 9 Zambia 9 Virgin Is. (U.S.) 9
Uganda 9 Bosnia Herz. 8 Kenya 7 Zimbabwe 7
Saudi Arabia
7 Ecuador 7 Macedonia 6 Bolivia 6 Comoros 6
Zaire 6 Lebanon 5 Puerto Rico 5 Namibia 5
Togo 5 Tajikistan 5 Paraguay 5 Albania 5
Sudan 4 Estonia 4 Camaroon 4 Ghana 4 Tuvalu
4 Costa Rica 4 Cuba 4 UAE 4 Tonga 4 W. Samoa
4 Tanzania 3 Syria 3 Bahamas 3 Ethiopia
3 Mongolia 3 Sri Lanka 3 Wallis Futuna Is. 3
Belarus 3 Bangladesh 2 Falkland Islands 2
Kuwait 2 Sierra Leone 2 Trinidad Tobago 2
Guyana 2 American Samoa 2 Andorra 2 Georgia 2
Cook Islands 2 Turkmenistan 2 Gabon 2 The
Gambia 2 Kazakhstan 2 Macau 2 Malta 2 Jamaica
2 Latvia 2 Turks Caicos 1 Bhutan 1 Ascension
Island 1 Cyprus
1 Mozambique 1 Tokelau 1 Greenland 1 Nepal 1
Swaziland 1 Iraq 1 Serbia 1 Barbados 1
Cambodia 1 Qatar 1 Saint Vincent 1 Laos 1 San
Marino 1 Libya 1 Benin 1 Angola 1 Chad 1
Gibraltar 1 Haiti 1 Guatemala 1 Malawi 1
Equatorial Guinea 1 Palau 1 Bermuda 1 Botswana
1 Suriname
38669 downloads in 2004 from globus.org
16

• 26776 U.S.
• 2753 China
• 1318 Japan
• 1017 India
• 750 U.K.
• 495 Italy
• 488 Germany
• 391 Brazil
• 328 S. Korea
• 306 Taiwan
• 268 France
• 241 Canada
• 211 Viet Nam
• 211 Spain
• 202 Russia
• 187 Pakistan
• 159 Australia
• 142 Singapore
• 131 Greece

68 Belgium 66 Venezuela 66 Romania 64
Indonesia 62 Mexico 61 Turkey 60 Malaysia 58
Portugal 57 Austria 54 Ireland 44 Hong Kong
40 Hungary 38 Egypt 38 Argentina 34 Uruguay
31 Ukraine 29 Slovakia 25 Israel 23
Yugoslavia 23 Iran 22 Bulgaria 22 Uzbekistan
22 Czech Rep. 22 N. Korea 21 Lithuania 21
Croatia
20 Finland 20 New Zealand 17 Nigeria 17 South
Africa 16 Jordan 16 Slovenia 16 Afghanistan
15 Denmark 15 Philippines 14 Vanuatu 14
Luxembourg 14 Tunisia 12 Virgin Is. (U.K.) 12
Peru 12 Yemen 11 Norway 11 Algeria 11 Armenia
10 Iceland 9 Zambia 9 Virgin Is. (U.S.) 9
Uganda 9 Bosnia Herz. 8 Kenya 7 Zimbabwe 7
Saudi Arabia
7 Ecuador 7 Macedonia 6 Bolivia 6 Comoros 6
Zaire 6 Lebanon 5 Puerto Rico 5 Namibia 5
Togo 5 Tajikistan 5 Paraguay 5 Albania 5
Sudan 4 Estonia 4 Camaroon 4 Ghana 4 Tuvalu
4 Costa Rica 4 Cuba 4 UAE 4 Tonga 4 W. Samoa
4 Tanzania 3 Syria 3 Bahamas 3 Ethiopia
3 Mongolia 3 Sri Lanka 3 Wallis Futuna Is. 3
Belarus 3 Bangladesh 2 Falkland Islands 2
Kuwait 2 Sierra Leone 2 Trinidad Tobago 2
Guyana 2 American Samoa 2 Andorra 2 Georgia 2
Cook Islands 2 Turkmenistan 2 Gabon 2 The
Gambia 2 Kazakhstan 2 Macau 2 Malta 2 Jamaica
2 Latvia 2 Turks Caicos 1 Bhutan 1 Ascension
Island 1 Cyprus
1 Mozambique 1 Tokelau 1 Greenland 1 Nepal 1
Swaziland 1 Iraq 1 Serbia 1 Barbados 1
Cambodia 1 Qatar 1 Saint Vincent 1 Laos 1 San
Marino 1 Libya 1 Benin 1 Angola 1 Chad 1
Gibraltar 1 Haiti 1 Guatemala 1 Malawi 1
Equatorial Guinea 1 Palau 1 Bermuda 1 Botswana
1 Suriname
Top 10
26776 U.S. 2753 China 1318 Japan 1017
India 750 U.K. 495 Italy 488
Germany 391 Brazil 328 S. Korea 306
Taiwan
38669 downloads in 2004 from globus.org
17
Overview

• The need for Grid technology
• Grid technology today
• Future challenges directions
• Summary

18
Theres Much More to DoFor Example, U.S. Dept
of Energy

• Fermi National Accelerator Lab
• Tevatron

• MIT
• Bates Accelerator Center
• Plasma Science Fusion Center

• Pacific Northwest National Lab
• Environmental Molecular Sciences Lab

• Univ. of IL
• Electron Microscopy Center for Materials Research
  
• Center for Microanalysis of Materials

• Argonne National Lab
• Intense Pulsed Neutron Source
• Advanced Photon Source
• Argonne Tandem Linac Accelerator System

• Stanford Linear Accelerator Center
• B-Factory
• Stanford Synchrotron Radiation Laboratory

• Brookhaven
• National Lab
• Relativistic Heavy Ion Collider
• National Synchrotron Light Source

• Lawrence Berkeley
• National Lab
• Advanced Light Source
• National Center for Electron Microscopy
• National Energy Research Scientific Computing
  Facility

Princeton Plasma Physics Lab

• Thomas Jefferson National
• Accelerator Facility
• Continuous Electron Beam Accelerator Facility

• Oak Ridge National Lab
• High-Flux Isotope Reactor Surface Modification
  Characterization Center
• Spallation Neutron Source
• (under construction)

Sandia Combustion Research Facility
General Atomics - DIII-D Tokamak
James R. MacDonald Laboratory
Los Alamos Neutron Science Center
Physics Accelerators Synchrotron Light
Sources Neutron Sources Special Purpose
Facilities Large Fusion Experiments
SC Laboratories
SC User Facilities
Institutions that Use SC Facilities
19
Theres Much More to DoE.g., U.S. Dept of Energy

• Goal Any DOE scientist can access any DOE
  computer, software, data, instrument
• 25,000 scientists (vs. 1000 DOE certs)
• 1000 instruments (vs. maybe 10 online?)
• 1000 scientific applns (vs. 2 Fusion services)
• 10 PB of interesting data (vs. 100TB on ESG)
• 100,000 computers (vs. 3000 on Grid3)
• Not to mention many external partners
• I.e., we need to scale by 2-3 orders of
  magnitude to have DOE-wide impact!

Rough estimate WAG
20
Scaling UpService-Oriented Science
Simulation code
Expt design
Simulation code
Content
Expt output
Certificate authority
Electronic notebook
Telepresence monitor
Simulation server
Services
Portal server
Data archive
Metadata catalog
Resources
Servers, storage, networks
Experimental apparatus
21
Establishing Policy Across Virtual Organizations
ATA Attribute Authority AZA Authorization
Authority
Subject ATA
Resource AZA
Subject AZA
Resource ATA
WS Resource
WS-Subject
GT4-enabled services such as CAS, VOMS, GridShib
are initial steps in this direction
22
Overview

• The need for Grid technology
• Grid technology today
• Future challenges directions
• Summary

23
Enabling eScience

• eScience methods no longer optional but now
  vital to scientific competitiveness
• Weve demonstrated feasibility, but now need to
  scale up
• Meanwhile, others are investing heavily
• EU 70M/yr for Grid infrastructure, tech
• UK 60M/yr for eScience apps and tech
• Germany, Italy, Netherlands, Japan, China, Korea,
  Singapore, Australia all have programs
• ? How does the U.S. stay competitive?

24
Acknowledgments

• Carl Kesselman and Steve Tuecke, my long-time
  Globus co-conspirators
• Gregor von Laszewski, Kate Keahey, Jennifer
  Schopf, Mike Wilde, other Argonne colleagues
• Globus Alliance members at Argonne, U.Chicago,
  USC/ISI, Edinburgh, NCSA, PDC
• Miron Livny, U.Wisconsin Condor project, Rick
  Stevens, Argonne U.Chicago
• Other partners in Grid technology, application,
  infrastructure projects
• DOE SciDAC program, NSF Middleware Initiative,
  NASA, IBM for generous support

25
For More Information

• Globus Alliance
• www.globus.org
• Globus Consortium
• www.globusconsortium.com
• Global Grid Forum
• www.ggf.org
• Open Science Grid
• www.opensciencegrid.org
• Background information
• www.mcs.anl.gov/foster

2nd Edition www.mkp.com/grid2