NIKHEFSARA Hardware

1
DutchGrid
Large Scale Distributed Computingin the
Netherlands
www.dutchgrid.nl
2
Grid Computing

• Grids are accepted world wide
• As the next step in distributed computing
• Mainly focussed on computing grids
• We (HEF and many others) need data grids
• LHC has embraced Grids for data analysis
• LCG has been launched
• Encompasses EU, US and AP
• NIKHEF is way out in front
• Were present from the beginning
• Prime partner in European DataGrid Project
• DutchGrid node most important with CERN and RAL

3
What we recently achieved

• ATLAS Data Challenge Task Force
• Produced significant k events on the EDG testbed
• NIKHEF was instrumental to get the softw. working
• Running on 24 cpus with a 0.5 TByte data cache
• LHCb produced many k events
• Cpu farm ( 24 cpus) at the Vrije Universiteit
• Maintained from NIKHEF
• D0 Monte Carlo Challenge
• Ongoing since almost 2 years now
• Produced 5M events PythiaGEANTrecoanalysis
• 100 cpus, 2 TByte disk cache, data transport to
  Fermi
• Developped softw for routine farm admin.

4
The scale of Nikhef/Atlas Comp.

• ATLAS will take data from 2007 onwards
• 100 Hz output of L3, 1 Mbyte per event
• 100 MB/sec, 107 sec/yr ? 1 Pbyte/yr
• RecoAnalysis at 1 event/1.5 minute
• To keep up with raw data need 10,000 cpus
• MC RecoAnalysis 1 event/5 minutes
• To simulate 30 we need another 10,000 cpus
• And storage for another Pbyte of data each year
• NL is 5 (Tier )
• Need to provide 1000 cpus for ATLAS
• And 100 Tbyte of backup storage per year
• And 10 Tbyte of disk cache per year (10 of all
  data)

5
The scale of Nikhef computing

• Other Nikhef/LHC experiments LHCb, Alice
• Need to provide 2000 cpus for LHC experiments
• D0, BaBar, Star, Antares, Zeus, Theory,
• Resources needed now and not only in 2007
• Total
• 2002 300 cpus, 10 Tbyte disk, 100 Tbyte tape
• 2007 3000 cpus, 50 Tbyte disk, 500 Tbyte tape
• Numbers not accurate within a factor of 2
• Need for cpu with time rises with their power
• Storage needs are difficult to predict

6
Dutch LHC Tier Hardware

• Farm size
• now 150 dual CPU nodes scavenging 200 nodes
• buildup to 1500 nodes in 2007

• Permanent Storage
• now 10 Tbyte
• buildup to 500 Tbyte in 2007
• Then after 100 Tbyte/yr
• Disk cache 10

7
The network needs

• In 1995 we thought there would not be enough
• 1 Gbit/sec links seemed a dream for 2007
• We now think there will be enough in 2007
• Now (2002) we have 2.5 Gbit/sec to CERN
• The new technology is optical networking
• And 100 Gbit/sec is quite likely in 2007
• Which is more than the raw data rate of Atlas (1
  Gb/s)
• This may change the whole picture again
• Atlas L3 input rate is 5 GByte/s (5 kHz, 1
  Mbyte/ev.)
• So 40 Gbit/s network needed to do L3 at other
  sites
• But then you could also store raw data at other
  sites

8
History of the SURFnet infrastructure
1 Tbit/s
2007
9
SURFnet5 in full operation
500,000 users 49 institutes connected at
Gbit/s End 2002 80 institutes Test network in
development
10
SURFnet International
2.5 Gbit/s
10 Gbit/s Tyco
New York
Amsterdam NetherLight
Dwingeloo ASTRON
10 Gbit/s Level3
DWDM SURFnet
2.5 Gbit/s SURFnet
2.5 Gbit/s SURFnet
Chicago StarLight
2.5 Gbit/s CERN
Geneve CERN
11
What it will cost for 2007

• Cpu power 4M
• 1 k for 1 cpu (now 2 GHz, 0.5 GB RAM, 60 GB
  disk)
• 3 M for 3000 cpus in 2007
• 1 M Infrastructure (racks, power, cooling,
  cables, ..)
• Data servers and Disk Cache 1 M
• 20 k for 1 Tbyte of raid disk
• 1 M for 50 Tbyte of disk in 2007
• Permanent Storage 1.5 M
• 1 k for 1 Tbyte (200 Gbyte tape costs 200 )
• 0.5 M for tapes 500 Tbyte in 2007
• 1.0 M for infrastructure (tape robot)
• Network Infrastructure 1M
• 1 M for routers and switches
• Not counting extra manpower

12
General Dutch Grid Infrastructure

• 15 M investment for 2007
• Ignore spending profile
• 7.5 M for grid computing in general
• 7.5 M for computing for HEP
• General grid infrastructure in The Netherlands
• For e-science (biology, earth observation,
  astronomy,..)
• For public sector (hospitals, government, ..)
• For commerce (telecom, car/airplane industry, )
• With other partners
• Surfnet (Gigaport) for network infrastructure
• SARA for Mass Storage
• Computer Science Research System DAS
• Virtual Laboratory for e-Science Project

13
Dutch Projects and Resources

• ICES/KIS III (min.of commerce)
• For infrastructure project in The Netherlands
• Call for proposals very soon now, money for
  2003/4
• 20 partners, requested budget 25 M (50)
• Industry involvement (IBM, KPN, Phillips,
  Unilever)
• NWO (min.of science)
• NOW-groot proposals for big projects (like Atlas
  det.)
• Projects typically 5 -10 M
• NCF (part of NWO)
• Flankerend beleid 0.5 M spend in 2002
• Supervises budget for super computing (5 M/yr)

14
DutchGrid in European DataGrid

• EU IST Project
• 10 M for personnel for Grid middleware
  development
• 10 M matched funding by the partners
• 200 people project, 21 partners, biggest EU IST
  project
• Nikhef one of the 6 principle partners
• Others are CRNS, INFN, PPARC, CERN, ESA
• Other sciences are biology and earth observation
• Industrial participation by IBM, Datamat, CS, ..
• Dutch associated partners are SARA and KNMI
• NL participates in
• Fabric management (large clusters of cpus and
  disks)
• Mass storage management
• Testbed and security
• Network management
• Applications (HEP and EO)
• Project management
• Nikhef is a core site
• With CERN, RAL, CCIN2P3,CNAF
• Expertise centre

15
New EU proposal EGEE in FP6

• 300 M available for grid infrastr. and network
  in Europe
• EoI submitted for 100 M for Grid Infrastructure
  in Europe
• Leading science is HEP
• Many more partners
• Almost all EU countries, national grid projects,
  computer centers
• Many more associated partners and industries
• EU wants to counter balance what happens in US
  and AP
• NL likely goes together with Scandinavians
• NorduGrid very much like DutchGrid (also similar
  style people)
• NorduNet very well connected to SurfNet
• Together we may become a principal partner in
  this project
• Common expertise in networking, security, fabric
  management

16
NorduGrid
Lund Univeristy NBI, Copenhagen Oslo
University Uppsala University Bergen
University SCFAB, Stockholm
2 M for 2003-2004 15 M for 2005-2008
Biomedical sciences Earth sciences Space and
astro sciences High Energy Physics
17
DutchGrid
Nikhef partners SARA Univ.of NL KNMI ESA/ESTEC T
elematica
2 M for 2003-2005 15 M for 2006-2008
Biomedical sciences Earth sciences Space and
astro sciences High Energy Physics
18
Researchdeployment areas

• Security Architectures and deployable security
• Fabric Gridification (EDG)
• AAAARCH research group (IRTF)
• Grid Accessibility and Portals
• Virtual Laboratory for E-Science (VL-E project)
• Federated Information Management (WTCW VL
  project)
• Grid Fabric Research
• On-demand installation and configuration (NCF
  NIKHEF)
• Storage Area Network (SARA)
• Networking
• Optical networking (SURFnet DataTag)
• Applications
• Atlas, Alice, D0-Grid, LHCb (NIKHEF partners)
• Materials analysis lab, DNA micro array, BioASP
  (WTCW VL)
• Earth observation, Atronomy (KNMI, ESTEC, Astron)

19
Back to particle physics

• Large investments are difficult for HEP alone
• Money from EU only for e-science in general
• Similar feelings at a national level in most EU
  countries
• Projects driven by particle physics
• CERN is well known for big science projects
• We need it for the LHC and will invest ourselves
• Had an excellent review for the Eur. DataGrid
  project
• Profit from other resources
• Cycle scavenging for MC production
• Infrastructures such as SARA

20
Back to CERN

• LCG project launched
• Focussed on data analysis for the LHC
• Includes US and AP
• Focussed on deployment for the data challenges
• HEPCAL RTAG
• Lead by Nikhef collaborator
• Regional Centres RTAG
• Nikhef participated
• Grid Deployment Board
• Not clear yet w.r.t. EDG testbed
• WG identified, not started yet
• Many issues to be clarified
• Between LCG, EDG, EGEE and National Grids

21
Back to Nikhef Conclusions

• NIKHEF is well established in the world of Grids
• Prime player in EDG
• In for EGEE together with Nordic countries
• DutchGrid for e-science on national scale
• Need medium term investment of 2 M
• For running HEP exeriments
• For participating in LCG and LHC data challenges
• For RD in grid middleware and infrastructure
• Need large investment of 15 M for 2007
• For general grid infrastructure for e-science in
  NL
• Particle physics as driving force
• Spending profile to be defined