Dutch Datagrid

1
DØ Monte Carlo Challenge

• A HEP Application

2
Outline

• The DØ experiment
• The application
• The NIKHEF DØ farm
• SAM (aka the DØ grid)
• Conclusions

3
The DØ experiment

• Fermi National Accelerator Lab
• Tevatron
• Collides protons and antiprotons of 980 GeV/c
• Run II
• DØ detector
• DØ collaboration
• 500 physicists, 72 institutions, 19 countries

4
The DØ experiment

• Detector Data
• 1,000,000 Channels
• Event size 250KB
• Event rate 50 Hz
• On-line Data Rate 12 MBps
• Est. 2 year totals (incl Processing and
  analysis)
• 1 x 109 events
• 0.5 PB
• Monte Carlo Data
• 5 remote processing centers
• Estimate 300 TB in 2 years.

5
(No Transcript)
6
The application

• Generate events
• Follow particles through detector
• Simulate detector response
• Reconstruct tracks
• Analyse results

7
The application

• Starts with the specification of the events
• Generates (intermediate) data
• Stores data in tape robots
• Declares files in database

8
The application

• consists of
• Monte Carlo programs
• gen, d0gstar, sim, reco, recoanalyze
• mc_runjob
• bunch of python scripts
• runs on
• SGI Origin (Fermilab, SARA)
• Linux farms

9
mc_runjob

• Creates directory structure for job
• Creates scripts for each jobstep
• Creates scripts for submission of metadata
• Creates job description file
• Submit job to batch system

10
The NIKHEF DØ farm

• Batch server (hoeve)
• Boot/Software server
• Runs mc_runjob
• File server (schuur)
• Runs SAM
• 50 70 nodes
• Run MC jobs

11
(No Transcript)
12
node

• At boottime
• Boots via network from batch server
• NFS mounts DØ directories on batch server
• At runtime
• Copies input from batch server to local disk
• Runs MC job steps
• Stores (intermediate) output on local disk

13
File server

• Copies output from node to file server
• Declares files to SAM
• Stores files with SAM in robot
• _at_ fnal
• _at_ sara

14
1.2 TB
fbs(rcp,sam)
mcc request
farm server
SAM DB
file server
fbs job 1 mcc 2 rcp 3 sam
fbs(mcc)
datastore
mcc input
FNAL SARA
mcc output
node
50
control
40 GB
data
metadata
15
SAM _at_ NIKHEF

• Stores metadata in database at FNAL
• sam declare import_ltjobstepgt.py
• scripts prepared by mc_runjob
• Stores files
• on tape at fnal via cache on d0mino
• on disk of teras.sara.nl and migrated to tape
• sam store --descripimport_ltjobstepgt.py
  --destteras.sara.nl/sam/samdata/y01/w42

16
SAM _at_ SARA

• No need to install SAM ?
• Declare teras directories in SAM as destination
• Access protocol
• May 2001 rcp
• October 2001 bbftp
• ?? gridftp

17
SAM on the Global Scale

• Locate files
• Monte Carlo data
• Raw data from detector
• Calibration data
• Accelerator data
• Submit (analysis) jobs on local station
• Stores results in SAM

18
SAM on the Global Scale
Datalogger
MSS
MSS
Reco-farm
Central Analysis
ClueD0
(Others)

• Current active stations
• FNAL (several)
• Lyon FR (IN2P3),
• Amsterdam NL (NIKHEF)
• Lancaster UK
• Imperial College UK
• Others in US

Stations at FNAL
MSS
Interconnected network of primary cache
stations Communicating and replicating data where
it is needed.
19
Future Plans for SAM

• Better specification of remote data storage
  locations, especially in MSS.
• Universal user registration that allows different
  usernames, uid, etc. on various stations.
• Integration with additional analysis frameworks,
  Root in particular (almost ready).
• Event level access to data.
• Movement toward Grid components, GridFTP, GSI

20
Conclusions

• NIKHEF DØ farm is
• easy to use (Antares, L3)
• easy to clone (KUN)
• part of DØ data grid
• moving (slowly) to grid standards

21
L3
antares