Using The EDG Testbed

1
Using The EDG Testbed

• The European DataGrid Project Team
• http//www.eu-datagrid.org

2
Summary

• Use Cases
• High Energy Physics
• Earth Observation
• Biomedical Applications

3
EDG Application Areas
Earth Observation Science Applications
Biomedical Applications

• High Energy Physics

4
High Energy Physics

• 4 Experiments on LHC

CMS
ATLAS
6-8 PetaBytes / year 108 events/year 103
batch and interactive users
LHCb
5
CERNs Network in the World
Europe 267 institutes, 4603 usersElsewhere
208 institutes, 1632 users
6
Data Flow in LHC
7
LHCb EDG Integration

• LHCb
• LHCb distributed computing environment
• Integration of DataGrid middleware
• Authentication
• Job submission to DataGrid
• Monitoring and control
• Data replication
• Resource scheduling use of CERN MSS

8
LHCb

• LHC collider experiment
• 109 events 1Mb 1 Pb
• Need a distributed model
• Create, distribute and keep track of data
  automatically

9
LHCb distributed computing environment
10
LHCb Environment using EDG Middleware
Replica Management
Submit jobs remotely via Web
Transfer data to CASTOR (and HPSS, RAL Datastore)
UserInterface
WMS
MetaDataCatalog
Execute on farm
Update bookkeeping database
Online histogram production using GRID pipes
InformationServices
Monitor performance of farm via Web
Data Quality Check Online
11
1. Authentication

• Issue grid-proxy-init to get a valid user
  certificate.

12
2. Job Submission

• dg-job-submit /home/evh/sicb/sicb/bbincl1600061.jd
  l -o /home/evh/logsub
• bbincl1600061.jdl
• Executable "script_prod"
• Arguments "1600061,v235r4dst,v233r2"
• StdOutput "file1600061.output"
• StdError "file1600061.err"
• InputSandbox "/home/evhtbed/scripts/x509up_u149
  ","/home/evhtbed/sicb/mcsend","/home/evhtbed/sicb/
  fsize","/home/evhtbed/sicb/cdispose.class","/home/
  evhtbed/v235r4dst.tar.gz","/home/evhtbed/sicb/sicb
  /bbincl1600061.sh","/home/evhtbed/script_prod","/h
  ome/evhtbed/sicb/sicb1600061.dat","/home/evhtbed/s
  icb/sicb1600062.dat","/home/evhtbed/sicb/sicb16000
  63.dat","/home/evhtbed/v233r2.tar.gz"
• OutputSandbox "job1600061.txt","D1600063","file
  1600061.output","file1600061.err","job1600062.txt"
  ,"job1600063.txt"

13
3. Monitoring and Control

• dg-job-status
• dg-job-cancel
• dg-job-get-output

14
3. Monitoring and Control
15
3. Monitoring and Control
16
3. Monitoring and Control
17
3. Monitoring and Control
18
Job
Job
19
Job
Job
20
Job
Job
21
Job
Job
22
Job
Job
23
4. Publish data on storage element

• Copy data file to storage element
• globus-url-copy file///chemin/L69999 \
  gsiftp//lxshare0219.cern.ch/flatfiles/SE1/lhcb/L6
  9999
• Register stored data in the catalog
• /opt/globus/bin/globus-job-run lxshare0219.cern.ch
  \/bin/bash -c "export GDMP_CONFIG_FILE/opt/edg/
  lhcb/etc/gdmp.conf \ /opt/edg/bin/gdmp_register_l
  ocal_file -d /flatfiles/SE1/lhcb"
• Publish catalog
• /opt/globus/bin/globus-job-run lxshare0219.cern.ch
  \/bin/bash -c "export GDMP_CONFIG_FILE/opt/edg/
  lhcb/etc/gdmp.conf \ /opt/edg/bin/gdmp_publish_ca
  talogue -n"

24
The ALICE Event
25
The ALICE Event Contd

• ( start_aliroot.sh)
• !/bin/sh
• mv rootrc HOME/.rootrc
• echo "ALICE_ROOT_DIR is set to ALICE_ROOT_DIR"
• export ROOTSYSALICE_ROOT_DIR/root/1
• export PATHPATHROOTSYS/bin
• export LD_LIBRARY_PATHROOTSYS/libLD_LIBRARY_PA
  TH
• export ALICEALICE_ROOT_DIR/aliroot
• export ALICE_LEVEL2
• export ALICE_ROOTALICE/ALICE_LEVEL
• export ALICE_TARGETuname
• export LD_LIBRARY_PATHALICE_ROOT/lib/tgt_ALICE_
  TARGETLD_LIBRARY_PATH
• export PATHPATHALICE_ROOT/bin/tgt_ALICE_TARGE
  TALICE_ROOT/share
• export MANPATHMANPATHALICE_ROOT/man
• ALICE_ROOT/bin/tgt_ALICE_TARGET/aliroot -q -b
  grun.C

----- Job Description for Aliroot
----- author roberto.barbera_at_ct.i
nfn.it Executable "/bin/sh" StdOutput
"aliroot.out" StdError "aliroot.err" InputSand
box "start_aliroot.sh","rootrc","grun.C","Confi
g.C" OutputSandbox "aliroot.err","aliroot.out
","galice.root" RetryCount 7 Arguments
"start_aliroot.sh 3.02.04 3.07.01" Requirements
Member(other.RunTimeEnvironment,"ALICE-3.07.01")

26
Earth Observation Application
Raw satellite data from the GOME instrument
(ESA)
2 different jobs are executed on the TESTBED,
using data provided via the sandbox model
Processing of raw GOME data to ozone
profiles With OPERA (KNMI)
LIDAR data
Validate GOME ozone profiles with Ground Based
measurements (IPSL)
Visualization
27
OPERA application (KNMI)

• From wave spectra measured by the GOME
  instrument on the ERS satellite ozone profiles
  can be calculated. ESA provides these spectra as
  level 1 data. This level 1 data is then processed
  using OPERA to produce ozone profiles, a level 2
  product. The algorithm and s/w (OPERA) are
  developed by KNMI.

GOME takes 30.000 usable measurements for ozone
profile retrieval per day. The calculation of 1
profile takes 2 min on a 800Mhz PIII. One day of
profiles will take 40 days on 1 computer.
28
Validation application (IPSL)

• Produced profiles by OPERA are validated by
  IPSL using ground based LIDAR measurements.
• Since the LIDAR data are in-situ,
  pre-selection of the global GOME data has to be
  performed to create a dataset which is
  geographically and temporally in coincidence.
• The main function of the program is to perform
  statistical operations like the bias between GOME
  and LIDAR data for different altitudes and its
  standard deviations.
• The output of the validation program are 2
  plots, generated by xmgr.

29
Used JDL file

Executable "o3gome-lidar_xmgr.final"StdOu
tput "appli.out"StdError
"appli.err"InputSandbox "/home/leroy/DEMO_190
202/o3gome-lidar_xmgr.final", "/home/leroy/DEMO_19
0202/obs20001019.dat", "/home/leroy/DEMO_190202/ob
s20001002.dat", "/home/leroy/DEMO_190202/obs200010
03.dat", "/home/leroy/DEMO_190202/obs20001004.dat"
, "/home/leroy/DEMO_190202/obs20001005.dat",
"/home/leroy/DEMO_190202/obs20001006.dat",
"/home/leroy/DEMO_190202/select_coinc.exe",
"/home/leroy/DEMO_190202/data_process_demoxmgr",
"/home/leroy/DEMO_190202/oho30010.gol"
OutputSandbox "out_proc.dat","profil_gome.dat
","profil_lidar.dat", "appli.out","appli.err"Re
quirements other.OpSys RH 6.2RetryCount
10Rank other.MaxCpuTime  Th
e produced profiles by OPERA are validated by
IPSL using ground based LIDAR measurements.
One Month of data (gome and lidar data) is used
to do a analysis between the different
measurements The result is visualized using
xmgr.
30
Validation Output
Figure 1 Estimation of the bias between
Gome and Lidar using one month of data.

• Figure 2
• example of 2 profiles Comparison between
  Gome profile and lidar profile for the 2nd
  October 2000.

31
World-Wide Ozone Distribution Mapping
Need for systematic and global mapping of
ozone distribution
GOME
SCIAMACHY
GRID
Large amount of information about atmosphere
gases stored in Terabytes of data
32
Example of Application Description
GRID
33
Further Information

• High Energy Physics
• http//datagrid-wp8.web.cern.ch/DataGrid-WP8/
• Bio-Informatics
• http//marianne.in2p3.fr/datagrid/wp10/index.html
• Earth Observation
• http//styx.esrin.esa.it/grid/