Title: Vladimir Litvin, Harvey Newman, Sergey Schevchenko
1Using of Grid Prototype Infrastructure for QCD
Background Study to the H ? ?? Process on
Alliance Resources
- Vladimir Litvin, Harvey Newman, Sergey
Schevchenko - Caltech CMS
- Scott Koranda, Bruce Loftis, John Towns
- NCSA
- Miron Livny, Peter Couvares, Todd Tannenbaum,
Jamie Frey - Wisconsin Condor
2CMS Physics
- The CMS detector at the LHC will probe
fundamental forces in our Universe and search for
the yet undetected Higgs Boson - Detector expected to come online 2007
3CMS Physics
4Leveraging Alliance Grid Resources
- The Caltech CMS group is using Alliance Grid
resources today for detector simulation and data
processing prototyping - Even during this simulation and prototyping phase
the computational and data challenges are
substantial
5Goal to simulate QCD background
- The QCD jet-jet background cross section is huge
( 1010 pb). Previous studies of QCD jet-jet
background have got the estimation of the rate,
Rjet , when jet might be misidentified as photon
and, due to the limited CPU power, for QCD
jet-jet background rates where simply squared
(Rjet2). Hence, the correlations within event
have not been taken into account in previous
studies - Previous simulations have been done with
simplified geometry and non-gaussian tails in the
resolution have not been adequately simulated - Our goal is to make full simulation of relatively
large QCD sample, measure the rate of diphoton
misidentification and compare it with other types
of background
6Generation of QCD background
- QCD jet cross section strongly depends on the pT
of the parton in hard interaction - QCD jet cross section is huge. We need reasonable
preselection at the generator level before pass
events through full detector simulation
- The optimal choice of pT is needed
- Our choice is pT 35GeV
- pT 35 GeV is safe cut we do not lose
significant fraction of events, which could fake
the Higgs signal, at the preselection level
7Generator level cuts
- QCD background
- Standard CMS cuts Et1gt40 GeV, Et2gt25 GeV,
?1,2lt2.5 - at least one pair of any two neutral particles
(?0, ?, e, ?, ?, ?, ?) with - Et1 gt 37.5 GeV
- Et2 gt 22.5 GeV
- ?1,2 lt 2.5
- minv in 80-160 GeV
- Rejection factor at generator level 3000
- Photon bremsstrahlung background
- Standard CMS cuts Et1gt40 GeV, Et2gt25 GeV,
?1,2lt2.5 - at least one neutral particle (?0, ?, e, ?, ?,
?, ?) with - Et gt 37.5 GeV
- ? lt 2.5
- Rejection factor at generator level 6
8Challenges of a CMS Run
- CMS run naturally divided into two phases
- Monte Carlo detector response simulation
- 100s of jobs per run
- each generating 1 GB
- all data passed to next phase and archived
- reconstruct physics from simulated data
- 100s of jobs per run
- jobs coupled via Objectivity database access
- 100 GB data archived
- Specific challenges
- each run generates 100 GB of data to be moved
and archived - many, many runs necessary
- simulation reconstruction jobs at different
sites - large human effort starting monitoring jobs,
moving data
9Tools
- Generation level - PYTHIA 6.152 (CTEQ 4L
structure functions) http//www.thep.lu.se/torbjo
rn/Pythia.html - Full Detector Simulation - CMSIM 121 (includes
full silicon version of the tracker)
http//cmsdoc.cern.ch/cmsim/cmsim.html - Reconstruction - ORCA 5.2.0 with pileup at L 2
1033 cm-2/s (30 pileup events per signal event)
- http//cmsdoc.cern.ch/orca
10Analysis Chain
11Meeting Challenge With Globus and Condor
- Globus
- middleware deployed across entire Alliance Grid
- remote access to computational resources
- dependable, robust, automated data transfer
- Condor
- strong fault tolerance including checkpointing
and migration - job scheduling across multiple resources
- layered over Globus as personal batch system
for the Grid
12CMS Run on the Alliance Grid
- Caltech CMS staff prepares input files on local
workstation - Pushes one button to launch master Condor job
- Input files transferred by master Condor job to
Wisconsin Condor pool (700 CPUs) using Globus
GASS file transfer
Caltech workstation
Input files via Globus GASS
WI Condor pool
13CMS Run on the Alliance Grid
- Master Condor job at Caltech launches secondary
Condor job on Wisconsin pool - Secondary Condor job launches 100 Monte Carlo
jobs on Wisconsin pool - each runs 1224 hours
- each generates 1GB data
- Condor handles checkpointing migration
- no staff intervention
14CMS Run on the Alliance Grid
- When each Monte Carlo job completes data
automatically transferred to UniTree at NCSA - each file 1 GB
- transferred using Globus-enabled FTP client
gsiftp - NCSA UniTree runs Globus-enabled FTP server
- authentication to FTP server on users behalf
using digital certificate
100 Monte Carlo jobs on Wisconsin Condor pool
100 data files transferred via gsiftp, 1 GB each
NCSA UniTree with Globus-enabled FTP server
15CMS Run on the Alliance Grid
- When all Monte Carlo jobs complete Secondary
Condor reports to Master Condor at Caltech - Master Condor at Caltech launches job to stage
data from NCSA UniTree to NCSA Linux cluster - job launched via Globus jobmanager on cluster
- data transferred using Globus-enabled FTP
- authentication on users behalf using digital
certificate
Master starts job via Globus jobmanager on
cluster to stage data
16CMS Run on the Alliance Grid
- Master Condor at Caltech launches physics
reconstruction jobs on NCSA Linux cluster - job launched via Globus jobmanager on cluster
- Master Condor continually monitors job and logs
progress locally at Caltech - no user intervention required
- authentication on users behalf using digital
certificate
Master starts reconstruction jobs via Globus
jobmanager on cluster
17CMS Run on the Alliance Grid
- When reconstruction jobs complete data
automatically archived to NCSA UniTree - data transferred using Globus-enabled FTP
- After data transferred run is complete and Master
Condor at Caltech emails notification to staff
data files transferred via gsiftp to UniTree for
archiving
18Condor Details for Experts
- Use CondorG
- Condor Globus
- allows Condor to submit jobs to remote host via a
Globus jobmanager - any Globus-enabled host reachable (with
authorization) - Condor jobs run in the Globus universe
- use familiar Condor classads for submitting jobs
universe globus globusscheduler
beak.cs.wisc.edu/jobmanager-
condor-INTEL-LINUX environment
CONDOR_UNIVERSEscheduler executable
CMS/condor_dagman_run arguments -f -t -l .
-Lockfile cms.lock -Condorlog
cms.log -Dag cms.dag -Rescue
cms.rescue input CMS/hg_90.tar.gz remote_
initialdir Prod2001 output
CMS/hg_90.out error CMS/hg_90.err log
CMS/condor.log notification
always queue
19Condor Details for Experts
- Exploit Condor DAGman
- DAGdirected acyclic graph
- submission of Condor jobs based on dependencies
- job B runs only after job A completes, job D runs
only after job C completes, job E only after
A,B,C D complete - includes both pre- and post-job script execution
for data-staging, cleanup, or the like
- Job jobA_632 Prod2000/hg_90_gen_632.cdr
- Job jobB_632 Prod2000/hg_90_sim_632.cdr
- Script pre jobA_632 Prod2000/pre_632.csh
- Script post jobB_632 Prod2000/post_632.csh
- PARENT jobA_632 CHILD jobB_632
- Job jobA_633 Prod2000/hg_90_gen_633.cdr
- Job jobB_633 Prod2000/hg_90_sim_633.cdr
- Script pre jobA_633 Prod2000/pre_633.csh
- Script post jobB_633 Prod2000/post_633.csh
- PARENT jobA_633 CHILD jobB_633
20Monte Carlo Samples Simulated and Reconstructed
21CPU timing
22- All cuts except isolation are applied
- Distributions are normalized to Lint 40 pb-1
23Isolation
- Tracker isolation
- Isolation cut Number of tracks with pT gt 1.5 GeV
in R 0.30 cone around photon candidate is zero - Still optimizing pT threshold and cone sizes
- Ecal isolation
- Sum of Et energy in the cone around photon
candidate, using Et energies of ECAL clusters - Isolation cut Sum of Et energy in R 0.30 cone
around photon candidate is less than 0.8 GeV
24Background Cross Section
25Conclusions
- The goal of this study is to increase efficiency
of computer resources and to reduce and minimize
human intervention during simulation and
reconstruction - proof of concept - it is possible to create the
distributed system based on GLOBUS and Condor
(MOP is operational now) - A lot of work ahead in order to make this system
as automatic as possible - Important results are obtained for the Higgs
boson search in two photon decay mode - the main background is the background with one
prompt photon plus bremsstrahlung photon or
isolated ?0 , which is 50 of the total
background. QCD background is reduced down to
the 15 of the total background - More precise studies need much more CPU time