ATLAS HLT/DAQ

1
ATLAS HLT/DAQ
CSN1 Trieste Settembre 2006
V. Vercesi for the ATLAS Italia HLT/DAQ Group
2

• S. Falciano (Roma1) Coordinatore Commissioning
  HLT
• A. Negri (Irvine, Pavia) Coordinatore Event
  Filter Dataflow
• A. Nisati (Roma1) TDAQ Institute Board chair e
  Coordinatore PESA Muon Slice
• F. Parodi (Genova) Coordinatore b-tagging PESA
• V. Vercesi (Pavia) Deputy HLT leader e
  Coordinatore PESA (Physics and Event Selection
  Architecture)
• Attività italiane
• Trigger di Livello-1 muoni barrel (Napoli,
  Roma1, Roma2)
• Trigger di Livello-2 muoni (Pisa, Roma1)
• Trigger di Livello-2 pixel (Genova)
• Event Filter Dataflow (LNF, Pavia)
• Selection software steering (Genova)
• Event Filter Muoni (Lecce, Napoli, Pavia, Roma1)
• DAQ (LNF, Pavia, Roma1)
• DCS (Napoli, Roma1, Roma2)
• Monitoring (Cosenza, Napoli, Pavia, Pisa)
• HLT/DAQ system commissioning and exploitation
  (Everybody)

3
ATLAS TDAQ
Dual(quad)-CPU nodes
500
1600
100
30
Event Filter (EF)
Local Storage SubFarm Outputs (SFOs)
LVL2 farm
Event Builder SubFarm Inputs (SFIs)
Event rate 200 Hz
Second- level trigger
Data storage
SDX1
pROS
DataFlow Manager
Network switches
stores LVL2 output
Network switches
LVL2 Super- visor
Gigabit Ethernet
Event data requests Delete commands
Requested event data
USA15
Event data pulled partial events _at_ 100 kHz,
full events _at_ 3 kHz
UX15
Regions Of Interest
USA15
Data of events accepted by first-level trigger
1600 Read- Out Links
150 PCs
VME
Dedicated links
ATLAS detector
Read- Out Drivers (RODs)
Read-Out Subsystems (ROSs)
First- level trigger
RoI Builder
UX15
Timing Trigger Control (TTC)
Event data pushed _at_ 100 kHz, 1600 fragments of
1 kByte each
4
Level-1 Muon Trigger System
Low-pT 6 GeV Threshold
Efficienza dellalgoritmo di selezione è gt
99 Tutte le inefficienze sono dovute a zone
dello spettrometro non coperte da RPC. In
particolare Settori speciali (Feet), ascensore,
crack ?0 e supporti del magnete.
Muon sources 6 GeV threshold Lumi1033 20 GeV threshold Lumi1034
?/K 9300 1090
b 1620 700
c 943 300
W 3 27
t Negligible Negligible
Efficienza globale Level-1 nel Barrel 83
Low-pT Trigger 79 High-pT Trigger
5
Pre-series system in ATLAS point-1
8 racks (10 of final dataflow, 2 of EF)

Partial Supervr rack- TDAQ rack- 3 HE PCs
One Switch rack- TDAQ rack- 128-port GEth for
L2EB
One Full L2 rack-TDAQ rack- 30 HLT PCs
Partial EFIO rack- TDAQ rack- 10 HE PC(6 SFI
- 2 SFO - 2 DFM)
Partial EF rack- TDAQ rack- 12 HLT PCs
Partial ONLINE rack- TDAQ rack- 4 HLT
PC(monitoring) 2 LE PC(control) 2 Central
FileServers
One ROS rack-TC rack horiz. Cooling- 12
ROS 48 ROBINs
RoIB rack-TC rack horiz. cooling- 50 of
RoIB
surface SDX1
underground USA15

• ROS, L2, EFIO and EF racks one Local File
  Server, one or more Local Switches
• Machine Park Dual Opteron and Xeon nodes,
  uniprocessor ROS nodes
• Operating System Net booted and diskless nodes,
  running SLC3

6
Event Building
TS number of requests from SFI nodes for event
fragments WT number of events processed in
parallel by L2 farm ? L2 accept ratio
Parameterizing observed EB measurements, we
better understand the conditions for stable
performant operations
7
Pre-series studies
8
ROD Crate DAQ

• Last major upgrades
• To accommodate detectors requests
• New API for operational monitoring
• To get coherent operation information
• Simultaneous module readout in a crate
• RCD creates a packet of info and sends it to
  Monitoring
• A monitoring task analyses the packet
• Further improvements in last TDAQ version
• OKS schema for RCD and ROS simplified
• Detector modules in a crate can be managed
• Serially or in multi-threaded mode
• Each state transition can be set as
  multi-threaded
• Can dramatically improve initialization time
• RCD plans
• The RCD structure should be stable enough
• All the main requests from detectors have been
  fulfilled
• We continue supporting the implementations
• Main task is now RCD commissioning
• Goes on with ROD commissioning

9
Monitoring Framework
10
Monitoring

• Reliable and stable GNAM CORE
• Configurable from on-line database, with
  histogram history
• Logging system (to migrate to ERS)
• Access Manager to ATLAS conditions DB
• Detector libraries
• MDT, RPC, Tile, CSC, (TGC) in use for
  commissioning
• Interface to Event Display available (for some
  detectors)
• Main issue Smart Monitoring
• Automatically catch most low-level problems
• Needed for reaching and maintaining stable
  running conditions
• Common tools needed
• Histogram comparison toolkit
• Error message routing and filtering
• Detector experts
• Identify most common problem sources
• Implement and test alarm strategies

11
HLT Selection algorithms

• Lots of ongoing activities on algorithm
  development side
• LVL1 and LVL2 algorithms run in 11.0.6 CSC
  production for all trigger slices
• EF algorithms almost finished (target 12.0.3,
  next week)
• Lots of ongoing work to improve/test/validate
  algorithms
• PESA algorithms being reviewed (Project
  Milestone)
• ID LVL2, ID EF, Calo LVL2 e/g reviews have
  already taken place
• Muon LVL2EF 90 complete, Calo EF e/g in
  October
• Calo jet/tau/etmiss simple extension of e/g
• Building up comprehensive information about
  performance
• Ongoing work for testing in athenaMT/athenaPT
• Measurements of system (timing) and physics
  performance
• LVL1/HLT AODs fully available in Rel 12 for
  trigger-aware analyses (Project Milestone)
• Most data in place for ESD AOD in release
  11.0.5, Apr 06
• More hypothesis algorithms, slice configs and
  TriggerDecision in 12.0.1

12
HLT Steering

• Il cuore delle sequenze di selezione e decisione
• Sviluppi basati sullattuale implementazione
  (12.0.2)
• Introduzione di selezioni topologiche elementari
• Primi test attualmente in sviluppo nel settore
  della B-fisica (B?mm, J/Psi?mm, BS?DSp con DS?fp)
• Introduzione di meccanismi per la
  sincronizzazione delle sequenze di algoritmi
• Selezioni combinate di diversi oggetti (ad es.
  singolo e, singolo jet, singolo t)
• Sincronizzazione delle selezioni topologiche
• Iterazione del design di implementazione
• Ridisegno dellimplementazione dello Steering
• Nuove interfacce, più generali e flessibili, per
  gli algoritmi di feature extraction e hypothesis
  testing
• Ristrutturazione della navigazione, disaccoppiata
  dallo Steering, per essere facilmente utilizzata
  nelle analisi Offline
• Interfacciamento con lintroduzione del concetto
  di Trigger Chain
• Fase di disegno quasi ultimata inizio
  dellattività di sviluppo e test

13
HighLighT e/gamma

• Electrons
• Eff. w.r.t truth for ET25GeV e
• Time per track in top events
  (1.3
  tracks/RoI with pTmin0.5GeV)
• Brem recovery to be optimized in progress
• Optimizations ongoing also for eff/rej
• Photons
• Use only calorimeter information at L2 and EF
• Use CSC data, single photons ET 20, 60 GeV,
  filtered QCD di-jets with ET(hard) gt 17 GeV

Algorithm KalmanFitter GSF
Time/track 9.5 ms 1.24 s
98
97
Eff ?20i Rate 2?20i Eff ?60 Rate ?60
L1 98.1 330 Hz 97.9 180 Hz
L2 Calo 92.7 6 Hz 94.4 49 Hz
EF Calo 85.3 4 Hz 84.3 42 Hz
14
Efficiencies for SM signals
Use trigger optimisations giving 80 overall eff
(outside crack) for 2e15i, e25i, e60.

• Efficiencies given after kinematical cuts
• 2e in ?lt2.5 with ETgt15 GeV for Z?ee
• 1e in ?lt2.5 with ETgt25 GeV for W?e?

trigger
2e15i 67.2 ---
e25i 92.9 79.6
e60 20.4 6.9
all 94.8 80.3
e25i

• efficiency stays flat after turn-on.
• Turn on curves similar for e60.

Method for single-electron trigger efficiency
determination from data in progress ( ,
)
15
Trigger efficiency from Z ? µµ-

• Double Object (DO) method
• Control sample Good Z from 2 offline ??- with
  loose selection cuts 1? trigger signature
  satisfied
• Trigger efficiency determined from counting in
  how many cases the second ? satisfies the
  trigger requirements
• EF TrigMoore (MuonSpectrometer standalone)
• EF TrigMoore (MuonSpectrometer InnerDetector)

Reco MC
eEF () 94.5 0.2
eL1EF () 78.3 0.3 80.2 0.2
For ?(µ20)gt70 statistical uncertainty after
30min at L1033cm-2s-1 1-2
Ongoing study using complementary method of
orthogonal signature from ID (DOS method),
almost finished
Reco MC
eEF () 96.4 0.2
eL1EF () 79.9 0.3 83.2 0.2
16
HighLighT muons

• Barrel
• Final update of the LUT available optimized with
  respect to the momentum resolution and to the
  efficiency
• Hypothesis exists and the PT cut is optimized for
  the 6 GeV threshold
• Endcap
• Bugs found in the TGC reconstruction code and in
  the analysis programs
• Situation improved harsh zones still remains,
  but results are understood now
• Only TGC data has been used try to measure muon
  PTand to parameterize the track path
• An offline study of the momentum resolution
  obtained fom MDT data is also under way
  preliminary results are in agreement with those
  we got using only TGC data.

17
HighLighT Jet/Tau/ETmiss

• Very active trigger slices, just one year ago
  only small part of the selections were available
  (new groups joining TDAQ)
• Jets
• Algorithms for LVL1/LVL2/EF reco and hypothesis
  in place
• Lots of work carried out to allow trigger-aware
  analysis
• Ongoing work to understand physics performance
• Tau
• Algorithms for LVL1/LVL2/EF reco and hypothesis
  in place
• Use tauRec at EF, support for other tau reco
  packages foreseen
• Ongoing work to find best sequence at LVL2 first
  calo reconstruction or first tracking
  reconstruction
• L2 selection ran successfully in athenaMT in
  11.0.6
• ETmiss
• Study performance of ETmiss at LVL1 and EF

18
Prescale Jets

• First place where to study and implement
  pre-scales
• Reconstructed Jet ET distribution for a
  combination of J1 to J8 samples, scaled according
  to their respective cross-section

19
Algorithms in Pre-Serie

• Motivation
• Get the trigger PESA algorithms working online
• Make sure the latest offline developments get
  running online
• Historical background
• Tools allowing the emulation of online running
  were developed
• athenaMT (LVL2) and athenaPT (EF)
• A group of people was formed in order to provide
  full slice job options suitable for online
  running
• Jets (Ignacio Aracena)
• Taus (Pilar Casado, Richard Soluk)
• Egamma (Xin Wu, Imma Riu)
• Muons (Alessandro Di Mattia, Diana
  Scannicchio)
• One single integrated job options built
  successfully able to run in the Pre-Serie L2 and
  EF farms from real data format (bytestream)
• Try this integrated job options in an online
  partition with different input files
• Single electrons, Top events, Muon events
• Produce a unified input BS file to be tested
  online
• Produce a BS output file from SFO and try
  extending the menu to other signatures (e.g.
  di-objects) using the same algorithms slices to
  see the trigger behaviour

20
Running
L2 partition running ONLINE single electrons
with the integrated job options in the nightlies
of 11.0.6
21
More running
22
Databases
Remote sites
ATCN Network
Calibration Updates including special Muons
calibration path
Tier-1 replica
CERN Computer centre
ATLAS Point1
Bypass (ATCN / CERN-IT)
Tier-1 replica
Offline master CondDB
Online OracleDB
Tier-0 recon replica
Tier-0 farm
Isolation / cut
Data in the DB, plus in (large) files
referenced from the DB
Diagnostics replica
RDB systems used Oracle, MySQL, sqlite layer
to decouple clients
23
Trigger Configuration

• TriggerTool
• GUI for DB population
• easy and consistent menu changes for experts
  (LVL1 and HLT)
• TriggerDB
• stores all information to configure the trigger
  LVL1 menu, HLT menu, HLT algorithm parameters
  (JO), HLT release information
• stores all versions used, with a key
• Configuration and Conditions DB
• DB available at Point 1 and replicated to
  external sites
• Retrieval of information for running
• get information by key via two paths
• extraction of data in XML/JO files
• direct read-out
• for both online offline running

Data Flow
shift crew
offline user
expert
TriggerTool
DB population scripts
TriggerDB
compilers
Configuration System
R/O interface
online running
offline running
24
Example database for TrigConf
keys stored in Conditions DB gt Trigger
conditions precisely known
HLT jobOptions
HLT menu
HLT release
LVL1
25
HLT Large Scale Tests

• Test complete HLT on many nodes
• Scalability of DAQ, DB, HLT sw
• Emphasis on DB, HLT (no LVL1)
• DAQ verify scalability stability
• DataBase
• Explore caching tools (DBProxy)
• Detector oriented tests
• 1000 to 1200 nodes
• LST 2005 600 (512 used for LVL2)
• Will be provided by IT (agreed) with lxbatch etc.
  machines
• Non-optimized network
• Big enough for full size Level2, ¼ size Event
  Filter
• Test Level-2 and Event Filter together (EB and
  SFO)
• As many trigger algorithms as possible
• Large number of events, recycled
• Collect monitoring info
• Physics plots (produced by algorithms)
• Detailed timing histograms, etc.
• Need to have SHIFTS!

26
LVL2 ID in cosmics

• Tracking algorithms
• IdScan, modified to handle single tracks with
  large impact parameters
• SiTrack, version with special LUT tuned for muons
  not coming from IP

• Variables studied
• For T2Id, Offline and MC
• Ntracks, d0, phi0, z0, eta0
• Efficiency calculations
• Data LVL2 wrt Offline
• Simulation LVL2 and Offline wrt truth

Cosmics at SR1 instrumented region, two SCT
sectors
27
Trigger-aware analysis

• Analyses using trigger information as a
  pre-processor to correctly evaluate
  efficiencies, physics reach, etc.
• The reconstructed objects, used by the trigger
  are saved in the ESD/AOD file
• They can be used for comparison with
  truth/reconstructed information
• It is possible to re-play the trigger decision,
  by running the hypothesis algorithms on these
  objects
• Only the settings of the hypothesis algorithms
  can be changed in the analysis
• The effect of different threshold settings can be
  measured

Production
Analysis
Data taking
28
Trigger Physics Weeks

• Follow up on TDAQ and Computing operation
• e.g. where are the bottlenecks on (instantaneous
  and average) rates?
• Updates on trigger slices and initial feedback on
  Trigger Aware Analyses in Release 12
• Work towards ultimate offline efficiencies and
  rejections, to be compared to corresponding
  figures for trigger selections
• Especially relevant for Combined Performance e,
  t, m, b-tag
• Follow up on minimum-bias event selection and
  analysis
• Follow up on menu for L 1 1031 cm-2s-1 (items
  and rates)
• Small group formed to study the issue and propose
  a strategy
• Ideas shown need to be developed into menus for
  commissioning and for early physics (optimisation
  for two purposes somewhat different)
• Follow up on physics with b-jets
• Efficiency to identify the jets for tagging in
  LVL2 in various physics scenarios - e.g. rates
  and RoI multiplicities
• Hot topics
• End-cap LVL2 muons
• Forward-jet trigger
• ETmiss rates (including beam backgrounds)
• Next week end of October

29
Planning

• Visibilità ragionevole
• fino a Marzo 2007

Schedula in revisione alla luce delle nuove
informazioni su LHC discussioni durante la TDAQ
Week di Settembre cioè adesso, in perfetta
sovrapposizione con la nostra CSN1 (Ho il volo
per Londra domani mattina)
30
ROS plan

• Prima tranche dei ROS già pagata al CERN (275
  KCHF)
• Seconda tranche e resto delle network card
  ordinata e consegna ultimata di recente,
  pagamento previsto entro la fine dellanno (275
  KCHF)
• Questo esaurisce il nostro contributo CORE al
  Read-Out System
• Impegno importante previsto in ME per contributi
  allinstallazione e commissioning dellintero
  sistema
• Sezioni coinvolte Pavia, Roma1,

31
Data Collection plan

• Other DataFlow PCs
• Price enquiry finished for SFI, DFM, L2SV and
  pROS
• Evaluations nearly complete
• Choose which company(ies) to buy from
• SFOs - specification to be finalised in 1 month
• 2007 CORE expected from Italy 50 KCHF
• Try and setup system capable of deliverying
  steady-state nominal SFO output from day 1
• Better position to exploit early running trigger
  and detector studies
• No correlation with total resources available in
  HLT farms
• e.g. send LVL1 output directly to off-line

32
Networks and Online plan

• Switch Procurements (Chassis Pizza Boxes)
• Plan finalised, shopping list produced for 2006
• Sharing between FAs agreed
• File servers and Monitoring PC specs finalised
• 135 KHCF su CORE 2006, inclusi monitoring PC e
  local file server end of contributions

33
Infrastructure plan

• SDX Infrastructure
• Orders started - cable ladders, nuts bolts,
  shelves for switches
• About to decide on power distribution
• Coolers for SDX
• Order placed for 43 racks (all of upper level)
• 2007 CORE expected from Italy 80 KCHF

34
HLT farms plan
TDAQ Resource Committee

• HLT Processors
• IT Market Survey completed
• Specifications has been drafted similar (but not
  identical) to ATLAS needs
• Not possible to use a blanket contract for 2006
  buy our own
• Lively discussion about spending profile for this
  items
• First iteration ended Saturday September 16th _at_
  344 (Big Ben time)
• Propose today our best approach based on
• Need to cover basic needs for commissioning,
  calibrations, first run period
• Avoid buying anything that is not absolutely
  necessary
• 95 KCHF back to CSN1 in 2006
• Other HLT/DAQ Items
• First bills are appearing here - Rack coolers,
  inter-rack fibres, etc

35
Revised Cost Profile (KCHF)
2004 2005 2006 2007 2008 2009 Total
Pre-series 140 0 0 0 0 0 140
Detector R/O 0 275 275 0 0 0 550
LVL2 Proc 0 0 30 95 365 160 650
Event Builder 0 0 50 50 110 70 280
Event Filter 0 0 110 140 670 380 1300
Online 0 45 135 0 0 0 180
Infrastructure 0 0 80 80 20 20 200
INFN Total 140 320 680 365 1165 630 3300
TDR Total 1048 3357 4087 4544 7522 4543 25101
36
Richieste ME (DAQ/HLT)

• Per le responsabilità vedi la prima trasparenza
• Installazione e commissioning ROS e HLT
• 4 mesi uomo a Roma1
• 2 mesi uomo a Pavia
• Commissioning DAQ muoni/pixel
• 4 mesi uomo a Roma1 (incluso ROD Crate DAQ)
• 2 mesi uomo a Pavia (incluso sviluppo Monitoring)
• 9 mesi uomo a Genova (incluso HLT steering)
• Event Building, SFI e SFO
• 8 mesi uomo a LNF (inclusi Large Scale Tests)
• Deployment degli algoritmi di selezione on-line
• 2 mesi uomo a Pavia

37
Conclusioni

• Stato attuale del progetto HLT/DAQ ben allineato
  con le scadenze future di ATLAS e di LHC previste
  nel 2007
• Il progetto è certamente complesso e anche le
  responsabilità e i finanziamenti italiani coprono
  diversi settori
• Il sistema di HLT/DAQ ha legami profondi con
  tutte le altre aree di sviluppo rivelatori,
  software online e offline, networking,
  performance di fisica, etc
• Sono necessari maggiori contributi alla forza
  lavoro per la parte di installazione e test
• Impegno fondamentale di consentire a tutti i
  rivelatori una fase efficiente di commissioning
• Inserimento degli algoritmi nei test di cosmici
• Realizzazione di catene complete di read-out e
  successivamente di Event Building
• Le slice HLT hanno raggiunto uno stadio di
  maturità avazata, sono integrate nellambiente
  on-line e costituiscono gli ingredienti per la
  costruzione e lo studio dei Menu di Trigger
• Il Progetto è in fase di revisione (management)
• Necessità di adatttarsi alla nuova fase
  dellesperimento
• Coerenza con lATLAS Operational Model
• Impegno perché gli italiani mantengano i ruoli di
  visibilità che si sono meritati