ATLAS Event Builder Push or Pull

1
ATLAS Event Builder- Push or Pull -

• ATLAS-J DAQ Group
• Yasushi Nagasaka, Yoshiji Yasu, Yoji Hasegawa,
  and Makoto Shimojima

2
Contents

• Overview
• TDR
• ATLAS HLT/DAQ/DCS Overview
• ATLAS Event Builder
• Japanese Contribution
• Gigabit Ethernet QoS
• Measurement Setup
• Push vs. pull scenarios w/o QoS
• 19 ROSs x 1 SFI configuration
• Push vs. pull scenarios w/ QoS
• 19 ROSs x 1 SFI configuration
• Conclusion

3
TDR Technical Design Report

• High-Level Trigger,Data Acquisitionand Controls
• HLT/DAQ/DCS Group
• CERN/LHCC/2003-022
• ATLAS TDR 016
• 30 June 2003

4
System Overview (1)
ROS Read-Out Sub-System SFI Sub-Farm
Input DFM Data Flow Manager
5
System Overview (2)

• ROS (Read-Out Sub-System)
• 150 PCs
• Event Builder Networks
• 250 Ports (Gigabit Ethernet)
• DFM (Data Flow Manager)
• 35 PCs (8GHz Dual CPU)
• SFI (Sub-Farm Input)
• 90 PCs (8GHz Dual CPU)

6
ATLAS Event Builder
Detectors
ROS
ID
Event Fragments
SFI
Computing Storages
DFM
7
Two scenarios of message flows

• PUSH Scenario
• The DFM assigns an SFI and informs all ROSs, via
  a multicast mechanism, to send their associated
  event fragments to that SFI. The SFI acts as an
  open receiver and builds the complete event out
  of the individual fragments received.
• PULL Scenario
• The DFM assigns an event to an SFI which in turn
  requests from each ROS its event fragment via a
  unicast messages. The SFI receives from each ROS
  individually an event fragment and builds the
  complete event.

8
Japanese Contribution

• Gigabit Ethernet
• QoS (Quality of Service) Control
• One of Traffic Shaping methods
• Sent packets Control at transfer-side computers
• Bandwidth Allocation, Error Rate, Transfer Delay,
  etc.

FIFO
Estimator
Packet
u32 Filter
Classifier
Scheduler
QoS
9
Measurement Setup (1)

• PCs
• Dual Xeon 2.2/2.4GHz PCs _at_CERN
• Linux kernel 2.4.20 SMP
• Kernel parameter HZ4096
• Linux scheduling is done in 4kHz instead of
  usual 100Hz while EB input rate is typically
  3kHz.
• Intel e1000 GbE NIC
• Switches
• Foundry FastIron-800 switch with 64 ports
• TDAQ software
• online-00-19-01 and DF-00-04-03
• Important DataFlow parameters
• TrafficShapingParameter, DecisionGroupSize, and
  ClearGroupSize

10
Measurement Setup (2)
PC Dual Xeon 2.2/2.4GHz
Gigabit Ethernet Switch Foundry FastIron-800
switch with 64 ports
11
Push vs. Pull scenarios w/o QoS on 19 ROSs x 1
SFI

• For QoS study, the worse case should be
  investigated. This configuration causes data
  concentration at SFI in push scenario .

12
EoE Rate / Throughput vs. Input Rate
EoE rate vs. input trigger rate
Throughput vs. input trigger rate
Fragment size 1kB DecisionGroupSize 1 for
push TrafficShapingParameter 30 for pull,
ClearGroupSize 50
For small fragments, the push scenario performed
better than the pull scenario.
13
EoE Rate / Throughput vs. RobDataSize
EoE rate vs. RobDataSize
Throughput vs. RobDataSize
Fragment size variable DecisionGroupSize 1 for
push TrafficShapingParameter 30 for pull,
ClearGroupSize 50
When a fragment size increases, the performance
in push scenario goes down over the size of 2kB
and that in the pull scenario does not. The pull
scenario performed better than push scenario over
the size of 2kB.
14
Reask and timeout vs. RobDataSize
SFI Reasks/Completed ratio vs. RobDataSize
Timeout /Assigned or completed ratio vs.
RobDataSize
SFI Reasks In push
DFM Timeout In push
SFI Timeout In push
For push scenario, lots of SFI reasks and SFI/DFM
timeout. For pull scenario, small SFI reasks and
no SFI/DFM timeout.
15
Push vs. Pull scenarios w/ QoS on 19 ROSs x 1
SFI

• Does QoS improve EoE rate and throughput in
  push scenario?
• Is QoS effective for Event Builder?

16
EoE rate/throughput vs. RobDataSize
EoE rate vs. RobDataSize
Throughput vs. RobDataSize
QoS limits the bandwidth from ROS to SFI to 24Mbps
QoS limits the bandwidth from ROS to SFI to
24Mbps
Push is improved
24Mbpsx19/857MB/s
Push is improved
Fragment size variable, DecisionGroupSize 1
for push TrafficShapingParameter 30 for pull,
ClearGroupSize 50
QoS improved EoE rate and throughput in large
fragment size in the push scenario. QoS also
worked in the pull scenario.
17
Reask and timeout vs. RobDataSize
SFI Reasks/Completed ratio vs. RobDataSize
Timeout /Assigned or completed ratio vs.
RobDataSize
improved
improved
SFI Reasks
SFI/DFM Timeout
QoS decreased SFI reask and DFM/SFI timeout in
the push scenario.
18
Throughput in pull / push w/ and w/o QoS
QoS in the push scenario did not improve
throughputs over a fragment size of 12kB.
19
Conclusion

• Japanese Contribution
• Gigabit Ethernet and QoS (Quality of Service)
• QoS Measurements Summary
• QoS improved EoE rate and throughput of EB in
  push scenario
• However, the performance in push scenario did not
  reach to that in pull scenario.
• Push or Pull?
• Pull scenario is a good way to avoid the data
  concentration at SFI.