Data Acquisition Backbone Core DABC

1
Data Acquisition Backbone Core Jörn Adamczewski,
Hans G.Essel, Nikolaus Kurz, Sergey LinevGSI,
Experiment Electronics Data Processing group
Motivation DABC software design (Jörn) IB
network tests (Sergei) DABC with
Mbs (Hans) DABC Life Demo
Work supported by EU RP6 project JRA1 FutureDAQ
RII3-CT-2004-506078
2
CBM data acquisition plan
W.F.J.Müller, 2004
3
CBM use case Frontend components test
FE
Front end board sampling ADCs (44),
clock distribution
FE
FE Frontend board DC Data combiner board ABB
Active Buffer board GE Gigabit Ethernet IB
InfiniBand MBS MultiBranchSystem
2.5 GBit/s bi-directional (optical) link data,
clock
8
DCB
Data collector boards, clock distribution to FE
DC
2.5 GBit/s data links
4
Gigabit Ethernet
ABB
ABB
DABC PCIe support
PCIe
PCIe
PC
PC
8-20 PCs dual quad
Active Buffer Board PCI express card
MBS
bidirectional event building

• A.Kugel, G.Marcus, W.Gao,
• Mannheim University Informatics V

IB switch
DABC
4
Use case DAQ m x n topology
Detector
Detector
Detector
Detector
Triggerbus
Trigger
Readout
ADC
ADC
ADC
ADC
Trigger
Readout
ADC
ADC
ADC
ADC
Trigger
Readout
ADC
ADC
ADC
ADC
Trigger
Readout
ADC
ADC
ADC
ADC
MBS Crates
Network
Multi Branch System GSI Standard DAQ (100
installations)
5
Motivation for DABC

• Use cases
• Detector tests
• FE equipment tests
• Data transport tests
• Time distribution tests
• Switched event building
• MBS event builder

• Requirements
• build events over fast networks
• triggered or self-triggered front-ends
• time stamped data streams
• data flow control (to front-ends)
• connect (nearly) any front-ends
• interface for application plug-in code
• connect MBS readout or collector nodes
• interface for several controls frameworks

gt DABC for CBM DAQ testing and as future
general purpose DAQ system
6
DABC design functional overview
DABC
PC
datainput
sorting tagging filter analysis
frontendDataCombinerr
frontendReadout
scheduler
GE
frontendother
IB
PC
PC
analysis archive
scheduler
sorting tagging filter analysis
datainput
archive
GE Gigabit Ethernet IB InfiniBand

• DABC data flow

7
Data flow engine Software objects and threads
(1)
A module processes data of one or several data
streams. Data streams propagate through ports,
which are connected by transports and devices
DABC Manager
DABC Module
Bufferqueue
port
port
process
Bufferqueue
Transport
Transport
Central data managerMemory pools
Device
Device
PCIboard
File
8
Data flow engine Software objects and threads
A module processes data of one or several data
streams. Data streams propagate through ports,
which are connected by transports and devices
DABC Manager
DABC Module
DABC Module
port
port
process
process
port
port
Central data managerMemory pools
Bufferqueue
Bufferqueue
locally (by reference)
Transport
Transport
Network
Device
Device
9
Object organisation and Plug-in mechanism
DABC Manager
Controls interface
Object registry
Memory Pools
Plugins
Commands
Parameters
Modules
Devices
SocketDevice
10
Controls interface Generic finite state machine

• State machine is defined by controls interface of
  DABC core
• ApplicationPlugins implement state transition
  actions (user code)
• Nodes in B-net may set up state machine
  hierarchy (e.g. 1 master node, n worker nodes)

11
Controls interface Parameters and Commands

• Parameters
• Registered by plugin or device class
• DIM field service with naming convention and DABC
  structure
• Also SOAP/ XDAQ infospace representation, web
  interface, xml input
• Recognized by generic GUI, automatic graphical
  presentation
• Types Simple, Status, Rate, Histogram, Info,
  Command description,...
• Commands
• DIM (and SOAP) implementation
• State machine commands (generic, with
  overwritten standard callbacks in plugins)
• Framework commands (generic, e.g. SetValue)
• User defined module commands (registered by
  plugin or device)
• execution in user callback
• recognized by generic GUI via command description
• command streamer and description (XML)
  independent from controls implementation!

12
Controls with XDAQ runtime environment
XDAQ Executive (process, addressspace)
Web browser
Web server (SOAP)
SOAP client Java GUI
DIM server
DIM client Java GUI LabView GUI EPICS GUI
XDAQ Application
Infospace
State machine
DABC Core
Modules command queue
GRIDCC
Infospace remotely accessible parameters
DABC core is not depending on XDAQ environment!
13
Summary DABC Status
DABC is a new general purpose DAQ system

• Todo
• Controls, GUI c't
• Data formats
• Combiner / time sort
• Programming Interface implementation c't
• Configuration DB
• Packaging and deployment
• Documentation

• Achieved
• Core infrastructure
• Data flow engine
• XDAQ runtime environment
• Generic Java GUI
• PCI(e) support
• MBS event building
• Programming Interface definitions

Next part Network performance tests (S.Linev)
14
InfiniBand

• High-performance interconnect technology
• switched fabric architecture
• up to 20 GBit/s bidirectional serial link
• Remote Direct Memory Access (RDMA)
• quality of service
• zero-copy data transfer
• low latency (few microseconds)
• Main problem for a long time lack of common API
  for different HW vendors
• now there is OpenFabrics (OFED) package solves
  this problem
• see www.openfabrics.org for more information

15
OpenFabrics software stack
Clustered DB Access
Sockets BasedAccess
Various MPIs
Access to File Systems
Block Storage Access
IP Based App Access
Application Level
Open SM
Diag Tools
User Level MAD API
UDAPL
User APIs
InfiniBand OpenFabrics User Level Verbs /
API iWARP R-NIC
User Space
SDP Lib
Kernel Space
Upper Layer Protocol
SDP
IPoIB
SRP
iSER
RDS
NFS-RDMA RPC
Cluster File Sys
Connection Manager Abstraction (CMA)
Mid-Layer
MAD
SA Client
SMA
ConnectionManager
Connection Manager
Kernel bypass
Kernel bypass
InfiniBand OpenFabrics Kernel
Level Verbs / API iWARP R-NIC
Hardware Specific Driver
Hardware Specific Driver
Provider
Common
Key
Apps AccessMethodsfor usingOF Stack
InfiniBand
InfiniBand HCA
iWARP R-NIC
Hardware
iWARP
slide from www.openfabrics.org
16
MPI evaluation

• MPI Message Passing Interface, was designed for
  high performance on both massively parallel
  machines and on workstation clusters
• MVAPICH MPI MPI2 over InfiniBand project.
  Supports
• non-blocking send/receive operation
• true hardware-based multicast, but only with
  blocking API
• Tests of data throughput and multicast
  performance were done. Good results for big
  (larger than 32K) packets, but difficulty to
  combine normal and multicast traffic.
• Can be as option in DAQ system, while it is
  supported on majority of modern massively
  parallel machines, where different interconnect
  technologies are used.

17
VERBS versus DAPL

• Both have very similar functionality and API
• memory, queues, completion events
• message and RDMA transport.
• But, VERBS provides extra functionality
• reliable/unreliable data transfer
• multicast support.
• We decide to switch to use OFED VERBS while
• it supports full InfiniBand functionality
• it is new official development line for Mellanox
  products.
• The only significant verbs problem lack of good
  documentation.

18
InfiniBand basic tests

• Aim Prove of principal as event building network
  candidate for CBM
• Tests last year
• GSI cluster - 4 nodes, SDR
• Forschungszentrum Karlsruhe (March 2007) 23
  nodes, DDR
• UNI Mainz (August 2007) - 110 nodes, DDR

Point-to-point tests
Multicast tests
thanks to Frank Schmitz, Ivan Kondov and
Project CampusGrid in FZK thanks to Klaus
Merle and Markus Tacke at the Zentrum für
Datenverarbeitung in Uni Mainz
19
Scaling of asynchronous traffic
20
Chaotic (async.) versus scheduled (sync.)
21
DABC and InfiniBand

• DABC fully supports InfiniBand as data transport
  between nodes
• connection establishing
• memory management for zero-copy transfer
• back-pressure
• errors handling
• thread sharing

DABC performance on GSI 4-nodes cluster
22
Socket transport

• Connection management
• many connections to the same node can be started
  at the same time
• All sockets working in non-blocking mode
• several sockets can be served in the same thread
  without lost of the performance (thread sharing)
• Socket close/error detection
• failed node can be reconnected on the fly
• Solution for the cases where InfiniBand is not
  available

DABC performance on GSI 4-nodes cluster
23
Conclusion

• InfiniBand is a good candidate for CBM event
  building network
• Up to 700 MB/s bidirectional data rate is
  achieved on 110 nodes cluster
• Mixture of point-to-point and multicast traffic
  is possible
• Further investigation of scheduling mechanisms
  for InfiniBand is required

24
DEMO with MBS

• MBS upgrade for DABC
• DABC plugins
• Control standard stand alone MBS with DABC GUI
• Control MBS frontends with DABC event builder

25
MBS upgrade for DABC event builder

• General upgrades
• Large buffers (up to now 32K limit) in MBS.
• Large buffers in LMD files.
• Remote control of MBS via DIM by DABC Java GUI
• DABC specific mode
• MBS transport in DABC mode blocks, if no DABC is
  connected.
• MBS transport sends variable sized buffers.
• Using large buffers and one buffer per stream no
  event spanning.
• New LMD file format
• No buffer structure.
• File header, data elements, index table (random
  access).
• No size limit ( gt 2 GB).
• Supported by event API.
• To do
• Event server in DABC
• File output (library ready, RFIO 64 ?)

• MBS application in DABC

26
MBS upgrade controls
Multi node MBS, controlled by DIM client (red)
message queue, to 2000
Main loop
Command thread
Message thread
Message clientthreads
Dispatcher
Task
Prompter
Master Logger
TCP 6004
Commandthread
command queue, to pid
Master node
Slave node
message queue, to 2000
Logger
Main loop
TCP 6005
Dispatcher
Command thread
TCP 6004
Commandthread
command queue, to pid
Task
27
Demo example Network event building for MBS
Linux PCs
lxi013
LynxOS
lxi015
TCP
input
MBS frontend
eventbuilder
x86-7
sender
MBS frontend
x86g-4
DIM, Java
GE
lxg0517
lxi016
lxi014
Event generator
sender
eventbuilder
controller
TCP
input
No commen trigger for demo
GE Gigabit Ethernet

• DABC structure

28
Demo example DABC side, MBS plugins
MBS
Second node
Device
Transport
DABC Module
DABC Module
port
port
port
Sender
port
Formatter
port
Local or over network
DABC Module
DABC Module
port
Transport
port
port
port
Receiver
EventBuilder
port
Device
Second node
File
29
Demo example DABC side, MBS plugins
DABC Device
DABC Module
MbsFrontend MbsFrontend () Create commands
Register parameters CreateTransport (port)
Create MbsRead
MbsFormatter MbsFormatter () Create commands
Register parameters Create ports (1 x input, 1x
output) MainLoop () Wait input port 1 Get
empty buffer Loop Add subevent ref. to buffer
Size gt max? Put buffer to port
Return EndLoop MbsFormatter ()
DABC Transport
MbsRead MbsRead (port) Connect MBS transport
Register port GetBufferSize () Return buffer
size Read (buffer) Read header Read data
Return MbsRead () Disconnect MBS
Thread
Thread
30
Demo example DABC side, BNet plugins
DABC Module
DABC Module
DABC Device
BNetReceiver BNetReceiver () Create commands
Register parameters Create ports (n x input, 1
x output) MainLoop () Wait for all input
ports Add input buffers to output list Put
list to output port BNetReceiver ()
BNetSender BNetSender () Create commands
Register parameters Create ports (1 x input, n
x output) MainLoop () Wait input port 1 Put
buffer to next output port BNetSender ()
BNetNetworkDeviceBNetNetworkDevice
()CreateTransport (port) ()
DABC Transport
BNetNetworkTransportNetNetworkTransport
(port)Write (buffer)
Thread
Provided by DABC
DABC Device
BNetNetworkDeviceBNetNetworkDevice
()CreateTransport (port)
DABC Transport
BNetNetworkTransportBNetNetworkTransport
(port)Read (buffer)
Provided by DABC
Provided by DABC
Thread
Thread
Thread
Provided by DABC
31
Demo example DABC side, MBS plugins
DABC Module
DABC Device
MbsFile MbsFile () Create commands Register
parameters CreateTransport (port) Create
MbsStore
MbsEventBuilder MbsEventBuilder () Create
commands Register parameters Create ports (1
x input, 1x output) MainLoop () Wait input
port 1 Get empty buffer Build event from
input list Put event to buffer Put buffer
to port (file store) MbsEventBuilder ()
DABC Transport
MbsStore MbsStore (port) Register port Write
(buffer) Filesize gt max?
close/open Write fragments to
file MbsStore () Close last file
Thread
Thread
32
Emergency

• Just in case the DEMO does not work!

33
(No Transcript)
34
(No Transcript)
35
(No Transcript)
36
(No Transcript)
37
(No Transcript)
38
(No Transcript)
39
(No Transcript)
40
(No Transcript)
41
(No Transcript)
42
(No Transcript)
43
(No Transcript)
44
(No Transcript)
45
(No Transcript)
46
(No Transcript)
47
(No Transcript)
48
(No Transcript)
49
(No Transcript)
50
(No Transcript)
51
(No Transcript)