Level 3 Trigger Data Acquisition

1
Level 3 Trigger / Data Acquisition
L3/DAQ Hardware
Presented by Dave Cutts
2
Level 3 / DAQ Personnel
Gena Briskin Dave Cutts Sean Mattingly Mike
Clements Kongbin Kang .
Jack Fowler Dave Rossi
Gordon Watts new postdoc new student .
Chris Johnson Andrey Karachintsev Ray Zeller
ZRL
3
(No Transcript)
4
Level 3 / DAQ Responsibilities
Brown/ZRL DAQ hardware DAQ module control
software diagnosticsmonitoring (low level) node
framework (partial) standalone systems
support module test setups module
support low-level DAQ support
Washington L3 Supervisor communication w/ COOR
(online coordination) run control, definitions,
parameter passing,... interaction with online DAQ
monitor, filter monitor,... node framework
(partial) L3 Node software integration build/port
ing issues monitor/diagnostic higher level
utilities
5
The DØ Trigger System
Detector
Trigger
Full
Information
Readout
DAQ
7 MHz
1 kHz
Level 1
Level 3
10 kHz
50-70 Hz
Level 2
Online
1 kHz
RIP
Tape Output
(in Feynman)
6
L3/DAQ Hardware glossary
Digitizing crate
VBD
VME Buffer/Driver
VBD Readout Concentrator
VRC

Segment Bridge Fibre Channel Interface Event Tag
Interface Data Cable Interface
FCI ETI DCI
SB
ETG
Event Tag Generator
MPM
Multiport Memory
Level-3 filter node
7
Level 3 /DAQ data paths
data collection path
VBD
VBD
VBD
o o o
VRC
primary FC Path
(1 of 8)
VBD
VBD
VBD
o o o
L3 data distribution to Level 3 farm
Levels 1,2
Levels 1,2
ETG
trigger data
trigger data
MPM
MPM
o o o
event tag
event tag
path
path

• direct flow from VBD to MPM
• each data block flows
• freely and independently
• blocks for different events
• flow simultaneously and asynchronously
• recirculation loops allow maximum
• use of data path bandwidth
• Segment Bridges use Event Tag data
• to perform complex realtime routing decisions

MPM
MPM
o o o
MPM
MPM
o o o
MPM
MPM
o o o
(stage C)
8
Front End Token Readout Loop
Front End Token Readout Loop
Front End Crate
Front End Crate
Front End Crate
Front End Crate
VRC 1
VRC 8
Front End Crate
Front End Crate
Primary Fiber Channel Loop 8
Front End Crate
Front End Crate
Primary Fiber Channel Loop 1
SB 1
SB 4
Event Tag Loop
Ethernet
Ethernet
ETG
To Collector Router
To Collector Router
Trigger Framework
Segment Data Cables
Segment Data Cables
)
)
9
VBD Module

• Digitized data enters DAQ system via a VBD module
• Typically 4-6 kBytes per VBD/event
• VBD drives data cable _at_ 48 MBytes/sec
• Synchronization is done via a token
• Upgrade status
• Built 12 years ago. For Run 2 new/improved
  VME/IO PACs
• New header/trailer
• Make Tracking and CAL data formats look the same
• VBD modification/reprogramming are being done now
• Some Upgraded Extended and short VBDs are at D0
  now!
• All Extended VBDs are ready to be delivered to D0

Front End Crate
Digitizers
Digitizers
Digitizers
VBD
Token Control
To next VBD or VRC
Data Pathway 48 MBytes/sec
From last VBD or VRC
10
VRC System

• VBD Readout Concentrator (VRC)
• One VRC drives two VBD readout token loops
• Output on Fibre Channel (100 MBytes/sec)
• First VRC Prototype will be at Brown in several
  weeks

FCI from last SB
VBD
100 MB/s
VBD
Loop 1
Data Cable
VRC
VBD
Token CLK
VBD
Token CLK
Loop 2
Data Cable
VBD
100 MB/s
FCI to 1st SB
VBD
11
ETG System

• Event Tag Generator (ETG)
• Links L3/DAQ with the L1/L2 trigger systems
• Uses trigger information for a given event to
  create an event tag which enables complex data
  block routing in real time

From Trigger FrameWork
ET Creator (CAM Lookup)
Monitor CPU
Ethernet

• 16 bit L3 Transfer Number
• Trigger Bits

ET Queue

• Event Tag Format
• Header
• Event Number
• Flags (diagnostics or regular)
• Segment Controller 1 Block
• Event Type
• FEC Readout Bit Masks
• Segment Controller 2 Block
• Event Type
• FEC Readout Bit Masks
• etc.

Return ETG Monitor
L1 Disable Lines


SB
SB
SB
SB
Level 3 Farm
12
SB System

• Segment Bridge (SB)
• Receives Event Tag from ETG accepts/declines the
  event based on characteristics of available L3
  nodes

13
L3 Farm Node
L3 Node Framework
VME Crate
Each 48 MB/s
Control, Monitoring and Error Module
MPM
Node-VME I/O Module
Ethernet
MPM
DMA capable VME-PCI Bridge
Shared Memory Buffers
Collector Router Module
L3 Filter Interface Module

• design of MPM upgrade is done
• prototype is available and has been tested

• Multiprocessor PC (likely quad)
• Run 2 systems will be purchased as late as
  possible
• Interim systems are supplied by Brown

Dedicated 100 Mbits/s Ethernet to Online
Collector/Router
14
Goals for L3/DAQ
1. Provide continuous DAQ capability to
detector groups 2. Develop and commission
DAQ upgrade a) build new modules VRC, SB,
ETG b) upgrade existing modules VBD, MPM
15
L3/DAQ Operational Plan

• Design build new modules at ZRL
• Test, interface integrate modules at Brown
• Install at D0

• Maintain support ongoing
• D0 DAQ installations at Fermilab

16
DAQ V0 System at D0 (now)

• DAQ V0 Goals
• 1) Full DAQ functionality and 2) Continuous
  operation
• at D0 in MCH, floors 1 , 2 and 3

Readout Control
Readout Control
Readout Control
Data Cables
Data Cables
Data Cables
VME
VME/PCI
VME
VME/PCI
VME
VME/PCI
Bridge
Bridge
Bridge
MPM
MPM
MPM
Read control
Read control
Read control
3
2
1
Ethernet
Network Bridge
Trigger FrameWork
ETGSB
Level 3 Farm
17
Standalone L3/DAQ Systems

• Three independent systems are currently in
  operation, away from D0
• Brown Development System
• Test hardware prototypes
• Integrate DAQ hardware into L3/DAQ
• Central development maintenance site
• Silicon Test DAQ
• Si crate dumps directly into a L3 Node
• Key element in silicon detector work
• Good test bed for L3/DAQ no failures!
• CFT Cosmic Test DAQ
• Ready to go!

(Photo to be supplied)
18
DAQ in the Moving Counting House
MCH-1
MCH-2
MCH-3
L1CT
LM FPD
Mu fwd
CFT
Cal
Si
Cal
VRC
Si
VRC
Si
Mu cen
Cal
VRC
VRC
VRC
Cal
L2glob
L2cal
STT
L2 Mu
Frame
STT
L2CT
L2PS
XX location of detector front end crates with
VBDs
19
DAQ in the Fixed Counting House FCH-2
MCH
FCH
L1/L2
VRCs
L3 nodes
Level 1/2 event trigger trigger disables
primary fiber data path (circulation loop)
event tag path (circulation loop)
local segment data path (circulation loop)
20
Timeline for data path upgrade modules
2000
2001


J
F M A M J J A S
O N D J F M
VRC
VBD Readout Concentrators
SB
Segment Bridge
ETG
Event Tag Generator
Cosmic Ray Commissioning
Phase II Fiber tracker, trigger,
preshowers, VLPCs, CTT, forward muon
Phase III Full Cosmic Ray Run
(add TRIG, SMT, CAL)
Phase I Central muon, DAQ, RECO,
trigger, tracker front-end
21
Timeline for D0 DAQ evolution
2000
2001


J
F M A M J J A S
O N D J F M
V0 DAQ (emulated VRC, SB, ETG)
VRC at D0
SB, ETG at D0
Multi crate readout CAL, muon, tracker, trigger
begin installing final L3 nodes
Cosmic Ray Commissioning
Phase II Fiber tracker, trigger,
preshowers, VLPCs, CTT, forward muon
Phase III Full Cosmic Ray Run
(add TRIG, SMT, CAL)
Phase I Central muon, DAQ, RECO,
trigger, tracker front-end
22
Timeline for D0 DAQ capability
2000
2001


J
F M A M J J A S
O N D J F M
Event rate into Level 3
10-50 Hz
100 - 2000 Hz
Level 3 filter nodes
8 - 48 nodes
2 - 6
begin installing final L3 nodes
Cosmic Ray Commissioning
Phase II Fiber tracker, trigger,
preshowers, VLPCs, CTT, forward muon
Phase III Full Cosmic Ray Run
(add TRIG, SMT, CAL)
Phase I Central muon, DAQ, RECO,
trigger, tracker front-end
23
Timeline for D0 DAQ installation/commissioning
2000
2001


J
F M A M J J A S
O N D J F M
VRC
MCH cabling
SB, ETG
FCH cabling
Interim L3 nodes
final L3 nodes
Cosmic Ray Commissioning
Phase II Fiber tracker, trigger,
preshowers, VLPCs, CTT, forward muon
Phase III Full Cosmic Ray Run
(add TRIG, SMT, CAL)
Phase I Central muon, DAQ, RECO,
trigger, tracker front-end
24
Summary

• Level 3 / DAQ is available as needed
• DAQ capability is well matched to the
  commissioning schedule
• Basic design unchanged from Run 1
• New modules optimize use of the data paths
• Sequence of installation complete MCH
• April-June 2000 in sequence with
  crate installations
• then follow with L3 farm-related work, in
  FCH
• Only installation manpower issue MCH tech
  help
• All elements of the Run 2 DAQ will be well
  tested in the cosmic
• ray commissioning run
• Delay installation of the complete set of final
  final L3 nodes
• as long as possible