On the same page with Streaming

1
On the same page with Streaming

• Adam Lyon
• (Please dont kill me!)
• Analysis Tools - 5/21/2002

• Outline
• The paper trail
• The streaming model as documented
• A possible implementation
• The other 10 (Exceptions)

2
The Paper Trail

• New Phenomena Comments on the D0 Run II data
  accessD0 Note 3313 (9/97, Bantly, Boehnlein,
  Hobbs for NP)
• Run I inclusive streaming 50 replication
• Disjoint classified streams based on Level 3
  Filter bit, grouped by L3 physics object. 2n-1
  exclusive streams for n classifications
• Stream min/zero-bias events separately
  (inclusive) for easy access to entire sample
• Study of Possible Run II Streaming Scenarios D0
  Note 3326 (10/97, Schellman, Bertram)
• Studied exclusive, partially exclusive, and
  inclusive streaming scenarios with 100,000 Run I
  events. Exclusive streaming with L3
  classification bits. Weighs pros and cons of each
  scenario

3
The Paper Trail

• Run II Streaming - How to Stream Events D0 Note
  3662 (6/99, Boehnlein, Fan, Schellman, de Souza)
• Recipe for streaming in Run II
• LogicalStream to Physical Stream Mapping
  schellma/upgrade_computing(2/2000 10/2000,
  Schellman, Yu)
• Tree algorithm scheme for combining the
  primitive streams into physical streams (2n-1 ?
  m m ltlt 2n-1) based on event probability to load
  balance the physical streams
• Plain English Explanation of Streaming
  schellma/upgrade_computing(6/2001, Schellman)
• Short summary of previous work

4
The Paper Trail

• Streaming and Level 3 http//hep.brown.edu/users/
  Greg/streaming/documentation.htm(8/2001, Hays)
• Combining Primitive Streams to Physical Streams
  -- two example tree algorithms
• Conceptual Design for Offline Data Analysis and
  D0 Streaming (8/2001, White) Gregs web page
• Offline streaming issues (SAM, Luminosity)
• Stream Definition Table (Strawman Proposal)
  (??/?? Landsberg) Gregs web page
• Streaming proposal wed like to implement as a
  test case

5
Streaming (as documented) at a glance
L3 Filter Script
From L2
?
MU_HI (0001)
L2 trigger (MUJET)
?
Primitives BitMap
Physical Stream
JT_HI (0100)
L2 trigger (ELEJET_HI)
L3 Filter Script
(0101)
MU_STRM
?
ELE_LO (0010)
Primitives bit map is OR of bits from filters of
passing L3 filter scripts (2n-1) possibilities
Event is mapped (via some algorithm) to physical
stream using BitMap m ltlt 2n-1 physical streams
?
JT_BIG (0100)
L3 Filter assigned a primitive bit for the
physics object it tests
6
How does L3 form the Primitive BitMap?

• Each L3 filter can set its bit when it runs and
  passes
• Gives people heebee-jeebees (what if filter
  writer forgets to code this step)
• Must OR BitMaps from passing filter scripts at
  the end
• Use a lookup table (L3 filter script ? Primitives
  BitMap)
• Since all filters of a filter script must pass,
  if a filter script passes, you know the resulting
  bit map a priori
• Download this information to L3 (in trigger
  database)Can this be made even better? See next
  slide
• L3 takes no time setting bits and it always works
• Can override settings easily
• Still must OR BitMaps from passing filter scripts
  at the end to form final word

7
How to go from BitMap to Physical Stream

• Run mapping algorithm in Level 3 (slide)
• Jon Hays already has this
• Must download algorithm configuration via XML
  file (Where is this configuration stored?
  Trigger database? Streaming database?)
• Use a Priority lookup table (slide) (L3 filter
  script ? Physical Stream skip Primitive BitMap
  step)
• Since all filters must pass and you know the
  mapping algorithm, you a priori know what stream
  an event passing one L3 filter script goes to
• The Priority handles events passing multi-L3
  filter scripts

8
BitMaps ? Physical Streamsfrom Hays document

• Simple one-to-one
• 2n-1 physical streams
• Always n decisions
• No ambiguity

• Priority
• Flexible
• Can sometimes decide in lt n decisions
• Can produce few streams, though must be careful
  that streams are not big
• You have to think! What if Muon Electron?

9
Priority Lookup Table example
Event
Event
MISSING_ET_STRM
MISSING_ET
MISSING_ET
MU_HIGH_STRM
MU_ELE
Event
MISSING_ET_STRM
MISSING_ET
MINBIAS
MINBIAS_STRM
JET_MULTI
10
Priority Lookup Table

• Object streaming Trigger Bit streaming
• Primal BitMaps and the Mapping Algorithm give the
  first draft of the lookup table
• Easy to fine-tune and override (necessary for
  special triggers, testing)
• One table does it all - fairly unambiguous
• XML Generator gets trigger list information
• XML Generator makes 1st draft streaming
• User fixes XML for special cases
• Runs database records stream table from XML

11
Exceptions (For now, the rule!)

• Special L1 triggers (with normal physics L3
  filters)
• Cosmic gap trigger
• Deans non-zero suppressed Cal trigger
• Hard-wire their streams in the XML
• Triggers with L3 Prescale 1 (no other L3 filters)
• 50 of current triggers
• Hard wire stream
• L3 Mark Pass (no L3 Filter satisfied)
• Go to monitor stream (inclusive)
• L2 Mark Pass not implemented yet

12
Conclusions

• Exclusive Streaming ? Some (most) triggers span
  more than one stream
• Minimize tapes
• A few groups win big, most groups still look at
  almost all the data
• Hard to do inclusive studies (e.g. commissioning)
• Hard to determine luminosity, trigger rates
• Hard to re-reco a certain class of events
• Lost data in low rate streams may be devastating
• TOOLS WILL BE CRITICAL TO KEEP USERS HAPPY!

13
Tools

• User wants to run over data from certain triggers
• Determine the streams to obtain
• Stream definitions can change over time
• Needs Run database, trigger database
• Must be able to work through mapping table
• Determine files for those streams
• SAM does this
• Determine good runs (run over only these)
• Determine bad luminosity blocks (skip these
  events)
• User wants luminosity for data
• Monitoring tools (trigger rates, verfication)

14
Tools