Concatenation status

1
Concatenation status
Rob SnihurSimulation mtg., Jan. 27, 2005.

• Old way (Summer2004)
• Uses 2 tape loggers fcdfdata004 fcdfdata0045
• Uses 2 disk buffers fcdfdata004 fcdfdata005
• Backlog
• Jqcd1i (800 GB) 50 done
• Sexo8t (500 GB) 0 done
• Cause distracted human (working on new concat
  scheme)
• Additional causes
• Large samples harder to concatenate
• More jobs crashing
• Disks full ? lose 1 tape logger
• Testing new scheme ate up disk space
• Solution
• Use new disks

2
New proposal

• One DCAF segment simulation makes many (small)
  output files, totaling X GB (where X gt 1)
• At the end, concatenate all small files within
  the DCAF segment
• MCfull
• For (iistartiltiedi)
• Call MCProd
• cp outfile outdir
• Concatenate outdir

3
Features

• Simple
• self-contained
• No need for durable cache
• instead use disk available on worker nodes how
  much ?
• No need for separate submission of concatenation
  jobs
• Less book-keeping
• Average user can do it
• Fewer segments to submit
• Drawbacks
• Need to know event size for each sample in
  advance
• Could run a 100-event test job

4
Testing

• Ran 80 wtop3i segments in 2 jobs (5030)
• Total segments 301 (904 run sections)
• Expected final output file sizes of 1 GB
• Got 0.2 1.4 GB
• Used large max runsection size (2000 events),so
  this probably explains large variation
• Kept final output files on disk, allowing
  validation before inserting to DFC.
• First job (50 segments)
• 45 OK
• 5 failed to copy output tarball (though output
  data may be OK)
• Got 49 output data files
• Missing 6 expected output files
• I suspect my knowledge of file naming
• Name for a file which starts with run section 2
  for lowest run number?
• Need to catch failure modes. Must carefully
  analyze log files, file sizes, names, etc.

5
CAF email output

• Submitted Thu Jan 20 060535 2005
• Ended Fri Jan 21 160640 2005
• Job duration 340105
• Wait times Max Min
  Abs.Avg. Start Avg.
• 121623 00048
  01443 01442
• Segment times Total Mean
  RMS Max
• Real 8971930 175647
  34415 270819
• CPU 8905951 174911
  71657 622212
• IO Data summary
• WaitDH 00106 00001
  00000 00003
• Read 51015.8M 1020.3M
  277.8M 1460.0M