Random Reco. Output

1
Random Reco. Output

• Mike Kordosky
• UCL

2
The Issue

• Running a single reco. job multiple times may
  yield different results
• differences are significant and high level
• not caused by random number generation
• Seen in past. Hit me most recently.
• R1.12, DB and script as in MDC
• Present in data ?
• Present in MDC....

3
MDC example
run 14301001 (first run), snarl 0
new

snarl event mcevent true_enu
reco_enu evtsigful reco_vtxz

0
0 20 34.761169 15.851137
97429 0.0539014 0 1
6 1.0525250 0.3122515 3719
3.6178998 0 2 13
7.1519975 6.8838410 72743 2.6675000
0 3 15 24.716342
26.601753 181936 6.2314901 0
4 15 24.716342 28.509492
7447 7.1818900 0 5
3 3.3432865 3.8706350 52970
6.1720900
o
rig

snarl event mcevent true_enu
reco_enu evtsigful reco_vtxz

0
0 20 34.761169 15.851137
97429 0.0539014 0 1
6 1.0525250 0.3122515 3719
3.6178998 0 2 13
7.1519975 6.8838405 72743 2.6675000
0 3 15 24.716342
43.492786 174123 6.3502903 0
4 15 24.716342 4.8288211
69800 6.2908902 0 5
3 3.3432865 3.8706350 52970
6.1720900

strip and slice variables identical
4
MDC example
run 14301001, snarl 0
new shw.slc 0, 1, 2, 3, 3, 4
shw.ndigit 23, 20, 60, 258, 13, 61
shw.nstrip 10, 10, 28, 112, 10, 27
shw.ph.raw 11633.000000, 3719.000000,
26662.000000, 171913.000000, 1917.000000,
37184.000000 shw.ph.siglin 11633.000000,
3719.000000, 26662.000000, 171913.000000,
1917.000000, 37184.000000 shw.ph.sigcor
11633.000000, 3719.000000, 26662.000000,
171913.000000, 1917.000000, 37184.000000
shw.ph.pe 141.165802, 44.427662,
271.825928, 1650.924194, 20.133766, 407.313995
shw.ph.sigmap 9973.845703, 3717.484619,
25937.574219, 180983.359375, 2200.174316,
32834.605469 shw.ph.mip 18.300634,
6.821073, 47.591877, 332.079559, 4.037017,
60.246979 shw.ph.gev 0.860843, 0.384069,
2.341103, 17.907658, 0.064218, 2.993784 orig
shw.slc 0, 1, 2, 3, 4 shw.ndigit
23, 20, 60, 258, 61 shw.nstrip 10, 10,
28, 112, 27 shw.ph.raw 11633.000000,
3719.000000, 26662.000000, 171913.000000,
37184.000000 shw.ph.siglin 11633.000000,
3719.000000, 26662.000000, 171913.000000,
37184.000000 shw.ph.sigcor 11633.000000,
3719.000000, 26662.000000, 171913.000000,
37184.000000 shw.ph.pe 141.165802,
44.427662, 271.825928, 1650.924194, 407.313995
shw.ph.sigmap 9973.845703, 3717.484619,
25937.572266, 180983.359375, 32834.605469
shw.ph.mip 18.300634, 6.821073, 47.591877,
332.079559, 60.246979 shw.ph.gev
0.860843, 0.384069, 2.341103, 18.493668, 2.993784
Small extra shower in new output 400 MeV shower
energy diff
5
MDC example
run 14301001, snarl 0
new trk.ph.raw 92100.000000,
62676.000000, 99292.000000, 5977.000000,
30383.000000 trk.ph.siglin 92100.000000,
62676.000000, 99292.000000, 5977.000000,
30383.000000 trk.ph.sigcor 92100.000000,
62676.000000, 99292.000000, 5977.000000,
30383.000000 trk.ph.pe 1027.830811,
684.602722, 992.473022, 50.121658, 333.724030
trk.ph.sigmap 73963.789062, 64402.093750,
102170.890625, 5166.960449, 21579.304688
trk.ph.mip 135.713364, 118.168976,
187.469513, 9.480661, 39.595055 trk.ph.gev
7.335858, 6.387512, 10.133488, 0.512468,
2.140273 trk.plane.n 130, 75, 29, 15, 20
trk.plane.nu 64, 40, 16, 7, 10 trk.plane.nv
66, 35, 13, 8, 10 trk.plane.beg 1, 45,
105, 107, 104 trk.plane.begu 1, 45, 105, 107,
105 trk.plane.begv 6, 46, 106, 108, 104
trk.plane.end 281, 191, 226, 166, 136
trk.plane.endu 281, 191, 191, 131, 131
trk.plane.endv 276, 118, 226, 166, 136 orig
trk.ph.raw 92100.000000, 62676.000000,
69800.000000, 4876.000000, 30383.000000
trk.ph.siglin 92100.000000, 62676.000000,
69800.000000, 4876.000000, 30383.000000
trk.ph.sigcor 92100.000000, 62676.000000,
69800.000000, 4876.000000, 30383.000000
trk.ph.pe 1027.830811, 684.602722,
686.017212, 41.968590, 333.724030 trk.ph.sigmap
73963.789062, 64402.093750, 43733.933594,
5498.340332, 21579.304688 trk.ph.mip
135.713364, 118.168976, 80.245750, 10.088698,
39.595055 trk.ph.gev 7.335858, 6.387512,
4.337608, 0.545335, 2.140273 trk.plane.n
130, 75, 31, 12, 20 trk.plane.nu 64, 40,
16, 7, 10 trk.plane.nv 66, 35, 15, 5, 10
trk.plane.beg 1, 45, 105, 107, 104
trk.plane.begu 1, 45, 105, 107, 105
trk.plane.begv 6, 46, 106, 108, 104
trk.plane.end 281, 191, 226, 136, 136
trk.plane.endu 281, 191, 191, 131, 131
trk.plane.endv 276, 118, 226, 136, 136
6
MDC example
run 14301001, snarl 0
original
event 4
event 3
new
7
MDC example
run 14301001 , snarl 5
new

snarl event mcevent true_enu
reco_enu evtsigful reco_vtxz

5
0 30 22.912065 16.856296
18459 0.0539014 5 1
13 14.041832 1.4660090 84962
2.3704998 5 2 25
4.7501931 3.9656720 58458 2.0734999

orig

snarl
event mcevent true_enu reco_enu
evtsigful reco_vtxz

5 0 30
22.912065 16.856296 18459 0.0539014
5 1 13 14.041832
5.9028296 83544 2.3704998 5
2 25 4.7501931 3.9656720
58458 2.0734999


The stp array in the original file apparently has
8 doubled hits. (same plane, strip, raw
ADC) Usually slc and stp arrays are identical
8
MDC example
run 14301001 , snarl 11
new

snarl event mcevent true_enu
reco_enu evtsigful reco_vtxz

11
0 0 3.8538007 0.7551127
8600 5.9938902 11 1
18 2.3171551 0.6547919 6674
3.6178998 11 2 14
16.132503 6.5016961 8846 5.1029000
11 3 10 7.9232831
6.7617726 63007 0.8260940 11
4 13 59.878768 3.8374157
39500 3.0238900 11 5
16 4.4987797 2.0107317 23901
5.6968903
o
rig

snarl event mcevent true_enu
reco_enu evtsigful reco_vtxz

11
0 0 3.8538007 0.7551127
8600 5.9938902 11 1
18 2.3171551 0.6547919 6674
3.6178998 11 2 14
16.132503 6.5016961 8846 5.1029000
11 3 10 7.9232831
6.7043390 62849 0.8260940 11
4 13 59.878768 3.8374157
39500 3.0238900 11 5
16 4.4987797 2.0107317 23901
5.6968903

pretty small change, but should be 0!
9
Frequency?

• Deduce from multiple runs over same file.
• 14 of the first 140 events (20 spills) in
  14301001 have some difference in reco. energy.
• sense is that it's typically the same physical
  events having problems
• Other events in exact agreement.

10
Cause?

• Cause not precisely known.
• Good guess
• In R1.12 Navigation sort depended on object
  address to break degenerate cases.
• I've run (few times) w/ newer Navigation and had
  repeatable results.
• But only masks problem.
• Code which defines a sort must be expected to be
  insensitive to degeneracies of the sort!
• Ntuple formation or ROOT I/O bugs not ruled out
• Also, Niki sees similar effect w/ fix devel
  release

11
What to do?

• More effort needed to fully understand issue
• ND only? MC only? Particular event topologies?
• please priority for core/reco experts
• Pursue sorting issue as best lead
• Nick method to intentionally permute sorted
  objects in degenerate case
• Czars is your package sensitive?
• Related JIRA offline bug tracker
• exists, but not used
• encourage use, bring to prominence on webpage

12
MDC example
run 14301001 , snarl 5
original
new
Look identical, but track fit results differ.