Mark Krasberg

1
Low Level Commissioning
2005/2006Collaboration Meeting,Baton
RougeMark KrasbergApril 12th, 2006
2
Commissioning a DOMat The Testing DAQ Level
Question Test
Does the DOM pass communications? MOAT / QuadTool
Does the DOM function properly? STF
Does the DOMs Local Coincidence connections work? LCCHAIN.PY
Does the DOM calibrate its HV? DOMCAL
Does the DOM have reasonable rates? Multimon/TestDAQ
Does the DOM take data properly? TestDAQ

3
Commissioning plan pretty much followed

• Could not study string 29 during freeze-in
• Ended up taking more data than original plan (not
  a bad thing)
• Cold temperature problem meant that not all
  elements of the plan could be followed

4
Problem Quads

• Leaking Quads
• 12/25 String 29 all quads
• 01/04 String 39 none (quad 7 SJB problem)
• 01/08 String 38 leak-checked late (oscillations)
• 01/14 String 30 quads 4, 7 (quad 14 SJB prob)
• 01/18 String 40 quads 13 and 14
• 01/22 String 50 leak-checked late (damaged)
• 01/25 String 59 leak-checked late (swamped)
• 01/29 String 49 quads 6, 9 and 10
• Anomolies
• Any DOM which exhibited anomolous behavior was
  unplugged
• Several wire pairs had high currents and their
  quads were unplugged and periodically tested
• Communications Issues
• Encountered a cold temperature (cold reboot)
  problem
• Issue was a moving target as DOMs cooled and
  changed temperatures
• Problem seems to be understood (final fix not
  applied at Pole yet)

5
String 39 two-week freeze-in movie
6
Triboluminescence

• Had a longstanding plan on how to cope with the
  triboluminescence (gt40 kHz individual DOM
  rates!)
• Use LC up.AND.down (Triple Local Coincidence)
• Use Scaler Deadtime (reduces LC trigger rate)
• The plan worked!
• able to take over 1 terrabyte of data during
  season and hand-carry north

7
Data-taking during Triboluminescence
Approximate maximum trigger rate for normal
TestDAQ data-taking
frozen-in LC rate
8
Surface DAQ Hardware

• Jan 15 string 21 hub stopped working, probably
  overheated
• Did not operate all hubs simultaneously after
  this event until the power distribution and
  cooling fan problems in the TCH were solved
• Jan 30 string 59 DSB card didnt work was
  replaced
• String 59 DOMHub suffers from domhub-app JVM
  crashes. This is the only hub which has had more
  than one JVM crash with TestDAQ. RealDAQ has had
  problems with other hubs.

9
Cold Temperature Problems

• DOMs found to have temperature-related problems
  (cold reboot problem)
• Problems occur primarily at top of string
  (colder)
• THIS MADE IT IMPOSSIBLE TO COMMISSION ANY DOMs at
  Pole
• communications problems were a moving target
• the list of DOMs which would not communicate
  properly would change daily, as the DOMs cooled
  down.
• Particularly problemmatic temperature is -28C

10
Study of cold reboot problem in ice

• Used internal heating of mainboard to probe a
  temperature range (like in FATs)
• String 29 Farsta
• 57 reboots between -34.44 C and -27.25 C
• 49 successful cold reboots
• All reboots (8) failed between -31.25 C and
  -30.56 C
• No successful reboots in this range
• Just as during FAT testing, the problem is
  isolated to a specific, narrow temperature range

11
Effect of cold temperature problems on
Commissioning
Question Test
Does the DOM pass communications? MOAT / QuadTool
Does the DOM function properly? STF
Does the DOMs Local Coincidence connections work? LCCHAIN.PY
Does the DOM calibrate its HV? DOMCAL
Does the DOM have reasonable rates? Multimon/TestDAQ
Does the DOM take data properly? TestDAQ

12
Able to deal with the expected unexpected
problems fairly easily(part of the plan!)

• When we hook up DOMs in a new environment we
  often encounter problems
• Each DOMHub had a full TestDAQ development
  workspace (ie all the code needed for data-taking
  could be modified and recompiled at pole).
• Patches could be emailed through Iridium
  satellite
• Patches could then be installed, recompiled, and
  tested within minutes
• Feedback could be given to the north almost
  instantaneously using Iridium satellite we did
  not have to wait for TDRS satellite. If we had,
  then the resulting code iterations would have
  gone on for days!

13
Communications now stable

• TestDAQ softboot problem eliminated March 8.
• DOM communications fix waiting to be deployed at
  pole
• Data taking became stable with TestDAQ as of
  March 8th (600 DOMs at that time) except for
  occasional string59 domhub-app crashes.
• DOMs can be commissioned since the communications
  problems have effectively been eliminated and
  data-taking is stable.
• AMANDA mainboards have been included in the
  readout in almost all TestDAQ runs since February

14
High Current DOMs

• Six high current wire pairs identified at pole
• Four of the high currents mysteriously went away
  (seemingly not related to temperature or
  freeze-in). One took weeks to go away!
• Two remaining high current pairs permanently
  plugged in at station close
• DOMs on both of these pairs have suffered
  problems in the months since they were plugged in
• In the future, high current pairs should not be
  used in normal data-taking for as long as is
  feasible (one year?)

15
High Current DOM Pairs

• High current pairs which are still high current
  pairs
• 39-07 Hippophilia 8b56bf63b908
  UP5P0888 0 3
• 39-08 Shamal 7b492c28a830
  TP5P0527 0 3
• 40-51 Juneberry ce9692f580c9
  UP4P0086 6 1
• 40-52 Alfa_Romeo_Spider 0a9dc8de00c3
  TP5Y0171 6 1
• High current pairs which are no longer high
  current pairs
• 49-01 Ceres 754dad1d3800
  UP4Y0030 0 0
• 49-02 Marzahn 60cad4124a82
  TP5Y0161 0 0
• 50-37 Volkswagen_Beetle 93c6fcb0fe68
  UP5Y0216 4 2
• 50-38 Warthog a095b31f89cf
  TP4P0315 4 2
• 40-57 Tortellini d54044599535
  UP4P0140 7 0

16
Date Problem
Jan 1 29-59, 29-60 short noticed while DOMs in water
Jan 21 38-59 Blackberry HV went bad soon after turn-on while DOM in water
Jan 27 and Jan 29 30-23 Peugeot_505 and 50-36 Ocelot wouldnt power up, post-refreeze (connectors showed leakage and DOMs were not powered up while in water)
Jan 29 50-58 Universitet LC problem (DOM?). This is a high current wire pair
Feb 5 or 6 LC connection between 59-51 Medborgarplatsen and 59-52 T_Centralen broke during refreeze (working earlier)
Mid-February to present 40-51 Juneberry 40-52 Alfa_Romeo_Spider must be cooled down to get into iceboot!
Between March 9 and March 13 39-08 Shamal (high current pair) LC problem
March 14 Flash chip problem for 49-17 Biometeorology (during flash upload?)
before March 14 59-59 Cosmology is found to have a flash problem which affects only realDAQ
between March 9 and March 14 29-18 Yurei suffers 40 gain shift between different DOMCal runs
March 20 Prediction of future problem 40-51 Juneberry or 40-52 Alfa_Romeo_Spider (high current DOM pair) we should unplug these DOMs and use for special tests
between March 24 and April 4 40-51 Juneberry and 40-52 Alfa_Romeo_Spider degrade

17
40-51 Juneberry 40-52 Alfa_Romeo_Spider
poor comms
50-58 Universitet bad LC
39-08 Shamal bad LC
59-59 Cosmology poor flash
38-59 Blackberry no HV
29-18 Yurei PMT?
49-17 Biometeorology bad flash
30-23 Peugeot_505 no power
50-36 Ocelot no power
29-59 Auroraphobia 29-60 Nix no power
18
There are currently 16 strings (not 9)!
hole 21 29 30 38
39 40 49 50 59 strings
1 1 2 1 2 2
2 3 2 total16
we had 17 mini-strings in UW FAT 6!
broken LC connection between two DOMs
Problem DOMs
a very short string!
cannot be used for hard LC
19
STF results for string 38, odd (U) DOMs
20
38-59 Blackberry failure
21
38-59 Blackberry failure zoom
22
Freeze-In 49-55 FusilliDOM was still in water
on Mar 22
23
String 49 drill stopped for one
hour at 2300 meters
24
Summary of Commissioned DOMs

• To date the 300 DOMs for which STF ran
  successfully at Pole can be regarded as
  commissioned (low level)
• when 49-55 Fusilli freezes-in STF will be run
  again, as per initial plan (dont expect any
  problems)
• Current Status 600 out of 604 communicating on
  March 20
• DOMs with serious problems
• Three communicating DOMs not useful for hard LC
  data-taking (this affects 4th DOM)
• 2 DOMs (40-51 Juneberry and 40-52
  Alfa_Romeo_Spider) are high current, poorly
  communicating, have recently degraded and should
  be unplugged for special testing
• DOMs with less serious problems
• 2 DOMs (49-17 Biometeorology and 59-59
  Cosmology) have flash problems
• 29-18 Yurei suffered a 40 gain shift in March
• 21-30 Phenol has sustained high dark rate
  problem, operating at lower gain
• 21-13 Wickueler does not like to be turned off
  (discovered during this season)
• Additional LC problem between 59-51 and 59-52
  (broken connection during refreeze)

25
(No Transcript)
26
49-55 Fusilli
27
(No Transcript)
28
(No Transcript)