Mark Krasberg

1
String Commissioning 06/07
IceCube Collaboration Meeting Lake Geneva,
Wisconsin Mark Krasberg, April 25th, 2007
2
Commissioning a DOMat The Testing DAQ Level
Question Test
Does the DOM pass communications? MOAT / QuadTool / DMT
Does the DOM function properly? STF
Does the DOMs Local Coincidence connections work? LCCHAIN.PY
Does the DOM calibrate its HV? DOMCAL
Does the DOM have reasonable rates? Multimon/TestDAQ
Does the DOM take data properly? TestDAQ

3
Cold Temperature Comm Problem

• DOMHubs were crashing (100 times!)
• Initially slowed commissioning down
• Problem was related to DOM temperature
• old strings were much more likely to crash
• John Jacobsen quickly modified the dor-driver to
  prevent hub-crashes
• DOMs would also not reliably softboot, which
  meant communications test programs would fail

4
Cable Checks (cable guys)

• A total of four quads were determined to have
  leaks (using Wet Connector Tester), went away
  after freeze-in, as expected.
• Two quads had DOMs which failed the Quad
  Connectivity Tester. After freeze-in, one of
  these DOMs was OK ! (string 66 quad 10).
• DOM which does not power up is 74-09 Immunology
• This DOM was one of several which had a partially
  broken penetrator connector

5
Power-up Current Checks

• Quickly read out the DOR card configboot current
• Move DOMs which are drawing too much current to
  sps-wczar DOMHub for study
• DOR card current readings are recorded in order
  to carefully track this issue

6
No uncorrected ultra-high current tail in this
plot. Those DOR cards were pulled during testing
using SCUBE at SPTS prior to being shipped to
South Pole.
7
S-CUBE 64 mainboards Used to test all DOMHubs
prior to shipment to Pole DOR card current
reporting could be calibrated
8
SLIDE FROM MARCH 2006 !!High Current DOMs

• Six high current wire pairs identified at pole
• All anomolous DOMs kept unplugged
• Four of the high currents mysteriously went away
  (seemingly not related to temperature or
  freeze-in).
• Two existing high current pairs plugged in at
  station close (we should reconsider this).

9
SLIDE FROM MARCH 2006 !! High Current DOM Pairs

• High current pairs which are still high current
  pairs
• 39-07 Hippophilia 8b56bf63b908
  UP5P0888 0 3
• 39-08 Shamal 7b492c28a830
  TP5P0527 0 3
• 40-51 Juneberry ce9692f580c9
  UP4P0086 6 1
• 40-52 Alfa_Romeo_Spider 0a9dc8de00c3
  TP5Y0171 6 1
• High current pairs which are no longer high
  current pairs
• 49-01 Ceres 754dad1d3800
  UP4Y0030 0 0
• 49-02 Marzahn 60cad4124a82
  TP5Y0161 0 0
• 50-37 Volkswagen_Beetle 93c6fcb0fe68
  UP5Y0216 4 2
• 50-38 Warthog a095b31f89cf
  TP4P0315 4 2
• 40-57 Tortellini d54044599535
  UP4P0140 7 0

10
High Current DOMs from 2006/2007

• There were 7 high current DOM pairs
• High Current DOM pairs are being treated with
  care
• Only being turned on very briefly every few days
  to measure their configboot current
• Pathology of high current DOMs looks quite a bit
  different this year!

11
string 58 67 66 74 73 65
72 78 48 57 47 46 56
High Current / LC breaks on the 13 new strings
High Current DOMs
Does not power up
LC break
12
DOMs should freeze-in here
13
Were there any high current DOMs found during
surface testing?
Arizona was a high current DOM during surface
testing. DOM name recognized on shelf in the TOS
during deployment! DOM pulled from hole. The
proudest moment in the history of the DOM names!
Arizona just arrived at PSL. Current draw
increases as voltage increases!
14
Running DOMs in water

• We ran three strings and a handful of other DOMs
  in water this season
• Two DOMs behaved very badly!
• In 2005 / 2006, there was
• 38-59 Blackberry

15
38-59 Blackberry failure
DOM stops triggering
Nasty temperature rise
HV never worked again, Otherwise DOM OK
Blackberry turned on in water here
16
38-59 Blackberry failure zoom
DOM stops triggering
Nasty temperature rise
HV never worked again, Otherwise DOM OK
17
Operating DOMs in water

• In 2006 / 2007 there were
• 78-49 The_Exorcist
• and
• 48-55 Captain_Spaulding

18
78-49 The Exorcist
Rate spikes!
Nasty temperature rise!
Normal temperature rise
Unstable pressure readings!
19
The_Exorcist and correlated spikes with nearby
DOMs
The_Exorcist
20
The_Exorcist may have been sparking
The_Exorcist
21
Water leaking into DOM?

• Allan Hallgren speculated water had leaked into
  the DOM, resulting in sparks
• Just before The_Exorcist was powered down, it
  had become high current (96 mAmps instead of 80
  mAmps) and it was not communicating properly!
• The_Exorcist was moved to the sps-wczar debug
  hub, pending freeze-in and retest.

22
48-55 Captain_Spaulding
small bump
Other than rate increases, no other anomolous
readings detected for Captain_Spaulding
23
Waveform/timing plot
There are lots of TestDAQ runs which were taken
while the sparking was going on
Light appears to originate at Captain_Spaulding
24
Sparking DOMs

• 78-49 The_Exorcist, after freeze-in, behaved
  normally.
• 48-55 Captain_Spaulding, after freeze-in,
  behaved normally.
• Powering down these DOMs may have saved them from
  destruction

25
Two TestDAQ string processors

• Made it possible to perform different
  commissioning tasks or take flasher runs for the
  verification group in parallel.
• Both string processors were used heavily (often
  24 hours per day)
• Increased satellite bandwidth meant that a large
  fraction of the data could be transferred by
  SPADE.
• Around four thousand TestDAQ runs taken, half for
  verification group.

26
Could take calibrated TestDAQ data very quickly!

• String 48 example
• String tied off at 920pm NZT
• DOMs powered up by 1030pm NZT
• Calibrations and DB insertions made quickly
• String 48 was ready to take calibrated data by
  midnight!

27
Other Issues

• Two DSBs blew fuses during the season
• Kael asked LBNL to send more DSBs
• 6 spares arrived on the last flight (and 50 spare
  fuses)
• All spare DSBs used as of last night
• A few cabling mixups which were easily corrected
• We hooked up SPOTS again (the South Pole
  Outdoor Test System) so we could do tests on DOMs
  which were not deployed
• Orphaned DOMs are being commissioned and moved to
  the strings as appropriate (currently, sps-wczar
  has twelve DOMs on it, of which six to ten should
  eventually be able to be integrated into the
  detector)
• Many DOMs were found to not have their flash
  completely loaded DOMAPP was missing! Problem
  quickly dealt with in almost all cases.

28
QUADTOOL results
74-10 Selaphobia (partner of Immunology) Sel
aphobia means fear of light flashes
29
(No Transcript)
30
(No Transcript)
31
(No Transcript)
32
(No Transcript)
33
Data taken with 22 strings on Feb 11, Animation
made at pole with help of Phil Roth, Georges
Kohnen 49-57 Popcorn is flashing 378 DOMs
fire All 22 strings are hit!
34
FLASHER BOARD 49-17 Popcorn
Standard Candle
35
2006/2007 InIce Commissioning
36
1,403 Commissioned DOMs

• There are 1,424 deployed DOMs
• 1,387 DOMs were commissioned by seasons end!
  (now 1,403)
• IceCube-9 596 of 604 DOMs commissioned
• Same as end of last year
• Zero failures as a result of DOMs being turned
  off for several weeks!
• Of the 596, two have broken LC and one has broken
  HV
• Other 8 two are on the debug hub sps-wczar
  (Juneberry, Alfa_Romeo_Spider)
• 13 new strings and new IceTop 807 of 820 DOMs
  commissioned
• Only one DOM, 74-09 Immunology, does not power
  up
• Twelve DOMs are on the debug hub, sps-wczar.
• Four pairs are high current, under study
• One pair has 56-58 Tengu which was still in
  water yesterday (40 kHz noise rate!)
• Last year, Fusilli didnt freeze-in until the
  beginning of May
• One pair communicates very badly
• Of the newly commissioned DOMs
• 58-45 Kokolik_River suffers from SHDR and
  DOMCal cannot calibrate its HV
• 1E7 gain tuned by eye
• One DOM requires a special DOR firmware setting
• 72-33 Mahischa, 72-35 Ant_Nebula and 72-43
  Spirograph have broken LC (like Shamal and
  Universitet from last year)

37
Some of these DOMs are high current pairs
(partners could be fine)
38
(No Transcript)
39
76 DOMs
589 DOMs
1390 DOMs
39-22 Liljeholmen stops communicating properly
30-60 Rowan stops communicating properly
40
76 DOMs
589 DOMs
1390 DOMs
39-22 Liljeholmen stops communicating properly
30-60 Rowan stops communicating properly
41
(No Transcript)