FinalCommissioning

1
Detector Reliability, Operations and Maintenance
June 5th, 2008
2
Reliability
3
Results from 2008 commissioning
4
(No Transcript)
5
(No Transcript)
6
Spontaneous DOM failures

• 39-61 Hydrogen stopped triggering (end of Dec)
• discovered during weekly TestDAQ runs
• Pressure inside sphere dropped by factor of 2
• PMT likely cracked
• No cause identified Len Shulman found no
  bulldozer tracks nearby
• 66-33 New_York and 66-34 Dou_Mu spontaneously
  became high current during the season
  (DOMHubMonitor)
• 54-47 Garbanzo_bean (new DOM) failed (March)
  (DOMHubMonitor)
• 39-21 Aspudden had DOMCal problems (March)

7
76 DOMs
589 DOMs
1390 DOMs
2515 DOMs

66-33 New_York and 66-34 Dou_Mu go high
current
39-22 Liljeholmen stops communicating properly
30-60 Rowan stops communicating
8
76 DOMs
589 DOMs
1390 DOMs
2515 DOMs
66-33 New_York and 66-34 Dou_Mu go high
current

39-61 Hydrogen PMT breaks
54-47 Garbanzo_bean stops communicating
39-22 Liljeholmen stops communicating properly
39-21 Aspudden slows down
30-60 Rowan stops communicating
9
76 DOMs
589 DOMs
1390 DOMs
2515 DOMs
66-33 New_York and 66-34 Dou_Mu go high
current
39-61 Hydrogen PMT breaks
54-47 Garbanzo_bean stops communicating
39-22 Liljeholmen stops communicating properly
39-21 Aspudden slows down
30-60 Rowan stops communicating
10
Operations
11
DOMs not part of IC40
12
DOMHubMonitor

• SUMMARY
• --------------------------------------------------
  --------------------------------------
• HUB AM 01 02 03 04 05 06 21 29 30 38 39 40 44
  45 46 47 48 49 50
• COMM 2 32 32 32 8 32 24 60 58 58 60 60 58 54
  60 58 58 60 60 59
• --------------------------------------------------
  --------------------------------------
• HUB 52 53 54 55 56 57 58 59 60 61 62 63 64
  65 66 67 68 69 70 71
• COMM 60 58 59 60 60 60 60 60 60 60 60 60 60 60
  56 60 60 55 60 59
• --------------------------------------------------
  ---------------------------------------
• HUB 72 73 74 75 76 77 78
• COMM 58 60 59 60 60 58 60
• --------------------------------------------------
  ---------------------------------------
• HUBS 47 COMM 2527 (max
  number is 2562)
• NO PROBLEMS FOUND

13
A Complex Configuration!

• See http//wiki.icecube.wisc.edu/index.php/Problem
  _DOMs

• TOTAL NUMBER OF IC40 DOMs with non-standard
  config 88
• Around 3 of the deployed DOMs

14
Maintenance
15
Major Sources of downtime

• 2 DOR cards failed
• 1 DOMHub hard drive failed
• 1 DSB failed
• Key hard drives filled up when something stopped
  working, which led to other problems

16
IC40 Uptime for May 2008
sps-2ndbuild disk fills
(slow) failure of sps-ichub68 DOR card
Jun 11, 2008
Detector Ops - IceCube Hartill VII
16
17
Remediation

• DOMHubMonitor issues alarms when there are
  DOM/DOR card related failures
• ComputerMonitor (deployed end of May) issues
  alarms when disk drives are filling up,
  successfully preempts problems from occurring
  BEFORE they occur
• May 29 sps-2ndbuild/usr/local/pdaq/ was filling
  up
• June 3 sps-2ndbuild/mnt/data/ was filling up
• would have led to pDAQ downtime
• An about-to-be-deployed centralized paging
  system will alert the winterovers when a problem
  has been detected
• Will prevent long downtimes during sleep periods
• Emergency South Pole contact list in the event
  that the problem is not understood
• Around 25 people on the list

18
Winterovers

• Responsibilities more carefully defined
• Install Asterisk-based PBX system for
  notifications
• Pagers have dead zones. In the future, paging
  system will also use phone/radio.
• Upgrading RT system to make it more useful and
  document its configuration and usage
• Going to make use of the paging system if a
  winterover does not return from the Dark Sector
  on schedule (for safety)