Tropical Storm Allison Recovery

1
Tropical Storm AllisonRecovery

• Charles Chambers
• Information Technology
• 2001

2
Time Line
Sa 9th Su 10th M 11th T 12th W 13th Th 14th F 15th Sa 16th Su 17th M 18th T 19th W 20th Th 21st F 22nd

• Saturday
• flooding begins (1 A.M.) IT team on campus
• Additional IT staff arrive pumping and power
  are critical
• Institutional priorities are focused Non-IT
• Sunday
• Pumping of network node facilities begin
• Power restored to UHS data center
• Internet connection reestablished
• 713-743-1000 call center established
• Systems and servers returned to service

3
Time Line
Sa 9th Su 10th M 11th T 12th W 13th Th 14th F 15th Sa 16th Su 17th M 18th T 19th W 20th Th 21st F 22nd

• Monday, June 11th
• WAN services restored except UHV
• IP telephones activated in critical areas
• Relocated some servers to Data Center
• Tuesday, June 12th
• Internet connectivity unstable due to ISP
• Servers in Data Center grow by 20
• UHV network connection restored
• Desktop assessment team inventories campus
• Network teams inspect buildings for damage

4
Time Line
Sa 9th Su 10th M 11th T 12th W 13th Th 14th F 15th Sa 16th Su 17th M 18th T 19th W 20th Th 21st F 22nd

• Wednesday, June 13th
• UH-CVN services restored, classes held.
• Payroll processed
• 50 of phone system returned to service
• Thursday, June 14th
• Data backbone restored
• Data connectivity to opened buildings
• Order Equipment for Storm Solutions for Hofheinz
  Pavillion

5
Time Line
Sa 9th Su 10th M 11th T 12th W 13th Th 14th F 15th Sa 16th Su 17th M 18th T 19th W 20th Th 21st F 22nd

• Friday, June 15th
• Majority of phone system restored
• Saturday, June 16th
• Equipment for Hofheinz Pavillion arrives,
  installed

6
Time Line
Sa 9th Su 10th M 11th T 12th W 13th Th 14th F 15th Sa 16th Su 17th M 18th T 19th W 20th Th 21st F 22nd

• Sunday, June 17th
• Hofheinz Pavillion available
• 270 IP telephones
• 314 data ports
• IT support booth created
• Monday, June 18th
• University Re-Opens
• First occupants moves into Hofheinz Pavillion

7
Time Line
Sa 9th Su 10th M 11th T 12th W 13th Th 14th F 15th Sa 16th Su 17th M 18th T 19th W 20th Th 21st F 22nd

• Tuesday
• Voice Mail system restored
• Touch Tone Registration restored
• Remainder of phone system restored
• Wednesday
• Majority of SWB trunking problems corrected

8
Time Line
Sa 9th Su 10th M 11th T 12th W 13th Th 14th F 15th Sa 16th Su 17th M 18th T 19th W 20th Th 21st F 22nd

• Thursday
• Friday
• Reconfiguration of Hofheinz Pavillion network

9
Technology Priorities
Infrastructure

• Phone System
• Internet Connectivity
• Inter-campus Network
• Campus Data Network
• Interactive Video Network
• Data Center Operation

10
Technology Priorities
Services

• E-mail
• Web Site
• WebCT
• IVR for Registration/Records
• Payroll
• Financial Application

11
Technology Priorities
Desktop Computers

• Inventory
• Repair
• Replacement Computers
• Data restores

12
Technology Priorities
Temporary Space - Holfheinz

• Used basketball pavilion
• Boxes became offices
• Basketball floor
• became labs and cubicles

13
Holfheinz Pavillion
From Basketball to Academics

• 27 boxes
• Cisco IP telephony 270 phones
• Switched data network
• 270 ports in offices
• 144 ports on basketball court
• Over 100,000 feet of copper and fiber
• IT support booth
• Voice and data network support
• Desktop computer support

14
Recovery Resources

• Over xxx person hours (UH Staff)
• Loss and Business Resumption
• Costs estimated in millions
• Over 250 IT related staff involved

15
Lessons Learned

• Move IT facilities above ground
• Move generators above ground
• Flood doors on underground entrances
• Established plans helped . . .
• The right people at the right places, making the
  right decisions was the key.
• Mobilize support efforts early
• Food
• Sleeping Quarters

16
Heroic Efforts

• Pumps/Pumping
• Saving the Animals
• Getting Primary DID in service
• Internet Connectivity
• Holfheinz Pavillion

17
How the Speedy Recovery

• Temporary fixes
• Great minds and enormous goals
• Dedicated staff
• Very long hours
• Multiple plans
• Quick vendor response
• Elimination of bureaucracy
• Communication of status and goals
• Central coordination point

18
Starting Recovery

• Pumping and Power
• Starting pumping PGH on Saturday
• Moved pumps to animal lab (SR2)
• Sunday power restored to Computing Center
• Internet Connection
• Reconfigured to computing center
• Web and e-mail up first

19
Recovery Partners

• Herculean efforts by all!
• Pumping water
• Removing damaged equipment
• Designing and installing new systems
• Power restoration
• UH Support
• College volunteers
• Facilities
• Vendor response was great!
• Staff on site (over 100)
• Rapid deployment of equipment

20
Initial Impact

• Primary Technology Areas Affected
• UHS Data Center
• Network Node in PGH
• Network Node in UCU
• Headend Facility in Moody

21
UHS Data Center

• Lost power
• No flooding or other damage

22
Network Node (PGH)

• Services lost
• Voice, data, video for UH
• Internet and interactive video for all UHS
• Damage
• Water 18 inches above raised floor
• Power systems total loss

23
Network Node (PGH)

• Telephone and Voice Mail Systems

24
Network Node (UCU)

• Services lost
• Voice network for 1/3 of campus
• Trunk lines from SWB
• High capacity circuits from SWB
• Damage
• Telephone Switch corroded
• Wire center destroyed
• 3600 stations

25
Network Node (UCU)

• Telephone system

26
Network Node (UCU)

• SWB
• Interface

27
Network Node (UCU)

• Wire Center Frame

28
Headend Facility (Moody)

• Services Lost
• CATV Network

29
Hofheinz Pavillion
Home of Phi Slamma Jama
30
Hofheinz Pavillion
Wiring was pretty open
31
An wide shot view of the inner-city of Houston,
Texas
32
Flood waters invade the City with no Mercy
33
Freeway Entrance ramp into downtown from the North
34
Freeway Exit ramp
35
A view of Allen-Parkway
36
Raging waters in the Tunnel System Downtown
37
Another view of the Tunnel System Downtown
38
And...Another view of the Tunnel System Downtown
39
Canoeing down HWY 59 near the Compaq Center
40
A view of 1 of 3 of the Suspension Bridges that
crosses over HWY 59 near the Compaq Center
41
A view of the Houstons Channel 11 - KHOU -
Television Studio
42
A view of Interstate Highway I-10
43
Jet Sking in the flood waters on Interstate
Highway I-10
44
A view of a 3-way Intersection located at Allen
Parkway and Sanford Street
45
Heres Who did it!

• CTS Crew at Storm Allison Luncheon