Lessons Learned from Hurricane Katrina

1
Lessons Learned from Hurricane Katrina

• Azim Ashraf
• Manager Network Security Incident Response

2
Personal Naiveté

• Personal Preparations
• Some sense of excitement
• Estimation of what may occur
• Weather Channel always on
• A bit of Snow Day mentality

3
Hurricane Katrina
Initial Projected Path
Thursday August 25
Sunday August 28
Saturday August 27
Tuesday August 23
4
Monday, August 29 - Landfall

• Katrinas Immediate Effects
• Makes landfall 610 a.m.
• Lower LA Parishes swamped by storm surge no real
  word out
• Parts of New Orleans flooded, at least one levee
  over-topped, but city seems to have survived
• SE Louisiana devastated by winds/rain
• Mississippi seems hardest hit
• Monday 5pm Meeting at LSUPD Station LSU is OK
• LSU Survived just a little damage on campus
• Data Center Lost power but fail-over to back-up
  worked perfectly
• Everything Looks Good to Go for Tuesday
  clean-up, Wednesday start-up, and
  Thursday-as-usual
• Mood lightened
• Power restored to campus 615pm

5
Tuesday 8/30 Bad gets worse

• First confirmed reports of a levee failure in New
  Orleans occur at 130AM CDT
• By mid-day gt80 of New Orleans is under water
• Evacuees en route
• LSU contacted about expanding routine special
  evacuee facilities into a broader purpose
• Medical Triage (Pete Maravich Assembly Center)
• Special Needs Facility (Field House)
• First IT needs Phones, phones and more phones

6
Called to assist

• IT personnel needed to respond
• It was not going to be anything like a snow day

7
First Impressions
8
First Impressions
9
First Impressions
10
LSU A city within a city

• Large H. Ed. institutions uniquely positioned to
  respond
• Infrastructure, knowledge, manpower, affiliations
• PMAC/Field House Became the largest acute care
  hospital to date in in U.S. history
• Over 40,000 (?) patients processed during
  Hurricanes Katrina and Rita
• Established a Hurricane command center
• Coordinated information for students, and
  evacuees, as well as directing resources to where
  they were needed
• Faculty, staff, and student volunteers
• Housing for responders
• Crowd control
• Food and laundry services
• Long distance charges
• Managed volunteers
• Received and distributed donations

11
LSU A city within a city (contd)

• Tracked patients, volunteers, responders,
  supplies, etc..
• Provided Web page re-direction (and other IT
  services) for UNO
• Leveraged communications hardware and services to
  facilitate data or phone support for
• Command centers
• Responders
• Govt. Agencies
• Affected Universities
• Evacuees
• Etc.
• LSU expended over 1M (not reimbursed)
• Over 100K out of CIOs budget
• LSU Became perhaps the most critical facility in
  support of disaster relief/response in the State
  of Louisiana

12
Lessons Learned at LSU

• Buildings can be rebuilt hardware can be
  replaced. Data is the basis of continuity.
• Knowing what youll need to do and having it
  organized is more important than knowing exactly
  how youll do it
• IT enables everything in the 21st Century
• IT Personnel First Responders
• Disaster Recovery and Business Continuity
  Planning is not a luxury
• Be prepared to be flexible adapt, improvise,
  overcome

13
Lessons Learned at LSU (contd)

• Have a good stock of networking equipment, and
  mobile and desktop computing in the storeroom
• Have strong relationships with key vendors
• And most importantly

14
People are your most key asset

• Know who does what and have them on reserve
• Expect them to be burdened with other priorities
• Be prepared to be amazed

15
Key changes in LSUs Plan

• Formal LSU EOC
• Formal Memoranda
• of Agreements (MOAs)
• State agencies
• Private sector
• diesel fuel from local refinery
• water from local bottler, etc.
• Secondary suppliers backing up primaries
• Chancellor requested written plans from all units
  on campus
• Full-time generator for PMAC
• Logistics now pre-planned

16
Traditional Disaster Recovery- Youre down,
everything else is fine

• Do you have a workable DR plan?
• Do you know where on campus youll go?
• Did you take necessary back-ups and do you have
  them ready to re-produce production files?
• What vendors will you need to tap and for what?
• How will you quickly re-establish network
  connectivity? Phone service? Web presence?
  E-mail? Mission critical information systems?

17
Broader Disaster Recovery- You (and everyone
around) you are down

• Are your off-sites conveniently (and perhaps
  tragically) close?
• Do you have arrangements to get key services
  restored at a distance
• Web, E-mail, Financial/HR, Student Information,
  CMS
• Hot-sites may be too expensive but can you find
  suitable raised floor/HVAC/power to re-build
• Can you support your administration in exile?
• Internet access, computers, cell phones, e-mail,
  IM
• Is your life-boat plan portable over larger
  distances?
• Can you grab your key people? Can you care for
  them?

18
One Possible Tool In The ArsenalData Center
Lifeboat

• Situation What if we had very short notice (4-8
  hours) notice of the need to abandon our data
  center/campus and set-up elsewhere (gt50miles
  away)
• Goal 1 Re-establish some critical subset of
  services
• Goal 2 Support the re-establishment of some
  subset of university administration

19
Lifeboat

• Key things to address
• Off-site storage of critical back-ups
• Ability to grab and go key data and hardware
• List of key hardware needed later from vendors
• Disaster Supplies Crate
• What would we put into an 8x12 truck for rapid
  evac?
• Equipment for a mobile or relocated university
  command post
• Laptops, radios, phones, etc.
• Identify Key IT personnel
• Who does what w/back-up
• Scoop em up
• Where might you go?

• Key things to recover
• Payroll/Financial Data
• Web presence
• Splash/priority information screens
• As much content as possible
• E-mail service for faculty/staff/students
• Portal interface
• Student Information Systems
• HR, Procurement Systems
• CMS
• What else?
• Budgets (25K, 50K, 100K)

20
Survivor Disaster RecoveryYoure the last ones
standing

• Dealing with unimaginable demands
• Start imagining it
• Do you have a stock of equipment to set up a
  large support operation in short-order?
• Networking gear, computers, cables, supplies,
  telephone service
• Value of a flexible and capable staff
• Consider how youll do all this on top of your
  normal jobs, as campus life resumes and student
  enrollment increases
• How ready is your campus administration to take
  on the role of disaster response center?
• Facilities, public safety/police, communications,
  academic affairs
• Is the CEO (Chancellor, or President) prepared?

21
Final Thoughts

• Imagine the questions first so that you can find
  the answers
• Next time, you may not be watching it on CNN
  you may be living it
• Do the right thing
• Now is the time to think, plan, and take action
  later it will be too late

22
Final Thoughts

• Data is the basis of continuity
• Have a flexible plan
• People are your most key asset
• Do the right thing because in the end its really
  all about

23
Service
24
Credits

• The staff of LSU ITS who helped make the relief
  effort a success.
• Brian Voss (CIO) In the Wake of Katrina
• Brian Nichols (CISO) At Katrinas Edge
• Frank OQuinn (DR) Weathering the Storm
• Sheri Thompson, Jim Zietz, and others-
  photographs
• John Borne excerpts from Masters Thesis
• Margo Jolet, LSU Office of Public Affairs - LSU
  in the Eye of The Storm

25
(No Transcript)
26
Lessons Learned from Hurricane Katrina

• Azim Ashraf
• Manager Network Security Incident Response