SELECTING A DISASTER RECOVERY SITE

1
SELECTING A DISASTER RECOVERY SITE

• D. J. BRAGG
• SANDIA NATIONAL LABORATORIES

Sandia is a multiprogram laboratory operated by
Sandia Corporation, a Lockheed Martin
Company,for the United States Department of
Energys National Nuclear Security
Administration under contract DE-AC04-94AL85000.
2
BACKGROUND INFORMATION

• Preliminary study in 2005 to characterize SNLs
  ability to recover from interruption of data
  center operation
• Limited off-site storage due to security
  requirements
• Designated recovery site 5 miles from main data
  center
• Data center loss would require gt 6 months to
  recover 80 of applications and data

3
BACKGROUND INFORMATION

• 2006 study to find and designate a disaster
  recovery site
• Evaluation team included SNL staff and Disaster
  Recovery/Business Continuity consulting firm
• Study included government owned, commercial, and
  build-to-suit facilities
• Selection criteria established with weighted
  grading
• Criteria included environmental, infrastructure,
  geologic, social, financial, regulatory and
  political considerations.
• The obvious answer is not always the correct
  answer

4
WHAT ARE THE OPTIONS?

• Outsourcing to a commercial or non-SNL facility
• Pro
• Equipment on hand and fully maintained
• Ability to leverage quick ship agreements with
  vendors
• 24x7 operations, physical security
• Con
• Limited access and availability for testing
• Data not under your direct control
• Require long-term, expensive contract
• Insourcing to user controlled facility
• Pro
• Lower cost
• Greater chance of meeting RTO
• No scheduling issues for testing/availability
• Control and security over data
• Con
• Facility identification, acquisition and
  maintenance
• Underutilized equipment, equipment maintenance

5
INDUSTRY STANDARDS

• Tier 4 Multiple active power and cooling
  distribution paths, redundant components, fault
  tolerant, 99.995 availability
• Tier 3 Multiple power and cooling distribution
  paths, but only one path active, redundant
  components, concurrently maintainable, 99.982
  availability
• Tier 2 Single or multi path for power, single
  cooling distribution path, redundant components,
  99.741 availability
• Tier 1 Single path for power and cooling
  distribution, no redundant components, 99.671
  availability
• Industry Standard Tier Classifications The
  Uptime Institute

4
3
2
1
6
SELECTION PROCESS
7
(No Transcript)
8
(No Transcript)
9
(No Transcript)
10
MOTHER NATURES RECORD
11
OTHER CONSIDERATIONS

• What is the likelihood of a disaster occurring at
  both the primary site and the recovery site?
• What potential actions or conditions could
  prevent access to the recovery site?
• Military installation closure
• Civil disturbances, protesters
• All flights grounded (9/11)
• What about other environmental aspects
• Proximity to interstate (fuel/chemical truck
  accident)
• Proximity to fuel storage facilities, industrial
  operations
• Seasonal blizzards, poor weather making travel
  hazardous

12
WHAT COULD POSSIBLY HAPPEN HERE?
13
THORNY ISSUES

• Physical and Data Security
• Security for facility, access control
• VTR Requirements for classified
• What about Sigma data?
• Clearance and badging of facility staff
• Surveillance, monitoring and auditability
• Adequate funding for startup and continued
  operation
• New facility 10-15M
• Update existing 1-5M
• Foreseeable technology changes
• Critical for planning of cooling, electrical
• 30kW racks in very new future
• 10G fibre

14
SCREENING PROCESS

• Survey questionnaire with 165 items addressing
  all criteria sent to candidate facilities
• Returned questionnaires were scored and ranked
• Site visits were conducted to qualitatively
  assess the site based on facility condition,
  existing operations, potential for growth, etc.

15
SITES CONSIDERED

• NWC Facilities
• LANL (10), KCP, LBL, LLNL, SNL (2), NNSA Abq
  Service Center
• Commercial
• Qwest (Albuquerque), Big Byte (Albuquerque),
  Qwest Cyber Center (Denver), Recovery Point
  Systems (Gaithersburg), Qwest Cyber Center
  (Sterling VA), Qwest Cyber Center Sunnyvale,
  Lockheed Martin Information Technology Data
  Center (Beltsville MD), Lockheed Martin
  Enterprise Information Systems (Orlando FL),
  Verizon Data Services Inc. (Temple Terrace FL),
  Verizon (NOC) DFW Airport, Verizon Enterprise
  Solutions (Arlington VA), VeriCenter Denver,
  VeriCenter Atlanta, Matic Site Laguna Industries
  (Laguna Pueblo)

16
FINAL FOUR

• Commercial Recovery Site
• Vericenter, Denver
• Established commercial operation
• Close to Albuquerque
• Build-To-Suit
• Matic Facility, Laguna Pueblo
• 50 miles from SNL
• Long term lease agreement needed
• DOE/NNSA
• Honeywell (FMT), Kansas City
• Would require line item authorization
• Sandia Livermore
• Sandia controlled
• Would require some facility upgrades

17
TACTICAL VS. STRATEGIC

• Tactical approach addresses immediate needs with
  existing resources and timeline
• SNL California selected as best fit
• Under SNL control
• Existing facility and trained staff
• Require some facility upgrade and modification
• Located in severe earthquake zone
• Strategic approach for long term capability
• Establish a third facility for both NM and CA
• Build a dedicated facility at another location
• May be government owned or long term lease/option
• Potential for shared facility, recovery
  capability with other NWC sites

18

• QUESTIONS?