Business Resumption Planning Lesson Learned One Companys Story

1
Business Resumption PlanningLesson Learned
One Companys Story

• September 11th, 2001
• by
• Fred V. Diers CRM, FAI
• Entium Technology Partners, LLC
• www.entium.com

2
We will discuss

• The impact of the terrorist attacks on the World
  Trade Center
• The lessons learned in the aftermath of the
  largest terrorist attack in US history
• The impact on traditional business continuity
  and disaster recovery planning
• The heroic recovery efforts of the Port
  Authority of NY NJ and Entiums Role
  supporting that recovery

3
The Port Authority of NY NJ

• First bi-state agency created by act of Congress
  in 1921
• Assumed operation of all major NY and NJ Airports
  in 1947
• Developed worlds first container port in Newark
  Bay
• Designed, built and operates the Holland and
  Lincoln Tunnels
• Designed, built, and operates 4 bridges
  connecting Manhattan Island and New Jersey

4
WTC Looking Up
5
The World Trade Center

• The Port Authority was tapped to undertake this
  project
• The PATH became part of the WTC Project
• The 1993 bombing killed 6 PA employees
• The following are specific facts about the World
  Trade Center complex

6
Facts - I

• WTC Complex Consisted of Seven Buildings
• Owner Port Authority of New York and New Jersey
• Architect Minoru Yamasaki Associates
• Engineer John Skilling Les Robertson, based
  in Seattle
• Contractor Tishman Construction Company
• Ground Breaking August 5th, 1966
• Opened April 4th, 1973

7
Facts - II

• Dimensions 200-Feet tower sides.
• Total Area 12 Million square feet 40,000 sq ft
  per floor
• Height 110 Stories high, 1368 and 1362
  feet (417 and 415 meters)
• Elevators 239 Elevators and 71 escalators
• Earthwork 1.2 Million cubic yards were excavated

8
Facts - III

• Steel 200,000 Tons
• Concrete 425,000 Cubic yards
• Windows 43,600 Windows600,000 square feet of
  glass
• Cooling 60,000 Tons of cooling capacity
• Antenna 360 Feet high
• Capacity 50,000 Employees

9
(No Transcript)
10
Satellite PhotoLower Manhattan
RECORDS INFORMATIONMANAGEMENT
WTC
11
Manhattan
12
Entiums Role

• Designed built network for the Port Authority
  Engineering Departments distributed 1,500 person
  workforce at WTC and other NY and NJ facilities.

13
PA Engineering Network Included

• High-availability servers, fault-tolerant storage
  area networks, mirrored network infrastructure
• Redundant tape back-up facilities
• Mutual hot-site back-up for NJ and WTC
• Integrated document and drawing management
  systems
• PA-wide intranet, browser-based front end for all
  Engineering Department applications

14
September 11th

• Entium staff was in early to its PA Engineering
  office on the 72nd floor of WTC1 (the north tower)

15
RECORDS INFORMATIONMANAGEMENT
Satellite PhotoSmoke from Burning Towers
16
The Escape

• From the time the first plane hit, Entium staff
  make an anxious and exhausting descent down
  smoke-filled stair wells
• This descent takes one hour
• Entium staff exited the World Trade Center on the
  Northeast side, moments before WTC2 collapsed
• Entium staff escapes lower Manhattan and walks
  north through Midtown
• Entium staff reunites Tuesday night at office in
  Jersey City

17
Recovery Chronology

• The Port Authority Command Center moves to Jersey
  City facilities
• September 12th
• DRP team establish contact with employees
• September 13th
• Entium staff reported to the PA Command Center
  where they
• Secured back-up tapes
• Initiated inventory of available equipment
• Initiated inventory of backup media and files

18
World Trade Center Recovery What Didnt Work

• Lost Internet Access from PA Wide Area Network
  Single ISP
• Communications Cell phones widely inoperative
• Netware Directory Services
• Inadequate Replicas
• Prolonged Recovery
• Corrupt Directory Services

19
Recovery Chronology

• September 14 16
• Initial emergency hardware orders placed
• Cisco switches, 400 PCs, and necessary network
  hardware
• Recovery of priority applications and data begins
• September 17
• Emergency hardware arrives
• Recovery of priority applications begins

20
The Aftermath

• 20 million square feet of office space lost in
  lower Manhattan.
• By September 17th , over 2.5 million square feet
  of office space snapped up in New Jersey
• Comdisco received over 70 disaster declarations
• An estimated 70,000 workstations and associated
  data lost

21
World Trade Center Recovery What Worked
RECORDS INFORMATIONMANAGEMENT

• Able to restore 95 of user data within 1 week of
  attack
• Able to restore 28 of 30 applications within 2
  weeks of attack
• Port Authority Wide Area Network Survived
• Lost primary SONET core
• Redundant back-up links (T3) survived
• Public Communications
• Internet Inherent Distributed Architecture
• Instant Messaging (Blackberry)

22
Lessons Learned

• Leverage internet during crises, e-mail, instant
  messaging, web sites
• Back-up off site
• Secure alternate work space
• Identify vital paper documents
• Duplicate and store off site

23
World Trade Center Recovery What Didnt Work

• Human Element
• Transfer of back-up tapes to off-site storage
• Off-line servers missed back-up regimen
• WAN segment downtime/performance
• User failure to follow established business
  procedures
• Use of desktop storage instead of LAN storage
• No desktop backup strategy
• No plan for critical and historical documents

24
Traditional Disaster Recovery

• Technology became integral to business processes
• Traditional 72-hour recovery window became 4 to
  24 hours
• E-Business demands 24x7
• Traditional plans covered physical site outages,
  power, fire, natural disasters, etc.
• Consolidated data center
• Alternate data center short distance from
  primary site

25
New Risks / Threats to Businesses

• The economy lives on proprietary information,
  the competitive edge increasingly at risk
• Outsourced service providers
• Content/application links to 3rd-party providers
• Internet Security Essential not an optional
  business communications tool
• Site security breaches
• Virus attacks
• Data Theft vandalism by disgruntled employees

26
New BCP Strategies

• Telecommute
• Remote access
• Remote work centers (prenegotiate rates)
• Standby contracts with 3rd-party providers
• Direct nonessential personnel to alternate
  communication sources
• Create virtual business teams
• Relocate mission-critical facilities to secure,
  low-profile locations

27
New BCP Strategies

• Prioritize business applications
• Take nonessential applications off line
• Limit use bandwidth-intensive applications
• Evaluate data replication options
• Host-based disk block replication
• Windows-based file replication
• Log-based database replication
• Outsource enterprise-level applications

28
New BCP Strategies

• Business Continuity Plan Components
• Disaster Recovery Plan
• Business Resumption Plan
• Business Recovery Plan
• Contingency Plan
• Business Impact Analysis, Risk Analysis, Recovery
  Time Objectives, Recovery Point Objectives
• Vital Records Identification and Protection Plan

29
New BCP Strategies
RECORDS INFORMATIONMANAGEMENT

• Fault Tolerant Networks
• Enterprises becoming increasingly dependent on
  network service providers (and level of
  resilience and redundancy built into those
  networks)

30
Results

• Businesses must be prepared for new risks
• Businesses must invest necessary resources
• Two out of five businesses that are struck by
  disasters cease operations within five years
• 60 of companies dont have adequate business
  continuity plans, let alone procedures for
  alternatework space

31
(No Transcript)
32
. . . at the convergence of connectivity,
systems, and knowledge
33
Contacts for Entium Technology Partners, LLC

• Sales and Marketing
• Don Burt Director
• 610.415.7207 office
• 610.213.3744
• dburt_at_entium.com
• Records and Information Management
• Fred Diers Principal
• 610.415.7227 office
• 610.212.9412 cell
• fdiers_at_entium.com
• Technology Solutions
• Document Management, Application Development,
  Content Management, Web Development
• William R. Anderson, Jr. (Bill) Principal

• Network Solutions
• George Broadbent Director
• 610.415.7216 office
• 610.247.0899 cell
• gbroadbent_at_entium.com
• Geo-database Solutions
• Joe Spollen Director
• 610.415.7218 office
• 610.213.9606 cell
• jspollen_at_entium.com
• Entium Technology Partners, LLC
• 610.415.7200 - main
• 610.415.7222 fax