Headline here

1
Testing Your Operational Readiness with Outside
Agencies
Tom Clark, CBCP Director, IT Infrastructure
Continuity Services Liberty Mutual Group
2
Operational Readiness

• Capability of an organization to continually
  perform the functions for which it is designed
  
• Assessed according to internal standards
• the difference between an organizations actual
  capability and its absolute potential
• Measured in terms of how soon an organization can
  reach its peak operational capacity

3
Testing Operational Readiness

• Every organization must test its operational
  readiness plans regularly to continually prepare
  for any disruption.
• Once a certain level of operational readiness has
  been reached within a corporation, it makes sense
  to continually improve its continuity capability
  by interjecting as much realism as possible into
  the exercise scenario.

4
Testing Operational Readiness

• Annual large scale exercises are recommended so
  the operational readiness of the organization can
  be tested.
• Although employees never know exactly how they
  will act or perform in a disaster, exercises
  allow them to learn specific skills that will
  improve the likelihood of success in a real
  event.

5
Exercise Effectiveness

• How can a real event be simulated so that all
  employees know their roles and responsibilities
  during an incident and have the opportunity to
  identify gaps in the continuity process?
• Involving outside agencies is one great method
  

6
Testing Operational Readiness with Outside
Agencies

• How?
• Involve outside agencies in the planning process
  and the actual exercise
• Why?
• In a true emergency situation, employees will
  need to know the roles of various external
  agencies
• Employees must know exactly what is expected from
  external agencies
• The processes for communicating with external
  agencies should be well defined and understood

7
Our Process
8
How We Established Relationships

• Met regularly with other local businesses
• Attended association meetings and emergency
  management conferences
• ACP local chapter (Association of Contingency
  Planners)
• NEDRIX (NorthEast Disaster Recovery Information
  X-change )
• State emergency management conferences

9
How We Established Relationships

• Meeting with Local Businesses
• Our organization is part of a group of local
  businesses that meet regularly for lunch and
  learn sessions.
• Every participating business works with the
  others to develop strategies for working together
  during an incident or a crisis.
• This may include something as simple sharing a
  parking lot, or even office space.

10
How We Established Relationships

• Local businesses share with each other the dates
  and times of exercises and drills.
• As an example, A drill occurred between the local
  hospital and the local S.W.A.T. team.
• Our security personnel and employees were made
  aware of what was going on next door.

11
How We Established Relationships

• During meetings with local businesses, members of
  state and local agencies are invited as guest
  speakers.
• We have found that this platform serves both
  parties well. In one hour or less, the speaker
  can get his or her message out to several
  representatives from multiple local businesses.

12
How We Established Relationships

• Speaking at one meeting, a local fire chief
  provided recommendations regarding employee
  safety.
• He identified the types of areas within a
  building that could potentially benefit from
  having floor plans and escape routes posted.
• Each business shared its strategy for
  establishing an Emergency Operations Center if
  there was an event.

13
How We Established Relationships

• We exchanged information with the fire department
  regarding roles and responsibilities during an
  emergency.
• This fire department, shared its strategy for
  establishing an Emergency Operations Center (EOC)
  to handle emergencies during a major storm such
  as a hurricane or noreaster.

14
How We Established Relationships

• Members from the local police department have
  also participated in these meetings.
• They collected written data from each business
• Who would be a primary contact and backup for the
  business during an event.
• How can these contacts be reached?
• Does the business have an emergency response
  team?
• Are there hazardous materials on site?
• How many employees are located in each facility?

15
Other Topics Discussed with Local Agencies

• What are the expectations once local agencies
  are on the scene?
• They will likely assume total responsibility of
  the event.
• How would they like us to communicate with them?
• What are their emergency management procedures?

16
Attending Conferences

• Many of our employees attend conferences
  throughout the year that many outside agencies
  also attend.
• For example, at a recent NEDRIX conference, FEMA
  had representatives in attendance.
• During the NH Emergency Preparedness Conference
  in 2008, our Incident Commander met a former
  division commander for a local police department.
  He is now a Deputy Sheriff with another local
  department.

17
Attending Conferences

• Because of our new relationship, the deputy
  sheriff was invited to participate in our event.
  He introduced us to colleagues that would like to
  take part in the exercise.
• This resulted in the local Deputy Fire Chief and
  the State Emergency Preparedness Coordinator for
  the Department of Homeland Security attending.

18
Building on Existing Relationships

• Our organization has been actively building
  relationships with various local and state
  agencies including
• Local Police Departments
• Local Fire Departments
• County Sheriff Departments
• State Police
• NH Emergency Management Agencies

19
Keys to Building on Existing Relationships

• When meeting with outside agencies, we focus on
• Better understanding their capabilities in a
  crisis
• Discussing how they handle incidents
• Defining their expectations of our organization
  during an event

20
Our Relationship with the Fire Department

• We invited the local fire department to assess
  our facility and provide training to our
  employees.
• Employees were given detailed fire extinguisher
  training, as well as basic guidelines for knowing
  when to attempt to put out a fire and when not
  to.

21
Our Relationship with the Fire Department

• The assessment of the facility gave the fire
  department a chance to see the layout and
  structure of our building.
• Employees were also provided with general
  information regarding structural fires and how
  that information related specifically to our
  facility.

22
Our Relationship with the Police Department

• The local police department has provided
  awareness programs to our employees.
• These programs include topics such as
• Refuse to become a victim
• Violence in the workplace

23
Our Relationship with the Police Department

• The phone system at our facility is set up in
  such a way that to call outside of the building,
  employees must first dial 9.
• This resulted in the police department responding
  to several false 911 calls when employees also
  had to dial 1 and an area code.
• Our company and the local police department
  worked together to developed a strategy for
  preventing false calls in the future.

24
Local Emergency Response Team (LERT)

• Liberty Mutual Group has adopted the Incident
  Command System (ICS), a nationally recognized
  structure originally designed in the 1970s to
  combat wildfires.
• In addition, our organization has an Initial
  Assessment Team (IAT) which is made up of
  executive management. This team has the
  responsibility of determining if the event
  requires activation of the LERT.

25
Local Emergency Response Team (LERT)

• The Incident Command System (ICS) is comprised of
  five teams
• Command
• Operations
• Logistics
• Planning Intelligence
• Finance

26
Planning Our Operational Readiness Exercise

• Operation Safehouse
• Determine exercise goals
• Scope
• Objectives and how we measure our success
• Participants and the description of their roles

27
Exercise Type and Scope

• A exercise involving enough complexity to test
  the operational readiness of our processes,
  people and technology
• Create an exercise that describes a disruptive
  event that would require concise communication
  between our company and various outside agencies

28
Exercise Goals and Objectives

• Involve multiple company physical facilities in
  an event that requires the utilization of their
  existing business continuity plan to manage the
  event
• Assess the ability of the team to manage a
  prolonged event.
• Assess the effectiveness of the communication
  between the team and outside agencies.
• Assess the ability of the entire team to activate
  plans effectively.

29
Exercise Goals and Objectives

• Assess the size and composition of the team.
• Assess the ability of the team to provide
  employee and stakeholder communications, press
  releases, and customer notifications.
• Assess the ability of the Incident Commander (IC)
  to manage the Emergency Operations Center (EOC )

30
Participants

• Local Emergency Response Team (LERT) in
  Portsmouth, NH
• Local Emergency Response Team (LERT) in Kansas
  City, MO
• Corporate Emergency Response Team (CERT) at our
  Boston Headquarters
• Local police departments
• Local fire departments
• State emergency management agencies

31
Preparing for the Exercise

• Choosing the scenario
• Involvement of and support from Senior Management
• Selecting the design team
• Selecting the simulation team (SIM Team)
• Coordinating the exercise with local agencies

32
Choosing the Scenario

• We researched the scenarios of past exercises and
  the lessons learned from those events
• We looked at current events to assist in the
  development of the scenario
• The basic concept for the scenario was developed
  five months prior to the exercise.

33

• Influential Current Events
• Alloy Fabricators of New England, Inc.
• Randolph, Massachusetts in April 2008
• One dead and one injured
• Atlantis Plastics
• Henderson, Kentucky in June 2008
• Six dead and one injured

34
Other Influential Current Events

• Economy
• The large decline in the stock market during
  September 2008 was the trigger for our
  assailants rampage. He was a temporary
  contractor with access to the company Data
  Center, whose contract had not been renewed. His
  frustrations resulted in reactions designed to
  cause a great deal of damage.
• Weather
• Tropical Storm Omar was also used as a factor in
  the scenario to increase the complexity. While
  Omar was not geographically close to the facility
  in NH, part of the exercise was to assess how our
  company could handle a second incident at another
  facility while a primary Data Center was
  disrupted.

35
Senior Management Involvement

• The success of the exercise depends heavily on
  the involvement of and support from senior
  management
• Only the CIO was informed of the plan.
• The Initial Assessment Team (IAT)
• Knew that the simulated event was going on, but
  not the details
• Knew that they would be called
• Had time scheduled on their calendar in advance

36
Senior Management Involvement

• To increase the realism of the exercise, a design
  team wrote scripts for senior management to use
  during the event
• These scripts provided other teams with realistic
  actions during an event such as this.

37
Selecting the Design Team

• To increase the complexity of the scenario, the
  Design Team was created to address realistic and
  potential gaps in processes, people, and
  technology
• We selected five Subject Matter Experts (SMEs)
  with expertise relative to the type of event
  selected.

38
The Design Team

• The areas represented by our subject matter
  experts
• Disaster Recovery
• Data Center Facilities
• Information Security
• Physical Security
• User Support Center (USC call center)

39
Creating the SIM Team

• As the exercise date approached, members were
  added to the design team to create the Simulation
  Team (SIM Team).
• The new members were not given any details of
  the scenario prior to the exercise. They were
  given their roles and responsibilities the day
  before the exercise.
• The SIM Team simulated calls to the Emergency
  Operations Center (EOC)
• They had pre-scripted roles to play throughout
  the event

40
Coordinating with Outside Agencies

• The participating members of the outside agencies
  assisted the design team in the preparing the
  scripts that would be used during the exercise
• They worked with the SIM team to create realistic
  inputs and outputs to provide a true
  representation of interactions with outside
  agencies.

41
Coordinating with Outside Agencies

• Members from outside agencies participated in
  role playing during the exercise.
• In addition, members of outside agencies were
  present in the EOC and observed the teams
  actions and reactions during the exercise.

42
Preparations

• In preparation for this exercise, we invited the
  NH State Police to visit our Portsmouth facility
  and give a presentation regarding violence in the
  workplace.
• The NH State Police took pictures throughout the
  building and offices to help identify safe and
  non-safe areas for our employees during a
  violence in the workplace type of event.

43
Preparations

• The NH State Police developed and delivered an
  assessment report to senior management with
  suggestions on what our employees should do in
  the event of an active shooter scenario.
• The assessment report reflected the types of
  areas that employees should avoid during an
  active shooter situation and how to make
  themselves less of a target.

44
Scenario of Operation Safehouse

• The day began at 800 AM with the LERT attending
  training in a company facility in Dover, NH
• Dover, NH is 15 miles north of the Data Center in
  Portsmouth, NH
• In addition, Tropical Storm Omar was heading
  towards Miami, FL

45
Scenario of Operation Safehouse

• Suddenly, an incident occurred at the Portsmouth
  Data Center.
• At 900 AM, a software contractor, with access to
  the Data Center, took hostages, shot some
  employees, and detonated an explosive device
  damaging equipment.
• It is suspected that additional explosive devices
  are in the Data Center and throughout the rest of
  the building.

46
Scenario of Operation Safehouse

• At 1000 AM the shooter shot and killed himself.
  
• At 1015 AM local authorities rescued the
  hostages, secured the body of the assailant, and
  declared the facility a crime scene.
• The Portsmouth Data Center was non-functional and
  the Disaster Recovery Plans had to be activated
  in the Kansas City, MO Facility

47
Scenario of Operation Safehouse

• The second part of the exercise moves forward two
  days
• Due to the incident
• One employee was killed by the assailant
• Ten employees were injured by the assailant due
  to shots fired
• Recovery activities have been in progress in
  Kansas City, MO

48
Scenario of Operation Safehouse

• Tropical Storm Omar has turned into Hurricane
  Omar
• It is expected to directly hit Miami, FL
• After two days, local authorities released the
  Data Center back to our security and facility
  teams to conduct a damage assessment.

49
Conducting the Exercise

• Morning Session
• The exercise began with a simulated radio
  broadcast
• - Breaking News -
• Reports of shots fired and hostages taken at
  local business in Portsmouth NH

50
Conducting the Exercise
Morning Session

• A simulated phone call came in from the Data
  Center facilities manager notifying the Incident
  Commander (IC) of the situation.
• The building had been evacuated, and the police
  department had arrived on scene.
• Tactical units were enroute.
• Not all employees had been accounted for.
• Reports of gunfire had been made by some
  employees.

51
Conducting the Exercise
Morning Session

• The Incident Commander (IC) activated the IAT and
  established a conference bridge to brief the IAT
  on the situation.
• The determination of the IAT was to activate the
  LERT
• The Incident Commander then reached out to the
  Corporate Emergency Response Team (CERT) in
  Boston to advise them of the situation and
  activation of the Portsmouth LERT.

52
Conducting the Exercise
Morning Session

• The LERT developed a series of short-term
  objectives focused around five key areas
• People
• Facilities
• Technology
• Mission-critical activities at risk
• Communication.

53
Conducting the Exercise
Morning Session

• The Incident Commander gave a short briefing to
  the entire Local Emergency Response Team (LERT)
  and then work began.
• Teams worked together to gather data and decide
  on action items moving forward
• The team developed an Incident Action Plan to
  determine both the operational and support
  activities to address the incident

54
The Incident Action Plan Addresses
Conducting the Exercise
Morning Session

• What do we want to do?
• Who is responsible for doing it?
• How do we communicate with each other?
• What is the procedure if someone is injured?

55
Conducting the Exercise

• Afternoon Session
• The second half of the exercise was two days
  after the shooting.
• We had not been able to gain access to any part
  of the company facility in Portsmouth, NH since
  the incident.
• After a lengthy search, the NH State Police Bomb
  Squad found no other explosive devices.

56
Conducting the Exercise Afternoon Session

• The building was finally released back to us and
  teams have begun the damage assessment process.
• The damage assessment identified several pieces
  of equipment to be replaced in order to restore
  the Data Center.
• The Disaster Recovery Plan implemented production
  at the alternate DC in KC which will be
  operational for at least 30 days.

57
Conducting the Exercise Afternoon Session

• Where possible, employees have been using VPN to
  work from either their home or other company
  locations.
• Additional staff required to support production
  at the alternate Data Center in Kansas City, MO
  have been deployed.
• Due to the traumatic incident, it was difficult
  to obtain specific technical resources qualified
  to support production.

58
Conducting the Exercise Afternoon Session

• Business units have been attempting manual
  workarounds while waiting for systems to be
  restored.
• Call center call volumes have dramatically
  increased due to the additional complexity of
  Hurricane Omar.

59
Outside Agencies and the Role They Played

• Several members from Outside Agencies were
  located in a separate room with the SIM team to
  initiate and receive simulated telephone calls to
  and from the EOC to create more realism
• Others representatives from Outside Agencies were
  positioned in the EOC to observe the interaction
  of our company team members

60
Outside Agencies and the Role They Played
Examples of role playing telephone calls

• Who is the Incident Commander?
• This is the Fire Department, we need to inform
  you that all power to the Data Center will be
  shut down due to concerns of electrocution.
• This is the police department, we need a list of
  all employees that have entered the building this
  morning as well as copies of any floor plans and
  video surveillance available at this time.

61
Outside Agencies and the Role They Played
Examples of role playing telephone calls

• We are searching the building and do not have
  access to certain areas, who can assist us in
  gaining access?
• This is the police dept we need to speak to
  someone from human resources. We need to gather
  any information available on the assailant.
• This is the fire department. Based on the
  condition of the Portsmouth facility, you will
  need a certificate of occupancy before your
  employees are allowed to return to work.

62
Outside Agency Observer Comments

• Deputy Chief of the local Fire and Rescue
  Department
• Your Organization displayed an amazing
  commitment to business continuity for your
  companies customers, and also caring for its
  employees during this exercise. Your
  organization is clearly a seasoned company in
  emergency planning and crisis management, having
  perfected the corporate Incident Command System
  (ICS) after many years of practice. As a citizen
  of this town and as someone who is insured
  through your company for auto and homeowners
  insurance policies, I am very impressed and feel
  that no matter what happens in our world that
  your company will go on and deliver. Thank you
  for inviting me into your exercise and allowing
  me to observe.

63
Outside Agency Observer Comments

• The Local Deputy Sheriff
• Based on my experience and training in both law
  enforcement and emergency management I would
  characterize my overall impression of your
  companys LERT Command Section during this
  exercise as outstanding. This overall impression
  is based on the Command Sections obvious grasp of
  their role in the ICS and their acceptance of
  responsibility for dealing with the complexity of
  the challenges presented during the exercise.

64
Criteria for Success

• Has there been sufficient cross training in roles
  and responsibilities for all team members?
• Are the communication processes clearly defined?
• Can every role be filled with more than one
  person?
• Are any applications expected to be up and
  running that may not actually be available in
  certain circumstances?

65
Criteria for Success

• Do employees understand the scope of the Disaster
  Recovery process.
• Is there a documented process for every team
  member?
• Are roles associated with a single employee? What
  happens if that person is not available?

66
Criteria for Success

• Using the LERT manual, could someone that has
  never participated in an exercise reasonably be
  able to perform some duties if needed?
• How will status reports be given?
• Who will give them?
• How will phone calls (both incoming and outgoing)
  be handled?

67
Our Results What Worked

• Using the Incident Command System made
  communications with outside agencies run much
  more effectively than in previous exercises
• It was evident that the increased amount of
  training prior to this exercise was of great
  value
• Teams anticipated action items and started
  working immediately using the LERT manual

68
Our Results What Worked

• The various teams (Command, Operations,
  Logistics, Planning Intelligence and Finance)
  assigned one person to handle messages between
  teams to improve communications
• This exercise was a great opportunity for the
  Kansas City, MO team to actively participate in
  the exercise

69
Our Results Areas for Improvement

• Employees began to slip out of their roles
  towards the end of the day
• Some roles were not as clearly defined as they
  could have been
• For example, the absence of two employees caused
  one team confusion regarding how to perform their
  specific tasks

70
Our Results Areas for Improvement

• Some teams were assuming that certain
  applications were readily available when in fact
  they were not
• Action planning meetings took longer than
  expected
• Improve the clarity of communications

71
Our Results Areas for Improvement

• Provide employees training on the Disaster
  Recovery Plan
• Improve the clarity of hand-off procedures for
  the shift changes during the exercise
• Add additional methods of communication such as
  whiteboards and overhead projectors

72
Our Results Areas for Improvement

• Increase the level of involvement with outside
  agencies
• Participate and observe outside agency exercises
  to gain insight into their process and procedures
• Some of our employees will play victims in a
  large scale readiness exercise simulated by a
  local County Emergency Management Agency

73

• Ultimately, our goal is to improve our emergency
  preparedness by working together with outside
  agencies in our community to guarantee the safety
  of our employees.