Title: Operational Recovery Planning
1Operational Recovery Planning
- Presented by the California State Information
Security Office
2Agenda
- Introductions name and agency
- CA State Information Security Office
- Definitions
- Four Types of Continuity Plans
- Review of BL 07-03 ORP Changes
- ORP-COOP/COG Alignment
- Discuss Test Scenarios
3 4State Information Security Office
- Vision
- Leading the way to secure the State's
- information assets.
- Mission
- To manage security and operational
- recovery risk for the State's
- information assets by providing
- statewide direction and leadership.
5Definitions
- Emergency Response
- Business Continuity Planning (BCP)
- Operational Recovery Planning (ORP)
- Continuity of Operations (COOP)
- Continuity of Government (COG)
6Emergency Response
- The immediate reaction and response to an
emergency situation commonly focusing on ensuring
life safety and reducing the severity of the
incident. - Definition from Disaster Recovery Journal (DRI)
website at http//www.drj.com/glossary/
7Business Continuity Planning (BCP)
- Process of developing and documenting
arrangements and procedures that enable an
organization to respond to an event that lasts
for an unacceptable period of time and return to
performing its critical functions after an
interruption. - Similar terms business resumption plan,
continuity plan, contingency plan, disaster
recovery plan, recovery plan. - Definition from Disaster Recovery Journal (DRI)
website at http//www.drj.com/glossary/
8Operational Recovery Planning (ORP)
- The management approved document that defines the
resources, actions, tasks and data required to
manage the technology recovery effort. Usually
refers to the technology recovery effort. This
is a component of the Business Continuity
Management Program. - DISASTER RECOVERY PLAN (also known as
Operational Recovery Plan) - Definition from Disaster Recovery Journal (DRI)
website at http//www.drj.com/glossary/
9Continuity of Operations (COOP)
- The activities of individual departments and
agencies and their sub-components to ensure that
their essential functions are continued under all
circumstances. This includes plans and
procedures that delineate essential functions
specify succession to office and the emergency
delegation of authority provide for the
safekeeping of vital records and databases
identify alternate operating facilities provide
for interoperable communications and validate
the capability through tests, training, and
exercises. - Office of Emergency Services (OES)
10Continuity of Government (COG)
- The preservation, maintenance, or reconstitution
of the institution of government. It is the
ability to carry out an organizations
constitutional responsibilities. This is
accomplished through succession of leadership,
the pre-delegation of emergency authority and
active command and control. - Office of Emergency Services (OES)
11Relationship of Plans
12Inter-Dependencies
13Three Phases of Continuity
Departments
Emergency Response - Life Safety First 72 Hours
Restoration Business back to normal
IT Operational Recovery up to 30 days
Planning, Documenting, Testing, and Training
Business Recovery up to 30 days
Damage Assessment First 72 hours
Phase I
Phase II
Phase III
14IMPLEMENTATION OF PLANS
- Disruption of business occurs and you are
informed, next steps - 1. Emergency Response safety and security of
staff. - 2. Securing the site.
- 3. Activate COOP/COG Plan to ensure the
continuation of essential functions. - 4. Implementation of the communication plan.
- 5. After assessing incident, determine if
implementation of BCP ORP is required. - 6. Contact SISO to report incident.
- 7. Implement BCP and ORP
15Budget Letter 07-03
- SAM Section 4843 Operational Recovery Planning
- Use results from risk analysis and business
impact analysis to identify critical business
functions. - Include the operational recovery considerations
and costs in FSRs. - Develop ORP as part of a complete continuity
program.
16Budget Letter 07-03 Continued
- SAM Section 4843.1 Agency Operational Recovery
Plan - Rewritten to clarify and enhance operational
recovery requirements. - Removal of minimum components from policy.
- SIMM 65A ORP Documentation for Agencies
Preparation Instructions - Requires ten minimum components in ORP.
- Additional three components for agencies without
a BCP or COOP/COG.
17ORP Documentation Revised
- Components to be included in the ORP updated in
January 2007. - The April and July quarterly filers must provide
a cover sheet indicating where the information
for each topic area in SIMM 65A is located in the
agencys Operational Recovery Plan. -
- All components listed in SIMM 65A must be
addressed and included in agencies ORPs
beginning in October 2007.
18Changes for ORP Development
- Overall
- Requires more details
- New Components
- Backup and offsite storage
- Data Center Services
- Contact information
- Removed from SAM and Policy
- Damage Recognition
- Preparation of cost-benefit analysis
- Selection of alternative
- SIMM Section 140A
19New Requirements
- ORPs must describe
- Agency Administrative Information
- Critical Business Functions/Applications
- Recovery Strategy
- Backup and Offsite Storage Procedures
- Operational Recovery Procedures
- Data Center Services
- Resource Requirements
- Assignment of Responsibility
- Contact Information
- Testing
20Supplemental Requirements
- Agencies that have not developed and implemented
a full business continuity plan or COOP/COG must
also address and include the following in their
ORP - Damage Recognition and Assessment
- Mobilization of Personnel
- Primary Site Restoration and Relocation
21Agency Administrative Information
- A communication plan should include strategy on
- How information will flow (escalation)
- Decision making processes
- Interrelationship among agency resources for
response, recovery and resumption
22Example - Escalation Process
- Single site, minor impact. User calls into Help
Desk with possible virus infection. Communication
Plan strategy includes - Process to dispatch field support to check PC
- If infected, take steps to identify and
quarantine - notify ISO and IT Management
- Eradicate virus
- Verify virus has not spread
-
23What would you do?
- Multiple site, major impact. The virus outbreak
has spread from your headquarters to your remote
offices and is running rampant. The anti-virus
software will not eradicate it and all the
systems in your agency are being impacted. - What would your communication plan need to
include?
24Communication Plan
- Document
- Who to contact and under what circumstances
- Lists name, phone , cell , home , email
address - Includes Chain of Command Management, other
pertinent staff (ISO, ORP Coordinator, etc), and
contractors - Distribute to applicable staff
- Providing training to staff
- Collect when duties change or staff leaves
25Sample Call Lists
- Wallet size cards
- Name, work , cell , home , email
- Call Tree
- Manager calls supervisor
- Supervisor calls his/her staff
26Critical Business Functions/Applications
- This section includes a description of
- Critical business functions and their supporting
applications - Maximum Allowable Outage (MAO) for each
application - Recovery priorities
27Example - Critical Business Function
- Single site, minor impact. Help Desk identifies
that the services on the email server are not
working. As a critical business function,
recovery strategy includes - Process for IT staff to check services
- If denial of service, follow internal procedures
to identify and mitigate. - Notify ISO and IT Management
28What would you do?
- Multiple site, major impact. The email server has
crashed, there are both hardware and software
failures. Rebuilding the server will require
replacement hardware, which will take several
days to acquire and configure. - What would your Critical Business Functions /
Applications need to include?
29Procedures for Critical Functions
- Document
- Critical Business Functions
- Recovery Procedures
- Responsible individuals or team for recovery
- Distribute procedures to applicable staff
- Provide training
30Sample Procedure
- Repair/replace hardware
- Restore database structure
- Restore post office
- Restore gateway connectivity
- Rebuild database
- Keep users/management informed
31Recovery Strategy
- Recovery strategy should include alternate
recovery site/sites that include - Location of all sites
- Requirements of facilities/equipment
- Contact numbers
32What would you do?
- Single site, minor impact Your department is
located in several locations. A building adjacent
to one location has a fire, the fire did not
spread to your site. The Fire Dept and Law
Enforcement block the street, so there is no
access into your building. - What would your recovery strategy need to
include?
33Recovery Strategy
- Communication plan for employees, management, and
contractors. - List all office locations.
- Identify the alternate location. If multiple
locations are available, prioritize them. - Address what functions could be restored at each
site. - Determine who would need to be called, include as
the contact list.
34Sample Recovery Strategy
- Department has three locations
- 1234 Headquarters St., Sacto, 95814
- 5678 Anywhere St., Sacto, 95825
- 9876 SomePlace St., LA 90210
- Critical operations would be restored at an
unaffected site (identify priority and equipment
needed). - Contact
- J Resto at (916) 555-1212 for Headquarters
- R Quick at (916) 444-1212 for Anywhere
- M Pia at (213) 555-1212 for SomePlace
35Backup and Offsite Storage
- The backup and offsite storage procedures should
include - Retention schedule
- Procedures
- List of authorized staff
- Account information
- Contacts of offsite storage
36What would you do?
- The data on one of your critical applications was
corrupted and its MAO is 4 hours. It is 530 pm
on Friday and Monday is a holiday. The business
area have staff scheduled to work Saturday on
this system. Technical staff has gone home, and
several are out of town for the weekend. - What would your backup and offsite storage
procedures need to include?
37Details Backup and Offsite Storage
- Document
- Retention schedule
- Detailed procedures
- Hardware and software (include version)
- Offsite storage details (location, acct )
- Retrieval of backups (contacts (24x7) and
personnel authorized to retrieve) - Process to identify data to be restored
38Operational Recovery Procedures
- These procedures systematically detail the
operational procedures for recovery in a timely
and orderly way, they should include - Detailed procedures that the backup or other IT
professional could follow - High-level network diagram that includes all
critical applications
39Data Center Services
- This section should include a
- Description of service to be provided.
- Interagency agreements, memorandums of
understanding, or contracts. - Specific coordination efforts with the data
center critical to the recovery efforts.
40Example Minor Impact
- Single site, minor impact. Your Web server
providing access to one of your critical
applications located at DTS has been compromised.
You have contacted DTS and DTS is working to get
the server back online within the hour. - What would your need to include?
41What would you do?
- Multiple site, major impact. There was a fire in
a facility adjoining DTS facility where the
servers are housed. The sprinkler system was
activated and the servers had to be powered down.
There is significant water damage. There is an
estimate that it will take 14 to 21 days to
reestablish services. - What would your plan need to include?
42Details - Data Center Services
- Expectations
- Meet with Data Center to identify
- Hardware/Software requirements
- Services required
- Timeframe for services
- Document Agreement Before its needed
- Create a Service Level Agreement (SLA) or
Memorandum of Understanding (MOU) - Develop Recovery Procedures
43Resource Requirements
- This is a comprehensive list of
- Equipment
- Software
- Telecommunication needs
- Data
- Hard copy manuals
- Personnel essential for recovery
44Assignment of Responsibility
- Designation of responsibilities and assignments
should be listed. Procedures should include job
title, and not individual names, for the recovery
process. - Individuals names can be placed in a single
location for ease of maintenance.
45Contact Information
- There are two types of contact information to be
collected - Employees, including management.
- Resource List including contractors, Major
Service providers, vendors, other government
entities, and outside resources critical to the
recovery process.
46Contact List
- Employee contact information should be designated
as sensitive, and provided to authorized
individuals. - Resource lists typically have business contact
information. This information can be provided
more widely.
47Testing
- Annual testing of the ORP is essential to
- Ensure for training the management and recovery
teams. - Validate that the procedures have the appropriate
level of detail. - Verify Call Back lists are current.
- Confirm that Recovery strategies are appropriate
for your environment.
48Governors Office of Emergency Services
- Introduction
- Mission and Goals of OES
- SEMS/NIMS
- Disaster Service Worker
49Planning
- Be Smart, Be Responsible. Be Prepared. Get Ready
Campaign - Your Intranets and Emergency Preparedness
- Executive Order S-04-06
- State Emergency Plan /
- COOP-COG/ORP
50Training and Testing
- Emergency Management Training Requirements for
Public Employees - The California Specialized Training Institute
(CSTI)/OES Training Branch - How to develop a Table Top Exercise (TTex)
- Definition of a TTex
- The 8 Step Process Used to Design a TTex
- After Action/ Corrective Action Process
- California Master Exercise Calendar (CMEX)
51State IT Strategic Plan Action Item
- To align the ORP and COOP/COG, a work group has
been established to - review processes
- define terminology
- evaluate reporting requirements
52Resources
- SISO web site http//www.infosecurity.ca.gov/ORP/
- Budget Letter 07-03 ORP Policy Changes
- http//www.dof.ca.gov/OTROS/StatewideIT/IT_BdgtLtt
rs.asp - ORP Policy in the State Administrative Manual
(SAM) - Operational Recovery Planning
http//sam.dgs.ca.gov/TOC/4800/4843.htm - Operational Recovery Plan http//sam.dgs.ca.gov/T
OC/4800/4843.1.htm - ORP SIMM 65A http//www.infosecurity.ca.gov/Pol
icy/
53Contact Us
- Rosa.Umbach_at_dof.ca.gov
- (916) 445-1777 ext. 3242
- Colleen.Pedroza_at_dof.ca.gov
- (916) 445-1777 ext. 3224
- SISO Office
- email security_at_dof.ca.gov
- Telephone (916) 445-5239
- www.infosecurity.ca.gov