PreDisaster Recovery Plans

1
Pre-Disaster Recovery Plans

• Presentation to
• NACRC Legislative Conference
• March 3, 2007
• Carol Foglesong, Orange County Comptrollers
  Office,
• Orlando, FL
• and
• Paul Ketz, Broward County Records Division,
• Ft. Lauderdale, FL

2
Agenda

• Introductions
• Purpose
• Unpredictability of Disasters
• Business Continuity Planning Definitions
• Reasons for a Pre-Need Contract
• What do you need to consider?
• Orange Countys Process to Contract
• Other Resources
• Questions

3
Purpose

• We are not here because we are experts in the
  field of Business Continuity/Disaster Planning.
• We are here because all of us are struggling with
  this topic.
• We would like to share some of our experiences
  with you, and learn from the experiences of
  others here today.

4
Cycle of Disaster Planning
5
So, whats a disaster?

• Disasters come in all sizes and shapes
• Big events Hurricanes, Tornadoes, Earthquakes,
  Blizzards, Floods, Vandalism, Terrorism
• Smaller events Fires, Water Pipes, Roof Leaks
• Disasters impact
• A large geographic area
• A localized area (county or city)
• A particular office operation/location

6
Hurricane Wilma

• Exposed office suites are seen in the damaged
  Regents Bank Plaza on Tuesday, October 25, 2005
  in Fort Lauderdale. (Sun-Sentinel/Joe Cavaretta)

7
Hurricane Wilma

• The Broward County School Board building as seen
  from the 30th floor of the AutoNation Building in
  downtown Fort Lauderdale. (Brian Hekman, Coconut
  Creek)

8
Hurricane Wilma

• Virtually every window on the west side of the
  Broward School Board building in downtown Fort
  Lauderdale was blown out by Hurricane Wilma.
  (Sun-Sentinel /Carl Seibert)

9
Hurricane Wilma

• Repairs are underway at the Broward County
  Courthouse offices and parking garage in downtown
  Fort Lauderdale. (Sun-Sentinel.com)

10
Whats a common theme?

• Disasters are typically unexpected
• Disasters happen when you have or are making
  other plans
• Disaster size or impact cant be readily
  predicted
• Disasters arent all BIG events it can be the
  little ones that cause the most concern and
  frustration

11
Disaster Readiness

• Business Continuity Plan
• Staffing plans
• How to get up and running again
• Disaster Recovery Plan
• Setting priorities on what has to be tackled
  first
• Papers versus computers
• Forecast Disaster Plan
• Before the event planning
• Office preparations when a disaster is predicted

12
But who do you call for help?

• Youve made a list
• Businesses with addresses, names, phone numbers,
  faxes, and email addresses
• Services offered and needed
• Youve identified responsibilities of staff
• Who calls whom when
• Whos in charge
• Do you and your staff keep a copy of your
  plan/phone lists with you? It wont help if your
  list is at the office!

13
How to guarantee availability?

• If a large area is impacted, there will be many
  organizations needing help you may need to get
  in line
• If a small area is impacted, will you get the
  specific help you need quickly?
• If its a particular building or location
  impacted, whats the best solution?

14
One answer

• A PRE-NEED CONTRACT
• A commitment to your organization by a service
  provider
• You have a spot on the top of their list
• Youve pre-negotiated prices and services
• Youve already checked out who the provider is
  and their references

15
Getting the Pre-Need Contract

• You will have to convince your purchasing/contract
  group that a pre-need contract is not only
  possible, but logical
• You knowingly commit to going through the
  purchasing process of your organization even with
  no known disaster on the horizon

16
Cost

• Be clear no services no dollars
• But money has to be available when services are,
  in fact, necessary
• Does your organization require a set aside?
• Does your organization have a reserve fund?
• Contract has to be renewed
• Annually?
• Every 3 years?
• What are your renewal options?

17
Whos in charge of the procurement process?

• You may be the designated Records Management
  Officer for your organization, but you will need
  
• The purchasing/contract group
• Likely users must buy in
• Agreement on who sets the priorities for service
• Recognition that the disaster may be small or
  large scale
• Approval by your governing body
• Who will negotiate the contract? Do they have a
  solid understanding of your business needs to
  secure the best vendor?

18
Other Considerations

• Can you opt for higher priority by negotiating a
  retainer? Will the retainer be applied against
  services rendered?
• If something happened to your materials, how long
  would it take to access them?
• Do you have 24/7 access?
• How do you get someone to the facility?
• Is HIPAA a consideration in the
  custody/restoration of your records?

19
Response Time

• Do you have a Continuity of Operations Plan
  (COOP)?
• What kind of time will it take to respond to an
  event?
• For Emergency Services
• Fire, Plumber, Repairmen
• For you and your staff
• For the Paper-Recovery Vendor

20
What events are importantbased on your geography?

• Where are you located and what kinds of events do
  you expect every year?
• Florida Hurricanes
• California Earthquakes
• Northeast Snow
• Low lying Flooding
• What is it for you???

21
Vendor Capabilities

• How many linear feet of paper can be recovered
  per hour per piece of equipment?
• Is the Vendors equipment wholly owned by
  company, or is some subcontracted?
• Does the Vendor have a schedule of who will
  respond after they are first notified?
• Process for chain of custody of records
• Where will the vendor set up base camp?
• What kind of turn-around time

22
Vendor Experience

• How much experience will you require?
• What kinds of references do you want to see?
• Past Performance
• Program Name
• Owner / Architect
• Contract Amount / Percent Complete
• Completion Date

23
Be clear about the scope

• What media do you want used to recreate your
  records?
• Film
• Electronic
• Back to paper?!?
• Who can activate the contract
• Do you have other agencies/business units who
  need access to this contract?
• If multiple agencies/business units can access
  this contract, have you told the vendor who has
  priority?
• Whose budget is used to pay for services?

24
Writing the RFP

• Plagiarize!
• Dont reinvent the wheel unnecessarily
• Each organization has their own standard language
  and format
• Stick with the assistance particulars that you
  might actually need
• Have a way to compare apples and oranges between
  vendors/proposers

25
Computers vs Papers?

• Typically, theres already a recovery plan for
  computers, servers, databases, etc.
• Communication is key talk to your IT people and
  be sure they have a plan too!
• Computer recovery involves not only the data, but
  also the equipment (machines, hardware)
• There are specialized companies who handle this
  specialized service
• Make sure your organization has an active and
  realistic plan in place

26
Paper or Paper Plus?

• Paper in a warehouse, awaiting its retention time
  or its microfilming
• Paper in all those office filing cabinets and
  individual desk drawers
• Microfilm?
• Microfiche?
• CDs?
• Maps or big drawings?
• Other media besides computers?

27
Which documents first?

• Are your documents prioritized?
• Do you store records for multiple agencies? What
  gets saved first? Are your boxes segregated?
• All things being equal, whose records are most
  important?
• Does it make sense to secure your truly
  irreplaceable records elsewhere?

28
What type of paper recovery process will be used?

• There are multiple types of document
  recovery/restoration techniques
• Vacuum Freeze Drying
• Molecular Sieve treatment
• Desiccant dehumidification
• HVAC decontamination
• Gamma Radiation (Remove Mold/Bacteria)
• Are there techniques that cannot/should not be
  used on your records?

29
Paper Contamination Factors

• Debris
• Sewage
• Silt
• Haz-Mat materials
• Asbestos
• PCB
• Other petroleum products

30
Extras

• Re-filing, Re-indexing, Re-shelving
• Daily equipment rentals
• Small equipment purchases (shovels)
• Will they provide their own generators, or does
  the vendor require electricity (water)?
• Freight / Transportation Costs
• Travel / Per Diem or subsistence costs
• General Debris removal at your site to gain
  building access
• Warranty on services performed/document
  reproduction quality

31
Who does what?

• Inventory
• Packing
• Sorting/Identifying
• Prioritizing
• Labeling
• Loading/Unloading
• Re-shelving
• Security

32
Price per cubic foot

• Pack out
• Gamma Radiation
• Cleaning
• Drying
• Ozone
• Deodorization

33
Price per each item

• Audio tapes
• Blueprints
• Maps
• Compact disks
• Diskettes
• Manuscripts
• Laser disks
• Photographs

• Slides
• Microfiche
• Microfilm
• Negatives
• X-rays
• Pack out boxes
• Pack in boxes

34
Price per hour

• Will any of the vendors services be provided per
  hour in consultation fashion?
• Does the vendor have per hour charges for
  anything not specifically covered in your
  contract?

35
Where will the actual restoration and recovery
take place?

• Where will the vendor set up base camp?
• Onsite
• Remote Location
• Vendors Site
• How will the materials be transported?
• What kind of turn-around time does the vendor
  give for recovery onsite? Offsite?

36
Before you go back in

• Who declares whether the building is safe to
  re-enter?
• Fire Marshal
• Building Inspector
• Facilities Management
• Some other authority?
• Where are your emergency triage supplies?
• What will you provide?
• What will the vendor provide? Where will the
  vendor set up base camp?

37
Before you go back in(contd)

• Will you need assistance to secure the structure
  itself?
• Water removal
• Temporary Power
• Temporary building repairs
• Securing Walls
• Roof Repairs
• What is your plan if your staff balk at going
  back in?

38
One solution

• Set up scenarios and make pricing those responses
  a part of the evaluation process
• Pick 3 events of varying significance
• Mix up the media to be recovered

39
Orange County FL Sample

• Full RFP and evaluation grading sheets being
  lifted to NACRC document library
• Understand that we decided our biggest problem
  was going to be water damage
• We dont get snow or many tornadoes
• Substitute your own boiler plate front end

40
Scope of Work

• Location for services (Orange County FL mostly
  mid-Orange County)
• Volume of work undetermined (who knows whats
  going to happen?)
• Not for computer systems
• What we asked for
• There shall be no retainer paid in order to keep
  the contract in effect. The resulting contract
  will be on an as-needed basis and used only in
  case of a disaster or emergency as so deemed by
  the County. The Contractor shall provide 24
  hours per day, 7 days per week emergency response
  service. Response by phone after the first
  notification shall be within 2 hours. The
  Contractor shall have a representative on site
  after first notification within six (6) hours
  with the necessary equipment to be transferred
  within twelve (12) hours.

41
Selection Criteria and Points

• CRITERIA
• Qualifications of Staff
• Qualifications of Firm
• Technical Approach
• M/WBE Utilization
• Location
• Fee Proposal
• TOTAL
• Welfare Recipient Hires

• WEIGHT
• 10
• 20
• 30
• 10 (minimum allowed)
• 10 (minimum allowed)
• 20
• 100
• 5 bonus points

42
Qualifications of Staff

• Include a listing of all staff to be assigned to
  provide the required services and resumes for
  each describing experience, training and
  education in the required consulting services.
• Reasoning and/or recommended changes
• Tell them to make the bios BRIEF
• You dont need or want 5 pages per person!
• Clarify the kind of training or certification
  staff has
• WHEN did they get that training or certification?
• Identify staff experience working with
  governmental entities and list those projects.
• Reasoning and/or recommended changes
• Hold the government entities and projects for the
  firms qualifications
• You just get the same information in two places

43
Qualifications of Firm 1

• Provide a description and history of the firm
  focusing on previous governmental experience.
  Provide the number of years in business showing
  proof of a minimum of 5 years in document
  recovery, the current number of full-time
  employees and the excess labor that can be
  provided in case of an emergency.
• Reasoning and/or recommended changes
• Does governmental experience actually matter?
• Five years in business seemed a logical threshold
  so that we werent getting a firm that sprung up
  in response to our 2004 hurricanes
• Full-time employees seemed important to evaluate
  whether the firm could respond to more than one
  disaster
• Excess labor? Probably would just hire locally
  and teach/train while on the job

44
Qualifications of Firm 2

• Identify your equipment inventory available to
  Orange County within the Southeastern United
  States.
• Identify the capacity to process in linear feet
  per hour for each piece of equipment at each
  location.
• Identify equipment wholly owned by your firm, or
  which is available through subcontracting.
• Reasoning and/or recommended changes
• Making up the equipment list was VERY interesting
• We decided on Southeastern US as a logical
  staging area for us
• We learned you can lease/rent anything!

45
Qualifications of Firm 3

• List at least five references, with a minimum of
  two from governmental entity experience, for
  which the firm has performed similar work
  including the contact name, address, telephone
  number, email, and date of the contract.
• Reasoning and/or recommended changes
• 5 references seems about the standard
• 2 government entities seemed to make purchasing
  people happier
• Make sure to ask for email so you can readily
  contact the references
• Send the reference organizations questions and
  then call to interview them

46
Technical Approach 1

• Provide a brief description of the firms
  approach to the project. Identify the equipment
  and recovery methodologies you employ to recover
  water-damaged paper documents, microfilm, CDs,
  maps and plans, etc.
• Provide a schedule (not later than) of arrival of
  your equipment in hours after first notification.
• Identify how you ensure records are properly
  inventoried and identified as well as the chain
  of custodianship.

47
Technical Approach 2

• Identify the methods and instruments used to
  assess moisture content of records.
• Provide an operational plan for rapid deployment
  of required supervisors to provide direction to
  County employee first responders to assess
  damage and take interim steps to minimize loss.
  Include number of people, and number of hours for
  arrival.
• Should no power be available, contractor shall be
  responsible for providing all power required to
  operate equipment.

48
Technical Approach 3

• Provide a plan for how your clients are
  prioritized in the event of a widespread
  emergency in which there is multiple-client
  demand.
• Identify your permanent dehumidifying equipment
  locations in the Southeastern United States and
  the capacity in pounds of water extracted per
  hour for each piece of equipment at each
  location.
• Present options available to provide
  dehumidifying of buildings and mold and mildew
  treatment.

49
Technical Approach 4

• Submit your warranty or guarantee of your
  services.
• Confirm the firms agreement to meet the minimum
  requirements of this Request for Proposal.
• Proposers may offer alternative solutions/options
  to achieve successful completion of the scope of
  work herein.

50
Technical Approach Notes

• This area provided us with the greatest
  flexibility in evaluating proposals and the
  largest points to award/withhold.
• We dont know if we got this area right as we
  havent had to use the contract.
• But we did get it in writing in advance from the
  firm with which we contracted.
• We drew upon notes from prior NACRC and disaster
  recovery sessions. We tried to imagine many/most
  scenarios.

51
Fee Schedule 1

• Submit a rate schedule for equipment, supplies,
  and staff services.
• Travel, lodging, and meals shall be at rates
  allowed to public employees under state law per
  state statute 112.061
• Identify the cost per cubic foot, separately, for
  each of the elements as listed in item below.
  Provide a cost for all recovery services.
• freeze conventionally
• blast freeze
• freeze dry
• rapid freeze
• treat for mold and mildew
• packing and unpacking
• treatment for soot

52
Fee Schedule 2

• Identify mobilization and demobilization rates
  for equipment and staff.
• Identify your markup for any equipment rented,
  and supplies purchased.
• Identify freight cost per mile by type of
  equipment.
• Provide the cost to implement the recovery
  process.
• Provide the cost of the Recovery Service.
• Describe, if offered, your pre-registration
  service and cost associated for such service.

53
Fee Schedule 3

• In order to adequately evaluate and compare
  services provided by proposers, three (3)
  hypothetical scenarios have been created for
  which proposers must provide a written response
  using prices submitted on Attachment A, the scope
  of work provided, and the technical approach of
  your firm. These scenarios are located in
  Attachment B.

54
Scenario 1 Roof Leak

• It has just been discovered that the roof of the
  Countys Records Center has been leaking for some
  time now. It is estimated from the buckling
  ceiling tiles that this has occurred over the
  past six (6) months.
• When the actual damage was assessed, 200 boxes
  (dimensions 10x12x15 1.2 cubic feet)
  containing paper documents and microfilm had been
  severely damaged by water, resulting in mold and
  mildew.
• Contents of the boxes are as follows 7/8 copier
  and bond paper and file folders, and 1/8
  microfilm.
• Based on the limited information in this
  scenario, the scope of work, and the services
  listed in Attachment A, the proposer is asked to
  provide estimated costs for complete recovery and
  restoration of the 240 cubic feet of documents.
  It is understood that the type or method of
  restoration will vary depending upon the type of
  records involved.

55
Scenario 2 Water Pipe Burst

• A water pipe has just burst in a County building,
  which houses permits and application forms of the
  County. Standing water, approximately 8 deep
  has seeped into the records (i.e., CDs,
  microfilm, paper, maps, and plans, etc.). The
  damaged records consist of 1/2 copier and bond
  paper and file folders, 1/8 CDs, 1/8 microfilm,
  and 1/4 building maps and plans.
• The room size for this County office is
  approximately 20,000 sq. ft. Housed in this
  office space are records contained in 10
  four-drawer vertical letter-size filing cabinets
  (_at_ 6.0 cubic feet per unit for approximately 60.0
  cubic feet total) 10 five-drawer lateral,
  letter-size filing cabinets (_at_ 13.0 cubic feet
  per unit) for approximately 130.0 cubic feet of
  records.

56
Scenario 2 (continued)

• This space also contains approximately 50
  workstation cubicles at 100 sq. ft. each, which
  is equivalent to an estimated 4 to 5 cubic feet
  of records housed at each workstation. The 50
  workstations contain approximately 250 cubic feet
  of water-damaged records.
• Based on the limited information in this
  scenario, scope of work, and services listed in
  Attachment A, the proposer is asked to provide
  estimated costs for complete recovery and
  restoration of the records. It is understood
  that the type or method of restoration will vary
  depending upon the type of records involved.

57
Scenario 3 Small Fire

• A coffee pot with a potential electrical shortage
  problem was left plugged into an outlet too long
  and sparked a small fire. Smoke from the fire
  triggered the sprinkler system, which caused
  water damage to some of the records.
• When the actual damage was assessed, 300 boxes
  (dimensions 10x12x15 1.2 cubic feet)
  containing paper documents, microfilm, CDs, and
  maps had been moderately damaged by both water
  from the sprinklers and soot from the fire. The
  damaged records consist of ¾ copier and bond
  paper and file folders, 1/8 CDs, and 1/8
  microfilm.
• Based on the limited information in this
  scenario, scope of work, and services listed in
  Attachment A, the proposer is asked to provide
  estimated costs for complete recovery and
  restoration of the records. It is understood
  that the type or method of restoration will vary
  depending upon the type of records involved.

58
Fee Schedule Notes

• Having the scenarios was very important
• We had trouble figuring out how to evaluate all
  the standard information the proposers
  submitted
• The scenarios gave us a window into comparing
  apples to apples side-by-side

59
Checking References

• You really have to do this part
• We designed a series of 10 questions (lifted to
  the NACRC document library) which we sent to the
  reference people in advance of our call to them
• We had at least 2 people and usually 3 people on
  each call and then we compared notes

60
Doing the Evaluation

• We created spreadsheets and allocated points
  among the criteria for each category (lifted to
  the NACRC Document Library)
• Each of the evaluators completed their own
  spreadsheet separately
• We then met together and compared scores we were
  very close to each other
• What one person couldnt find or figure out,
  another person did

61
The Contract

• We selected our finalist just as Hurricanes
  Katrina and Rita plowed in
• Getting our contract fully executed dragged out
  for 3 months and we only needed their properly
  executed insurance form!
• Since the contract was finally and fully signed
  in January 2006, we have not had to use the
  contract
• But were sure sleeping better!

62
Additional Thoughts

• Start now dont wait for your season
• Anticipate delays along the way
• From first concept to actual completion was 2
  years for us
• Writing the scenarios and considering all the
  possibilities requires multiple brains and
  various styles

63
Lessons Learned from Katrina

• Having a pre-need contract crosses the Ts and
  dots the Is in advance.
• Remember to look at the whole process
• New Orleans Parish experience with their vendor,
  The work that they did was excellent but the
  collection department was awful. They relied on
  a technical line in the work authorization to
  keep our books in Chicago until they were paid
  eliminating the opportunity for an audit of their
  bill. I suggest that any pre arrangements
  include reviewing the billing and collection
  methods.

64
Use your NACRC Contacts

• Use NACRC as a resource to find vendors
• Talk to other jurisdictions about their
  experiences
• Conferences
• Listserv
• Email
• Conference Calls

65
Questions Answers

• What experiences have you had? Good or bad?
• When will you have a pre-need contract?
• How will you develop your potential proposer list
  to which to send the RFP?
• Can you join with other organizations in your
  area?
• Can you tag along to an existing contract
  someone else has?

66
Additional Resources

• NACRC www.nacrc.org
• Florida DOS State Library Archives
  http//dlis.dos.state.fl.us/disasterrecovery/HVAC
  
• National Parks Service Wet Collections Recovery
  http//www.nps.gov/hfc/products/cons/wet-recovery.
  htm

67
Thanks and for any follow-up

• Carol Foglesong, Assistant Comptroller, Orange
  County Comptrollers Office, PO Box 38, Orlando,
  FL 32802 407.836.5982 carol.foglesong_at_occompt.co
  m
• Paul Ketz, Broward County Records Division, 115
  S. Andrews Ave., Room 120, Ft. Lauderdale, FL
  33301 954.357.7272 pketz_at_broward.org