FIMS Data Validation Process

1
FIMS Data Validation Process

• FIMS Workshop
• Wyndham Phoenix
• May 23-25, 2006

2
FIMS Data Validation Process

• New initiative is underway to roll-out a standard
  FIMS data validation process
• Initial validation efforts to be focused on the
  23 FRPC data elements.
• Process based on the LMI validation model
  developed for DOE in September 2004.

3
Why Validate?

• Need for a statistically valid, DOE-wide quality
  assurance program for data verification
• Ensure data accuracy and a level of confidence
• Accurate real property inventory is a key
  requirement of the Presidents Management Agenda
  and is a DOE Scorecard item.
• Need to ensure accuracy of all data being
  reported to the Federal Real Property Profile.
• Generate recommendations for continuous data
  improvement
• FIMS is used on a daily basis to make real
  property management decisions

4
FIMS Data Elements to be Validated

• Property Type
• Property Usage
• Owned/Ingrant
• Status
• Historical Designation
• Using Organization
• Gross Sqft
• RPV
• Deferred Maintenance
• Actual Maintenance
• Status Utilization

• Net Occupiable Sqft
• Mission Dependency
• Annual Operating Cost
• Main Location
• City
• State
• County
• Congressional District
• Zip Code
• Restrictions

5
Process Outline

• Assemble the Validation Team
• Develop a Schedule
• Determine the Sample Size
• Randomly Select Buildings for Validation
• Identify Outliers to be Validated
• Conduct the In-brief Meeting
• Execute the Validation Process
• Develop the Scorecard
• Conduct the Out-brief Meeting

6
Assembling the Validation Team

• Size of validation team will vary by site, but at
  a minimum should include
• FIMS Administrator
• Individual(s) who is familiar with the facilities
  to be validated and the source documentation
• DOE Site Office representative
• Third party, not associated with the site being
  validated
• Others may be rotated in and out of the team as
  subject or facility experts

7
Develop the Validation Schedule

• Several weeks prior to the validation
• Distribute the validation templates
• Schedule times for review and facility visits
  that are convenient and efficient
• Send site visit agenda, schedule, and discuss
  required data/source documents with participants
• Identify date that FIMS data will be extracted
  for use in the validation
• Within two days of the validation
• Extract FIMS data based on previously disclosed
  date

8
Determine Sample Size
9
Determine Sample Size

• Randomly selected buildings that will be
  validated based on the number of facilities at a
  particular site and the confidence level you hope
  to achieve
• To achieve a 90 confidence level in the accuracy
  of a particular data element, the following table
  will assist you in determining the random sample
  size

10
Sample Size For 90 Confidence Level
11
Generate Random Sample Set
12
Random Building Generator
13
Random Building Generator (contd)
14
Random Building Generator Output
15
Selecting Outliers
16
Outliers

• Outliers are typically facilities that have high
  RPV, Gross Sqft, or Deferred Maintenance cost
  that were not selected as part of the random
  selection of buildings but should be included in
  the validation process.
• Reinforces the level of confidence and accuracy
  of the data

17
Outlier Selection
18
Outlier Selection (contd)
19
Outlier Report Output
20
Outliers to be Validated
21
Outliers to be Validated (contd)
22
Outliers to be Validated (contd)
23
In-Brief Meeting

• Introduction of team members
• Establish Logistics
• Identify persons and systems required for access
  at planned times
• Identify location where the validation will occur
• Review validation schedule
• Set goals and discussion protocols
• Share facilities validation list
• Discuss Outliers to be validated
• Select facilities for walk-throughs
• Identify any initial issues

24
Beginning the Validation Process
25
Source Documents

• Source documents contain detailed comprehensive
  site data that should be the accurate data of
  record.
• Can be a hard or soft document
• Information contained is authoritative, has an
  owner and is used in the sites decision-making
  process

26
Source Documentation

• FIMS values must be validated against independent
  source documents. Acceptable source documents
  are
• Signed construction drawings and blueprints
• As-built drawings
• Condition Assessment Reports
• Work order requests and work orders
• Databases maintained on site that are managed and
  used for daily decision making by the data owner
  (CAIS, Maximo)
• Physical Measurements
• Internal memorandums/documented processes
• Official documented reports

27
Source Documentation (contd)

• Questionable Source Documents
• Databases that are repositories of information
• Spreadsheets of compiled data
• Undated printout of information
• Interim or draft reports
• Design development drawings

28
Source Documentation (contd)

• Site / Program Databases
• Site or program specific databases may or may not
  be credible sources
• Site or program specific databases are only
  acceptable if they are a part of the management
  decision making process
• If these sources are used, they should be clearly
  identified

29
Validation Form
30
Validation Form (contd)

• Identification Information
• HQ Program Office, Site, Area, Prop ID, Property
  Name
• Data Element
• Corresponds to the FIMS data element to be
  validated
• FIMS Value
• Contains the FIMS data value that was extracted
  prior to the start of the validation process

31
Validation Form (contd)

• Source
• Most recent source document for the data element
• Can also be a site database
• Year of Source
• Year source was created or last updated
• Automatically creates a variance if
• Older than 1 year for Status Utilization,
  Deferred Maintenance, or RPV
• Older than 5 years for condition assessment
  survey
• Value from Source
• Value from source document that corresponds to
  the FIMS data element being validated

32
Validation Form (contd)

• Variance (FIMS-Source)
• A percentage that is calculated based on errors
  for all numeric fields
• Numeric values are only considered in error when
  the material variance is greater than 10
• Example FIMS Value Source Value / Source Value
• (28,000 gsft 25,000)/25,000 gsft 12 material
  variance
• Material Variance (Y/N)
• A value of Y if the numeric data element has a
  material variance greater than 10 or other
  non-numeric data elements were found to be in
  error.

33
Validation Form (contd)

• Most likely cause of variance
• When a material variance occurs, validation team
  should identify the likely cause
• Out of date, inaccurate, or incomplete sources
• Sources probably exist but cant be located
• No source documentation
• Multiple or conflicting sources
• Incorrectly calculated value
• Incorrectly entered value
• Additional Comments
• Space provided to document other observations or
  issues
• Further explanation of data sources and variances

34
Evaluation and Reporting Variances
35
Evaluation and Reporting Variances (contd)

• Total number of buildings with material variance
• Basically identifies the number of errors found
  in the validation of a particular data element
• Total number of buildings sampled
• Total Frequency of Material Variance (errors) at
  a 90 confidence level
• Indicates a percentage of buildings that will
  have a material variance
• Calculation No of buildings with material
  variance/ Total no of buildings sampled
• Example 4 buildings with error in gsft /25
  sampled buildings 16 Frequency of Material
  Variance

36
Evaluation and Reporting Variances (contd)
37
Walk-Throughs

• Although the validation is largely a paper
  validation, it is critical for the validation
  team to physically see some of the assets to
  verify utilization, usage, and condition
• This is not a condition assessment survey
• This process relies mainly on observation and
  communication with building manager or other site
  personnel
• Beneficial to generate the FIMS complete
  information report for all walk-through facilities

38
Development of a Scorecard

• A Scorecard is developed at the conclusion of the
  validation process. This is preferred over a
  lengthy written report
• Can be largely completed at the site
• Saves time and energy
• Focuses attention on the facts and findings
• Compact and conveys results quickly
• Aligns with OMB and Departmental scoring

39
Scorecard (contd)

• Provides an overall rating of the FIMS data and a
  rating by data element
• Overall rating includes a score for status and
  progress
• Status A weighted roll-up of the findings per
  data element
• Progress Subjective recognition of improvement

40
Scorecard (contd)

• A Scorecard is designed like a stop light chart
• Green Continue to do great things
• Yellow Proceed with caution
• Some defects noted but not deemed to be affecting
  overall FIMS quality
• There is an opportunity for improvement or at
  least monitoring
• Red Stop and address a problem
• Defects noted affected data quality
• Corrective Action Required

41
Scorecard Sample
42
Scorecard Rating Data Element

• At the data element level, ratings are assessed
  based on error rate
• Red - gt 10 or greater
• Yellow - Between 6 and less than 10
• Green - 5 or less

43
Scorecard Rating Overall Status

• A site will receive an overall status of red if
  one or more of the following critical data
  elements is in red status
• RPV
• Deferred Maintenance
• Actual Maintenance
• Status Utilization
• Mission Dependency
• Annual Operating Cost
• Or two of the remaining data elements are red in
  status

44
Scorecard Rating Overall Status

• An overall status rating of yellow is received
  if
• Any non-critical data elements are red
• Any critical data element is yellow
• RPV
• Deferred Maintenance
• Actual Maintenance
• Status Utilization
• Mission Dependency
• Annual Operating Cost

45
Scorecard Rating Overall Status

• To obtain a green status rating
• No red data elements
• All critical data elements are green
• RPV
• Deferred Maintenance
• Actual Maintenance
• Status Utilization
• Mission Dependency
• Annual Operating Cost

46
Scorecard Rating Progress

• A progress rating is important because the status
  rating may not tell the entire story
• Progress rating is more subjective and considers
• Statistical improvement since last validation
• Establishment/meeting the goals of a corrective
  action plan
• The progress rating is a leading indicator and a
  way to indicate positive or negative momentum
  that will yield status improvement or decline

47
Scorecard Rating Progress

• Red indication that red status will not improve
  or yellow status will decline.
• Lack of attention and urgency
• Corrective action plan not being met
• Yellow Indication that limited, positive
  efforts are being taken that may improve the data
• Corrective action plan is in place but progress
  is slipping
• Green results-oriented plan in place with goals
  and necessary resources
• Clear level of attention and demonstrated sense
  of urgency
• Typically, a site that is green in status by
  default is green in progress

48
Out-brief Meeting

• Out-brief to Site Manager and Staff
• Present a Scorecard resulting from the validation
• Resolve any outstanding issues. If not resolved,
  assign action items and completion dates.
• Share recommendations for improvement and note
  good and best practices
• Disagreements
• Capture and document disagreements with
  validation findings and recommendations for
  results

49
Measure of a Successful Validation

• Review/validate the randomly selected FIMS
  building records along with the selected outliers
• Generate either agreement (or documented
  disagreement) why a specific data element in FIMS
  is inaccurate
• Walk through a number of random buildings (four
  or five) and examine condition, use, and
  utilization
• Understand how FIMS data is being maintained and
  managed
• Document other notable facts and data
• Determine significant findings, highlight
  successes, prepare a scorecard and generate
  recommendations to address shortfalls and areas
  for improvement

50
Results of a Good Validation
51
Results of a Bad Validation
52
Retention of Information

• What should be kept
• FIMS random buildings set
• Reference to source documents used
• Completed validation templates
• Validation Scorecard
• Information should be retained for no less than
  five years for the purposes of tracking and
  trending.

53
Data Validation Tips for Success

• Assemble the proper validation team
• Includes FIMS Administrator and subject matter
  experts as well as DOE representation
• In-brief Out-brief meetings
• Having good source documentation
• Documents containing detailed site data that
  should be the accurate data of record
• Can be a hard or soft document
• Information contained is authoritative, has an
  owner and is used in decision-making

54
Data Validation Guidance/Training

• Guidance forthcoming from the Office of
  Engineering and Construction Management (MA-50).
  
• Guidance currently in draft form
• Planned to be signed out and distributed in FY06
• Goal is for full implementation in FY07
• Data Validation Training
• All FIMS Administrators and appropriate Site
  Management should attend
• Contact your HQ Program Office FIMS
  representative to express interest in attending

55
Questions/Discussion