Quality Assurance QA

1
Quality Assurance (QA)

• Implementation of the Quality Assurance
  Surveillance Plan
• From an Oversight Perspective

March 2009
1
2
Presentation Outline

• Some Key Roles
• Quarterly A-76 Office of Competitive Sourcing
  (OCS) Reports
• Independent Validation and Verification
• What Process is being used for surveillance?
• What are the current methods of surveillance?
• What do we do with the feedback?
• What is a Corrective Action Program?
• What is a Corrective Action Request?
• Current IM-61 generated Reports
• What is the QASP doing for the EITS?
• What the QASP outlines?
• Program Contact Information

2
3
Some Key Roles

• Quality Assurance Evaluators (QAE) and
  Contracting Officers (CO) have been designated to
  oversee and report cost and performance data as
  an outcome of the OMB Circular A-76 Post
  Competition Accountability.
• Subject Matter Experts, Contracting Officers
  Technical Monitors (COTMs) and Sr. Technical
  Advisors (STA) are appointed for each
  task/subtask and serve as first line managers.

3
4
Some Key RolesCont.

• QAEs perform actual contract surveillance and
  report to DGRs.
• Customer Feedback QAEs investigate cases of
  valid poor or non-performance complaints of
  customers and ensure corrective actions are
  taken.

4
5
Quarterly A-76 Office of Competitive Sourcing
(OCS) Reports

• ISIO performs a comparison analysis of the
  Energy Information Technology Services (EITS)
  Quarterly Performance Assessment to include the
  quality and timeliness of work produced as
  compared to the QASP standards and actual costs
  for performance period against estimated costs.
  The assessment provides an evaluation of the
  service providers compliance with the Quality
  Control Plan and Master Work Hours produced
  versus Performance Work Statement (PWS). The
  assessment also compares other workload hours
  produced compared to PWS workload. An
  explanation is provided for any differences.

5
6
Quarterly A-76 Office of Competitive Sourcing
(OCS) Reports Cont

• ISIO compares the Agency Tender to the current
  staffing. It also makes a comparison of the
  estimated and actual contract administration
  support, including positions and overhead
  staffing support required.
• ISIO prepares a comparison analysis of the EITS
  Cost Report quarterly to show Agency Tender Cost,
  adjusted budget cost, and estimated actual cost.
  It also provides an explanation for any
  difference between adjusted budget cost and
  actual cost. It documents cost savings quarterly
  for submission in the OCIO 647(b) annual report.

6
7
Quarterly A-76 Office of Competitive Sourcing
(OCS) Reports Cont.

• A quarterly running personnel comparison analysis
  is performed to show personnel loss, additions,
  and exceptions.
• Ensures the task monitor evaluation parameters
  capture and depict the EITS SPs most accurate
  performance results which are submitted to COTMs
  to be evaluated and scored.

7
8
Quarterly A-76 Office of Competitive Sourcing
(OCS) Reports Cont.

• ISIO evaluates the EITS to ensure services
  provided are in compliance with the Quality
  Control Plan. It shows the areas of the Quality
  Control Plan (QCP) that are evaluated in response
  to the quarterly reporting requirements. The
  independent assessment findings indicate whether
  the EITS stays consistent with the OCP. An
  overall rating is provided, inclusion of the
  areas evaluated would specify if corrective
  action reports would be generated.
• There is an in-house report validation of all
  information prior to delivery/submission of
  Office of Competitive Sourcing (OCS) report.

8
9
Independent Validation and Verification

• All QAEs are non-government, non EITS
  contractors with a non-vested interest in EITS
  performance results.
• QAEs have no authority to direct or allow EITS
  to deviate from contract requirements, including
  changes which
• would impact task/subtask costs or
  schedule/deliveries.
• QAEs rely on customer feedback to implement PIPs
  and CARs.

9
10
What process is being used for surveillance?

• The following areas/items are monitored as agreed
  upon
• between the Service Provider and the OCIO-
• Performance requirement
• Performance Standards
• Method of measurement
• Performance metrics
• Performance incentives

10
11
What are the current methods of surveillance?

• 100 Percent Inspection Complete inspection of
  contract requirements as they relate to critical
  and substandard performance areas.
• Periodic Inspection Designed to inspect some
  part but not all products and services being
  monitored.
• Random Sampling Used when number of occurrences
  of a service is very high (e.g., 10,000 help desk
  tickets).

11
12
What do we do with the feedback?

• Evaluate and perform comparison analysis of the
  collected data (QCP Compliance Quarterly
  Reporting, Monthly Reports, Weekly Activity
  Reports (WAR), etc.). Data extracted from the
  WAR is then measured against stipulated
  performance metrics outlined for the subtask in
  the Performance Work Statement.
• In the interim a Trend analysis is performed of
  the collected data and provided to the COTMs to
  support quarterly performance evaluations
  submitted to the Office of Competitive Sourcing.

12
13
What is a Corrective Action Program?

• Corrective Action Program should meet the
  criteria of resolving and preventing recurrences
  of identified deficiencies and areas of
  improvement. The program encompasses the
  continuous monitoring of work performance.
• The essence of a successful corrective action
  program at any level involves a combination of
  processes, people, and follow-up.

13
14
What is a Corrective Action Request?

• Corrective Action Request (CAR) is a process that
  is used to identify deficiencies in service,
  barriers or obstacles that prevent or delay
  effective, efficient, work processes and
  ultimately resolves any deficiencies.
• This CAR process is prompted by any of the
  following recommendation from an internal or
  external representative of services rendered,
  customer evaluations and analysis performed of
  data pertaining to work performance, and/or
  customer survey feedback.

14
15
What is a Corrective Action Request? Cont.

• Evidence provided as recommendation for a CAR
  should be significant.
• The generalized process for a CAR is as followed
  
• identify and report deficiency findings, evaluate
  each finding, establish a corrective action plan
  to resolve the deficiency, resolve
  deficiency/finding, close Finding.
• The ISIO QAs Corrective Action Guide is
  currently under revision to further address
  policies and procedures for an effective
  comprehensive process/program.

15
16
Current IM-61 generated reports

• Monthly Trend analysis of the Weekly Activity
  Reports and Service Level Agreement (SLA) Reports
• Comparison analysis performed of staffing, risks
  issues, milestones, metrics, financials,
  deliverables, and action items
• A-76 Office of Competitive Sourcing (OCS)
  Quarterly Reporting
• Weekly Activity Reports
• QCP Compliance Quarterly Reporting

16
17
What is the QASP doing for EITS?

• Implementation of the Quality Assurance
  Surveillance Plan from an oversight perspective
  reveals an unbiased evaluation of IT support
  services provided by the MEO. In addition to
  ensuring quality control methods are implemented
  across the scope of services, QA surveillance
  verifies and validates the EITS performance,
  staffing, and financial status against the PWS,
  Letter of Obligation (LOO), Federal Acquisition
  Requirements (FAR) and various DOE guidance's,
  policies and procedures.

17
18
What is the QASP doing for the MEO? Cont.

• QA surveillance results provides all EITS current
  and potential customers with actual IT support
  products and services performance metrics and
  validates that services rendered meet PWS
  metrics, Federal Acquisition Regulations, and DOE
  policies and procedures.

18
19
QASP Outlines the following

• EITS expectation for delivering products or
  services in accordance with the contract.
• Expectation for producing, maintaining, and
  providing for audit, quality control records and
  reports including all records associated with the
  investigation and resolution of customer
  complaints.

19
20
QASP Outlines the followingCont.

• Expectation for implementation of Quality Control
  Plan (QCP), which describes EITS methods for
  ensuring all products and services provided under
  contract meet performance standards.
• Implementation of Customer Feedback Program as
  stated in the PWS.
• Key Government personnel within ISIO in addition
  to the CO and DOE customers that are responsible
  for Quality Assurance.

20
21
CONTACT INFORMATION

• ALTHEA T. VANZEGO
• PROGRAM MANAGER
• INFORMATION STRATEGY AND INNOVATION OFFICE
• OFFICE OF THE CHIEF INFORMATION OFFICER
• althea.vanzego_at_hq.doe.gov

21