Evaluation of Enterprise CSTARS Pilot

1
Evaluation of Enterprise CSTARS Pilot
Commerce STandard Acquisition and Reporting
System
Briefing for the Commerce IT Review Board, Chief
Financial Officers Council and the CSTARS
Enterprise Team Final Report July 26, 2001
2
Overview

• CSTARS Background
• Assessment Summary
• Key Issues (Security, Interfaces, Risk)
• Recommendations
• Risk Mitigation
• The Road Ahead

3
What Is CSTARS?

• Enterprise-wide e-Initiative
• supports e-Commerce, e-Business, and
  e-Government policies and pilot programs
• Automated Procurement System
• provides a robust procurement/acquisition
  functionality, powered by a Competitively Awarded
  COTS Solution (Comprizon.Buy )
• eliminates duplication with CAMS/CFS
• Promotes Standard Business Practices for the
  Acquisition Workforce and Their Customers

4
CITRB - Overarching Goals

• Streamline operations
• Reduce cost by solving systems and business
  process problems at the enterprise level
• use COTS solutions wherever practicable
• Reduce the duplication of system functionality
• Support e-Government

5
CITRB - Direction for the DOC Procurement
Executive

• Acquire a Commercial-Off-The-Shelf (COTS)
  solution for DOC-wide procurement automation
  needs
• Charter an Automated Procurement System
  Enterprise Team (APSET) to establish the
  foundation and strategic direction for
  implementation
• Pilot the proposed solution at OS and NIST under
  the direction of the APSET

6
CITRB - Conditions for COTS (CSTARS)

• No customization
• Implementation will not interfere with the CAMS
  core financial system (CFS) implementations at
  NOAA and Census
• All operating units will use CSTARS , regardless
  of financial system. This will be coordinated
  with the Enterprise Team
• Present to the CITRB results from successful
  deployment of the NIST and OS pilot
• A single instance of CSTARS will be installed on
  centralized servers with multiple databases

7
Enterprise Team Goals

• Provide management data for decision-making
• real-time current information gathering tools
• Solve problem in procurement operations
• immediate improvements from legacy systems
• Accommodate future technology trends
• flexibility, EC and web-based functionality
• Standard Enterprise Practices
• streamlined, use reforms, market-place driven
• Selection of COTS product
• Comprizon.Buy CSTARS

8
Champions and Partners for Change

• Government
• CITRB (also CIOs Office of Computer Services)
• CFO Community
• Procurement Community
• CAMS Executive Board
• CSTARS Enterprise Team (Bureau Heads of
  Contracting Offices and stakeholders)
• Federal Comprizon.Buy Users Group
• Commercial
• CACI, Federal Inc.
• Booz-Allen Hamilton
• I. M. Systems Group

9
Enterprise Model for CSTARS

• Re-order and reduce scope of implementation
• Limit functions to internal procurement needs
• OS implementation as of October 1, 2001
• NIST implementation as of November 1, 2001
• interfaced with the NIST financial system
• Centralized database architecture - CIOs Office
  of Computer Services (OCS)
• Evaluate, Assess, Report Findings and
  Recommendations

10
Commerce Dept. demonstrates sound business
practices for progress.
Management Approach
Towards an Enterprise-Wide Procurement System
1998 1999 2000 2001 2002
OMB Circular A-11 ITRMA FARA
FASA
Business Case
Enterprise Pilot
Continuous Improvement
Pilot Evaluation
Procurement Automation Team
Program Management Plan
Migration to e-commerce e-marketplace
6 Bureaus with procurement authority 9 Different
procurement systems
? Standard Business Practices ? Lower Purchasing
Costs ? Leverage Technology Investments
11
Pilot Evaluation - schedule and scope
Detailed Evaluation Plan
Finalize Report Briefings
Prepare report
Perform Analysis
Conduct Initial Assessment
Conduct Evaluations
March 01 April 01
May 01 June
01
Weekly Status / Progress Briefings
Project Start Feb 23

• 1. Conduct Initial Assessment
• Reviewed Program Management Plan (PMP), dated
  Dec 10, 99.
• Updated Evaluation Criteria (Appendix 6 to PMP).
• Identified/collected existing documentation.
• 2. Conduct Evaluations
• Survey technical architecture. Review
  documents, records, logs.
• Interview OS and NIST users.
• Collect performance metrics.
• 3. Perform Analysis
• Enterprise Standardization.
• System Setup.
• System Performance.
• 4. Identify Impact
• CSTARS Program Objectives.
• Department Goals.

12
Evaluation Criteria - from Appendix 6 of
Program Mgmt Plan
Revised Performance Measures Confirm existence of
data dictionary. Number of data
elements. Confirm data dictionary contains data
formats. Confirm capability to handle
non-standard formats. Confirm existence of
process documentation.Identify number.Confirm
capability for Bureaus to review
processes. Number of data elements
identified Confirm existence of ICDs (no change)
NIST OS (no change) NIST
OS (no change) NIST OS (no
change) NIST OS Number of vendors
loaded into database. NIST
OS Number of contracts migrated.
NIST OS Number of routing paths
created/loaded NIST OS Load
test standards exist? Expected transaction
response times. of successful data
transmissions (no change) (no change) (no
change) (Define transactions) (no change)
NIST OS Capability exists to track
processing time? Number of days. Capability
exists to track cost? Cost to accomplish specific
actions. Provides a baseline measurement for
further evaluations.
13
Evaluation Criteria - Progress Summary
DATA SUMMARY NIST OS NOAA Processes
documented 136 124 Data elements documented
285 216 Users with profiles 130 42
88 Vendors in database 3400 175
21,000 Contracts migrated 800 113 Users
trained 130 22 Completed transactions 3000 345
Mapping Sessions Completed
Mapping Sessions Completed
Begin August 01
Begin October 01
Begin October 01
Data represents approximate values, and is
intended to be used as baselines to indicate
progress over time. Completed Transactions
include all new contracts/purchase orders,
delivery orders, and modifications using CSTARS.
This does not include in-progress requisitions
and solicitations or cancellations.
14
Assessment Summary

• Standardization
• CSTARS Program is progressing towards commonality
  of processes, formats, and data.
• Set Up
• Implementation (data initialization, process
  migration, interface control) is more difficult
  than originally planned.
• Production
• Process is taking longer. Can be attributed to
  learning curve issues, process trade-offs,
  quality gains, and change management challenges.

Bottom Line We found no technical show
stoppers that indicate any reason to
discontinue CSTARS implementation throughout the
Department.
15
Major Lessons Learned

• Implementation
• More difficult, more resource intensive than
  originally planned.
• Better to have a bureau person become CSTARS
  expert than to have a CACI person become a bureau
  expert.
• Training
• Suspected system shortcomings are usually traced
  to user lack of understanding.
• Bureau dedicate resources as a core group.
• Desk side training is critical 6-month burn-in
  is to be expected.
• Reporting
• Current Ad hoc reporting capabilities do not
  satisfy user requirements.
• DoC needs to define reporting standards.
• Interfaces
• Currently - lots of re-keying going on.
• The tall pole - financial system interface.
  Process integration needed.
• Management commitment
• Top-level commitment of CIO, CPO, CFO is
  critical.
• Support to Bureaus
• Written agreement from OAM - who pays for what
  - will facilitate bureau planning.
• Credible, reliable SLA with OCS is critical for
  bureau-level confidence.
• Planning factors are needed to develop
  cost-loaded implementation plans.

16
Transaction cost estimates
CSTARS supported at OCS. Based on Oracle CPU
usage.
Estimates can be used as planning factors.
Example For an organization during early
months of CSTARS use
Based on current cost per CPU.
(Data from controlled CSTARS runs conducted
through OCS, May 2001)
17
Other Observations

• Security
• CFS-CSTARS Interface
• Risks

18
Security
19
Security - assessment in progress

• System Security Plan in development
• Process-oriented approach, based on data
  sensitivity
• CSTARS has capability for access segregation,
  based on roles

Procurement Process
User Groups/ Roles
20
Security - observations
Observations - No overall security policy
between organizations. - Each box requires
separate username and password for access. - Goal
is to facilitate use of the system, and Prevent
unauthorized access to C systems/data Restrict
authorized users to appropriate systems/data.
References NIST Special Publication 800-18
Guide for Developing Security Plans for IT
Systems NIST Special Publication 800-XX Self
Assessment Guide for IT Systems NSA Infosec
Assessment Methodology
21
CFS CSTARS - Planning Horizons
FY 01 FY 02
FY 03 FY 04
Jan
Oct Jan
Oct Jan
Oct Jan
Oct
Comprizon.Web Initial Release
CSTARS-CFS Interface Release
CSTARS Pilot
CSTARS Implementation at NOAA, NIST, OS
Transition to CSTARS - WEB
CSTARS -CFS Interface Development
CFS at NIST Live
CFS/CAMS Continued Development Deployment
CAMS in production at Census EDA as of Oct 1,
2000
CAMS in production at the NIST Crossed Serviced
Bureaus as of July 1, 2001
CAMS in production at NOAA as of Oct 1, 2002
FY 03 Commitments
Bureaus finalize program funding for FY 03.
FY 04 Commitments
DRAFT
22
Enterprise Application Integration
Current DoC Systems
Financial System
IPT CONOPS BUSINESS PMP PILOT WEB
CASE
DEV
FINANCIAL REQUIREMENTS
IPT CONOPS BUSINESS PMP PILOT WEB
CASE
DEV
PROCUREMENT REQUIREMENTS
Procurement System
DoC demonstrates sound business processes,
but... CAUTION BUILDING INERFACES IS NOT
ENOUGH.
Perpetuation of Stovepiped Systems
Requirements Integration
Enterprise Application Integration
YOU NEED
Process Integration
23
Risks

• 1. With any COTS solution -
• Who controls products evolution?
• Vendors respond to paying customer. All others
  must adjust.
• 2. Migration to web-based application(s)
• Todays solution tomorrows legacy (how are
  your interfaces doing?)
• What are the ancillary costs? (training, data
  migration, protocols, security)
• 3. Integration of Enterprise Systems (CFS and
  CSTARS)
• No process integration (leads to stove-piped
  requirements)
• Financial Standards not same as Procurement
  Standards
• 4. Cohesive enterprise OR waning credibility
• As landscape changes, how will Dept support
  Bureaus?
• Good strategy without resources delays,
  burnout, death march .
• Long-term top-level commitment needed for managed
  expectations.
• 5. Distributed implementation could drive
  divergence from standardization over time.

24
Putting the Pilot in Perspective
25
Recommendations
1. Continue CSTARS implementation. Leverage
NIST and OS experience to enable implementation
at other Bureaus. 2. Re-examine Centralized vs
Distributed implementation of enterprise-widee-pr
ocurement system. 3. Update the CSTARS Program
Management Plan (Dec 10, 1999) to include current
implementation guidance, schedule, and transition
to web-CSTARS. 4. Establish standards and for
centralized reporting requirements. 5. Restore
contractor support to supplement OAM program
management - Revise 10-Year Life Cycle Cost
Estimate - Leverage best practices, lessons
learned - Facilitate implementation, migration
to web-CSTARS 6. Provide for on-going strategy
assessment, to include impact of technologies
such as web requisitioning, e-catalog,
e-marketplace.
26
The Road Ahead ...
27
Major CSTARS Events
28
Cost - Business Case versus Actual/Projected

• CITRB Approval from Business Case
• Actual (FY99-00) and Projected (FY01-05)

Data from Business Case, January 19, 1999
Data based on OAM current WCF budget analysis.
29
Risk Mitigation

• Evaluate implementation process
• Task order to evaluate and assess security
• Contract for program management assistance
• Insure enterprise integration with CAMS
• Look for e-Opportunities for e-Commerce
• Initiate Data Warehouse implementation
• Revive the CSTARS Enterprise Team
• Participate in the Comprizon.Buy Federal Users
  Group

30
Next Steps ...
? What concerns do you have? ? What
involvement do you want? ? When do you want the
next update?