Developing Credible Cost Estimates Using CSDR Data

1
Developing Credible Cost Estimates Using CSDR
Data
2
Outline

• Background
• Benefits of CSDR data
• Cost Estimating Process
• Estimate Structure Development
• Data Collection
• Cost Model Construction
• Examples using CSDR data (Note Data used in
  examples is purely fictional and is used for
  demonstration purposes only)
• Summary

3
Background - DoD Cost Growth
In a number of programs, the Department of
Defense has experienced significant growth over
the original estimates in the development and
procurement of weapon systems. These cost
increases are detrimental to the warfighter and
the taxpayer. The result is generally later
delivery of capability and a reduction in the
quantity purchased.
Hon. John J. Young Jr. Under Secretary of Defense
for Acquisition, Technology and
Logistics July 30, 2007 Memo to the Services
Defense Agencies.
4
Background Importance of CSDRs to Cost
Estimation?

• GAO identifies the availability of valid data as
  a basic characteristic of a credible cost
  estimate
• Numerous sources of suitable, relevant, and
  available data should be used
• Relevant, historical data should be used from
  similar systems to project costs of new systems.
  The historical data should be directly related to
  the system performance characteristics
• Cost and Software Data Reports (CSDRs) constitute
  relevant historical data

Deriving quality cost estimates depends on the
quality of historical databases. In most cases,
the better the data are, the better the resulting
estimate will be. Source GAO Cost
Assessment Guide, Best Practices for Estimating
and Managing Program Costs (July 2007)
5
What are the benefits of CSDR data?

• Structure of actuals is consistent with structure
  of estimates
• Product oriented vs. Process oriented
• Separate identification of recurring and
  nonrecurring HW costs, which in turn facilitates
  estimation of
• Design cost
• Prototype vs. Production hardware cost
• Separate identification of functional category
  costs (e.g. Engineering, Manufacturing
  Operations, Materials)
• CSDR data can likewise be tremendous benefit to
  future estimation of two high cost risk elements
• Software
• Platform integration
• The use of CSDR data as the basis for cost
  estimates yield
• Credible cost estimates
• Realistic budgets
• Executable program

6
Cost Estimating Process

• Ground Rules and Assumptions Establishment
• Estimate Structure Development
• Cost Element Structure (CES)
• Work Breakdown Structure (WBS)
• Data Collection
• Cost Estimate Construction
• Total System Estimate Test
• Documentation Preparation

CSDRs play a critical role in these process
steps
7
Estimate Structure Development

• Cost Element Structure
• The CES groups costs into system-specific and
  appropriation-discrete cost elements
• The Army CES is a standard structure that breaks
  costs into 6 phases
• 1.0 RDTE and 2.0 Procurement are covered by CSDR
  reporting (see next slide)
• 3.0 MILCON, 4.0 MILPER, 5.0 OM, and 6.0 AWCF are
  generally not covered by CSDR reporting
• Work Breakdown Structure
• Cornerstone of every program because it defines
  in detail the work necessary to accomplish a
  programs objective.
• Start with MIL-HDBK-881A (Standard structure to
  level 3)
• Tailor and expand based on scope of work expected
• WBS and CES are complementary standard structures
  
• Both used in the development of a cost estimate
• Key CES elements are broken out further using the
  WBS
• It is critical that the CSDR reporting structure
  provide data that can easily support the
  estimating structure

8
Estimate Structure Development Army CES
9
Estimate Structure Development - WBS

• Cornerstone of every program because it defines
  in detail the work necessary to accomplish a
  programs objective.
• Start with MIL-HDBK-881A
• Tailor and expand based on scope of work expected

10
Data Collection

• Data collection is driven in part by the WBS, the
  prospective cost modeling techniques, and an
  (economic and engineering) understanding of cost
  behavior
• Nonrecurring and recurring efforts behave
  differently and should be modeled differently
• E.g., recurring effort and costs vary with
  quantity, while nonrecurring effort and costs are
  relatively inelastic wrt quantity (within
  relevant ranges)
• Different functional activities have different
  types of resource requirements and end products.
  Functional effort should be modeled discretely.
• E.g., Development Engineering prepares designs,
  whereas Manufacturing produces hardware end
  products
• Larger data sets are better the more similar
  (homogeneous) the data in a sample the better
• Actual data is much better than estimated data
• Data review
• How does data compare with prior cost reports?
• How does data compare with other historical
  experience?
• Review and follow-up with contractor areas that
  have been classified incorrectly in the past
• Data normalization for
• Escalation, quantity, learning, production rate,
  other manufacturing, contractual, economic
  considerations.
• Sometimes requires supplemental information
  obtained from site visits to better understand
  context

Analysis underpinned by large, homogeneous data
samples of actuals will produce superior
results when compared to small or heterogeneous
samples or estimated data
11
Cost Estimating Methods

• Engineering
• Examines work segments in detail
• Requires extensive knowledge of system,
  associated processes
• Most difficult method to apply
• Expert opinion
• Delphi questionnaire
• Cost knowledge bases
• Parametric models
• Relates cost to physical attributes, e.g.,
  height, horsepower referred to as Cost
  Estimating Relationships
• Relies on careful analysis of actual contractor
  costs
• Analogy
• Direct comparison with historical data of
  similar, existing systems
• Requires judgment, considerable experience
• Relies on careful analysis of actual contractor
  costs

Can be supported by CSDR data
12
Estimating Army Cost Elements

• The comparative magnitude of costs should be a
  key consideration when selecting cost estimating
  methodologies
• Cost drivers should garner the cost analysts
  time and attention
• Larger databases (if possible)
• Thoughtful economic and engineering based
  methodologies
• Lower cost elements can be adequately addressed
  via cost factors
• WBS element Data is typically low cost
• Other support elements tend to decrease in
  relative magnitude with on-going production lots,
  often following a decreasing percentage function
• Training
• Peculiar Support Equipment
• Common Support Equipment
• DD Form 1921 Cost Data Summary Reports from
  historical completed contracts can be used to
  quickly derive support element cost factors of
  prime mission product (PMP)

13
Example 1 Recurring Production (Army CES 2.02)

• Includes all production hours including
  scrap/rework to build end item
• Application Estimate production cost of future
  lots based on early production lot data
• Analytical approach
• Regress 1921-1 Part 2 data and analyze results
• Compare results with slopes/T1, T100s with
  similar 1921-1 Part 2 data
• Adjust and recalculate as appropriate

Source OSD CAIG Briefing Use of the Contractor
and Software Data Reporting (CSDR) System An
Analysts Perspective Sept 2006
14
Regressed Recurring Manufacturing
T1 25 hours/lb S 69.1
Regression results over first 5 lots -- recurring
manufacturing hours/lb versus quantity
Source OSD CAIG Briefing Use of the Contractor
and Software Data Reporting (CSDR) System An
Analysts Perspective Sept 2006
15
Analysis of Recurring Manufacturing
Highest Historical Hrs/lb and Slope 85
Projected Hrs/lb and Slope Using 1st 5 Lots of
Data
Lowest Historical Hrs/lb and Slope 72
Source OSD CAIG Briefing Use of the Contractor
and Software Data Reporting (CSDR) System An
Analysts Perspective Sept 2006
16
Example 2 Recurring Production (Army CES 2.02)

• Includes all production hours including
  scrap/rework to build end item
• Application Estimate production costs based on
  experience in prototype manufacturing
• Analytical approach
• Regress 1921-1 Part 2 data and analyze results
  for System X. Determine slope and first
  prototype unit cost for System X..
• Relate cost of first prototype unit and first
  production unit for ground combat vehicles in
  general. Develop factor.
• Apply factor to System Xs first prototype unit
  cost to determine System Xs first production
  unit cost.
• Estimate costs of succeeding lots.

Source OSD CAIG Briefing Use of the Contractor
and Software Data Reporting (CSDR) System An
Analysts Perspective Sept 2006
17
Example 3 Support Element (Data) Army CES 2.08

• Production Phase
• Estimating support element cost for WBS Data
• Using DD Form 1921 Cost Data Summary Report
• Develop Factor against Prime Vehicle

18
1a. PROGRAM Generic Surface Vehicle X
Surface Vehicle System Primary Vehicle (PV)
Hull/Frame
TOTAL K 441,776.3 133,973.6 16,508.1
30,114.1 4998.3 1344.0 576.0
19
Estimating Cost Element Data 2.08

• Consistently derived support element cost factors
  of prime mission product (PMP), from similar
  programs, can be grouped and averaged
• Applied to calculate support cost estimates

Data costs as a factor of Primary Vehicle
Costs Six Production Phase Tactical Vehicle
Programs Program 1 3.7 2 2.8 3 2.2 4 5.
5 5 4.7 6 1.5 3.4 average 1.5
standard deviation
Generic Surface Vehicle Z 44.20 Primary Vehicle
Cost (M) 3.4 Average Cost Factor for Data
Element 1.55 Data Cost Estimate (M)
20
Summary

• We actually use this information
• CSDR data directly supports the development of
  program life-cycle cost estimates
• However, we cant use this information in a
  vacuum
• Must have changes noted in remarks section to
  BEGIN to understand the data
• Still need to discuss with industry
• Data serves many and allows for numerous
  estimating techniques
• CSDR data can provide value to industry as well
• Contractors need accurate estimates of the costs
  required to complete a task in order to ensure
  maximum productivity and profitability.
• Contractor estimates that are too low can reduce
  profits if the contract is firm fixed price
• Contractor estimates that are too high will
  diminish a contractors ability to compete in the
  marketplace.

21
Summary (cont)

• CSDRs provide basis of cost estimates of future
  systems
• Long and painful DoD experience has shown that
  actual costs of developing and producing
  systems are, by far, the best basis for cost
  estimates
• Credible cost estimates lead to realistic budgets
  and executable programs
• CSDRs are DoDs only systematic mechanism for
  capturing definitionally consistent actual cost
  and software data across weapon system commodities

22
Developing Credible Cost Estimates Using CSDR
Data