Tenet

1
Tenet 4

• NASA Cost-Risk Assessment is Composed of
  Cost-Estimating Relationship (CER)
• and Technical Risk Assessment
• plus Cost-Element Correlation Assessment

2
A Projects Technical Descriptionis Not Enough

• A Technical Description (as provided in the CARD,
  for example) Does not Contain All Information
  Needed for a Realistic Cost Estimate
• The Technical Description Does not Describe How
  Difficult It is to Build the System, vis-à-vis
• Beyond State-of-the-Art Technology
• Software Development, Integration, and Test
• Other Risk Issues
• Yet System Cost Depends Heavily on How Difficult
  it is to Overcome the Risk Issues
• Difficulty Can be Translated into Additional
  Money and/or Additional Time
• Ignoring Such Difficulty Can (and Does) Lead to
  Cost Overruns and Schedule Slips

3
The Risk-Management Plan

• A Projects Risk-Management Plan Supplements its
  Technical Description by Providing Project
  Managers with Additional Information
• A Watch List of Risk Issues that May Cause
  Problems in Bringing the Project to a Successful
  Conclusion on Budget and on Time
• An Assessment of How Each Listed Risk Issue Can
  be Circumvented or Satisfactorily Resolved
• An Estimate of Additional Time and Resources,
  Including Personnel, that May Have to be Applied
  to Each Risk Issue
• Information from the Risk-Management Plan Can
  Support the Cost-Estimating Process
• (Additional Time)x(Additional Personnel)
  Additional Cost
• But Not All Risks Will Come to Pass Thats Why
  They are Discussed in the Risk-Management Plan,
  Rather than the Technical Description

4
Error Sources in Estimating Costs

• Basic Estimating Methodology
• Statistical Error Inherent in CER
• Not Quite Perfect Analogy
• Variability of Bottom-up Assessment
• Unreliability of Vendor Quote
• Characteristics of Specific Program
• Technical Risk
• Programmatic Risk, Including Schedule Risk
• Risk Associated with GFE and COTS

5
Error Sources of CER-BasedCost Estimates

• Inability of Any CER to Account for All
  Influences on Cost, No Matter How Many Inputs it
  Allows
• Incorrectness of Algebraic CER Model to which
  Cost Numbers in Data Base are Statistically Fit
• Explicit CERs are Derived from Historical Cost
  Data by Minimizing a Quality Metric, typically
  the Standard Error of the Estimate (SEE), that
  Depends on the Algebraic Model
• SEE Is Calculated by Minimizing Sum of Squared
  Differences between CER-Based Estimates and
  Actuals (in Either Dollar or Percentage Terms)
  and Dividing by a Factor Involving Number of Data
  Points Contributing to Development of CER
• SEE is an Estimator of True Standard Deviation
  ? of Errors in the Knowledge Base of Historical
  Cost Data Points, Assuming the Algebraic Model is
  Correct
• Location of Cost Driver Value x among Parameter
  Values Comprising Historical Cost Data Base
• If x is Located Near Center of Range of Parameter
  Values, CER will Provide Fairly Precise Estimate
  of the Systems Cost
• If x is Located Far From Center of Range,
  CER-based Estimate will be Considerably Less
  Precise

6
CER-Based-Estimating State of the Art

• Ordinary Least Squares (OLS)
• Model Cost as a Linear Function of One or More
  Cost Drivers
• Estimating Problem is Completely Solved
• Explicit Algebraic Formulas Exist for the Upper
  and Lower Bounds of Confidence and Prediction
  Intervals for Any Value of Cost-Driving Parameter
  at Any Level of Confidence
• Width of Interval Depends on Both the CERs
  Standard Error of the Estimate and Location of
  the Cost-Driver Value x
• Special Nonlinear CER Forms
• Model Cost as One of a Particular Class of
  Nonlinear Functions
• Such Nonlinear Forms Can be Made OLS-Solvable by
  an Algebraic (usually Logarithmic) Transformation
• Confidence and Prediction Intervals Can be
  Calculated in a Roundabout Way by Applying the
  Inverse Transform
• Unfortunately, the Geometric Distortion that
  Results from the Inverse Algebraic Transformation
  Makes It
• Impossible to Establish Symmetric Intervals
• Difficult to Compute the Most Efficient
  Intervals Based on the Data Available
• General Nonlinear CER Forms
• Model Cost Using Any Nonlinear Functional Form
• Standard Error of the Estimate Can be Calculated,
  as Well as Some Information About Variances of
  the Coefficients
• But Problem of Confidence and Prediction
  Intervals Appears Not to Have Been Solved

7
Precision of Estimate Over Entire Range of
Possible Cost-Driver Values
Cost Driver Mean 67.30 Standard Error of
Estimate 5.26 Cost Driver Range 53 to 79
8
Bounding the Predicted Cost at Cost-Driver Value x

• Prediction Interval Based on the Variance of the
  Difference Between the Actual Cost Y and the
  Estimated Cost Y is
•  
• Degree of Confidence Associated With This
  Interval is Again (1-?)100, which is Enforced by
  the Choice of the Percentage Point of the t
  Distribution, Namely t?/2,n-2

9
Fred Timson on Value of Prediction Intervals to
Cost Estimators

• The prediction intervals are so wide for even
  the best regression that there is little
  likelihood that the realized cost of a future
  weapon system acquisition program will be near
  the predicted cost.
• The predictive statistics (sampling
  distributions for future observations) overlap to
  such an extent for some regressions that the
  ability to discriminate between the distributions
  for airframes that differ in weight by a factor
  of two is very doubtful.

10
Mathematical Issues in Prediction Using
General-Error Regression CERs

• Deriving the General-Error CER
• For the Model yi (a bxic)Ei, the Sum of
  Squared Multiplicative Errors
• is to be Minimized
• It is Generally Impossible to Obtain Explicit
  Formulas for A, B, and C by Calculus or any Other
  Mathematical Method, so Some Kind of Iterative
  Procedure is Needed for Convergence to Acceptable
  Numerical Values of the Coefficients
• In 1974 Wedderburn Published Approximate
  Expressions for the Variances and Covariances of
  the Coefficients A, B, and C However, He Did
  not Carry Through his Derivation to the Point
  Needed for Prediction Intervals, Namely to the
  Point of Calculating the Variance of the
  CER-based Estimate Itself

11
Wedderburns Matrix for IRLS-Based CERs of the
Form Y abXc

• Wedderburns Variance/Covariance Matrix Has the
  Form
• In the Case of Y abXc, D is the Matrix Inverse
  of

12
Specific Program-Related Risks

• Define Impact of Program-Related Risks Using
  WBS-Element Probability Distributions
• Technical Risk, e.g., Probable Additional
  Development Costs due to Requirements for Beyond
  State-of-the-Art Technology
• Programmatic Risk, e.g., Probable Additional
  Costs Associated with Schedule Slippage due to
  Various Causes
• GFE and COTS Risk, e.g., Probable Additional
  Funding Needed to Cover for GFE and COTS
  Inadequacies
• Program-Related Risks are Typically Correlated
• Ignoring or Failing to Account for Inter-Element
  Correlation Leads to Narrow Total-Cost
  Distributions
• Assumption that Correlations are Negligible Masks
  Estimating Uncertainty

13
Tenet 6

• NASA Cost-Risk Probability Distributions
• are Justifiable
• and Correlation Levels are Based on Actual Cost
  History to the Maximum Extent Possible

14
A Projects Technical Descriptionis Not Enough

• A Technical Description (as provided in the CARD,
  for example) Does not Contain All Information
  Needed for a Realistic Cost Estimate
• The Technical Description Does not Describe How
  Difficult It is to Build the System, vis-à-vis
• Beyond State-of-the-Art Technology
• Software Development, Integration, and Test
• Other Risk Issues
• Yet System Cost Depends Heavily on How Difficult
  it is to Overcome the Risk Issues
• Difficulty Can be Translated into Additional
  Money and/or Additional Time
• Ignoring Such Difficulty Can (and Does) Lead to
  Cost Overruns and Schedule Slips

15
The Risk-Management Plan

• A Projects Risk-Management Plan Supplements its
  Technical Description by Providing Project
  Managers with Additional Information
• A Watch List of Risk Issues that May Cause
  Problems in Bringing the Project to a Successful
  Conclusion on Budget and on Time
• An Assessment of How Each Listed Risk Issue Can
  be Circumvented or Satisfactorily Resolved
• An Estimate of Additional Time and Resources,
  Including Personnel, that May Have to be Applied
  to Each Risk Issue
• Information from the Risk-Management Plan Can
  Support the Cost-Estimating Process
• (Additional Time)x(Additional Personnel)
  Additional Cost
• But Not All Risks Will Come to Pass Thats Why
  They are Discussed in the Risk-Management Plan,
  Rather than the Technical Description

16
Achievable Software-DevelopmentSchedules
17
CER for Military SpaceGround-System Test Software
18
Software Cost-Risk Experience

• Cost Histories of Software-Development Projects
  Show a Definite Trend Toward Significant
  Underestimation of Number of Lines of Code and
  Cost
• Aerospace Corp. Study Found Lines-of-Code Growth
  of about 150 for Space-Related Ground-System
  Software Projects
• Naval Center for Cost Analysis Found Average
  Lines-of-Code Growth of 63 for Software Projects
  of Various Types (http//www.ncca.navy.mil/softwar
  e/handbook/software.htm)
• Developer Productivity, Measured in Lines of Code
  per Developer-Month, is Typically Overestimated
• This Results in Cost Growth, Even if
  Lines-of-Code Estimate is Accurate
• Data Collected Over Time Appear to Show Some
  Productivity Improvement, but not Enough to
  Overcome Estimating Optimism

19
Lines-of-Code Estimating Risk
20
Historical Software Coding Rates
21
What is the Risk Multiple for Software?

• Its 8 (times the contactor estimate)
• Why?
• Number of Lines of Code Grows by a Factor of
  About 2.5
• Programmer Productivity, Almost Always Initially
  Estimated at 300 Lines Per Programmer-month,
  Inevitably Slips to Around 85 As the Project
  Moves Forward Equivalent to a Cost-growth Factor
  of 300/85 3.5
• 2.5 3.5 8.75
• So Were Being Nice About It
• This Multiple is Applied Wherever in Each WBS
  Element the Cost of Software is Estimated
• The Cost Distribution Will Be Right-Triangular
• L M, but H 8M

22
Maximum Possible Underestimation of Total-Cost
Sigma (Theoretical)

• Percent Underestimated When Correlation Assumed
  to be 0 Instead of r

23
Selection of Correlation Values

• Ignoring Correlation Issue is Equivalent to
  Assuming that Risks are Uncorrelated, i.e., that
  All Correlations are Zero
• Square of Correlation Represents Percentage of
  Variation in one WBS Elements Cost that is
  Attributable to Influence of Anothers
• Reasonable Choice of Nonzero Values Brings You
  Closer to Truth
• Most Elements are, in Fact, Pairwise Correlated
• 0.2 is at Knee of Curve on Previous Charts,
  thereby Providing Most of the Benefits at Least
  Commitment