Cost Risk Analysis Panel

1
Cost Risk Analysis Panel

• Timothy P. Anderson
• NASA Cost Symposium
• March 9, 2004

2
Predicting the Future

• A TECHNICAL REALITY Fundamental inability to
  predict the future!
• Since it is impossible to make exact predictions
  we have
• Cost estimating relationships (CERs)
• based on historical data
• for the purpose of making statistically-based
  educated guesses
• about the cost of systems that have yet to be
  built.

3
Probabilistic Nature of Cost Estimates

• Typical CERs are based on historical cost data.

• And, typical cost estimates have probability
  distributions.

• Cost estimates are really probability
  distributions and not deterministic!
• But, prevailing practice is that cost estimators
  report, say, the 50th percentile of the
  probability distribution as the cost estimate,
  and decision-makers then choose to budget at a
  value that is even less than that!

4
A Narrative

• Consider the following exchange between a
  decision-maker and his cost analyst
• Mr. OKeefe So, Joe, how much will our new
  system cost?
• Joe Hamaker Sir (looking at the 50th percentile
  number), our estimate is 115M.
• Mr. OKeefe Well, Joe, thats too high. The
  contractor says it will only cost 72M. Well
  just have to manage to that number.
• Joe Hamaker Aye, aye, sir.
• Sound familiar?

5
The Consequences

• Since cost probability distributions tend to be
  right-tailed, then the 50th percentile turns
  out to be less than the expected (mean) cost
  estimate.
• Also, if cost is estimated at the 50th
  percentile, then there is a 50 chance that the
  actual cost will be greater than the cost
  estimate!
• Moreover, when decision-makers choose to budget
  at a value less than the 50th percentile, say the
  5th percentile, then they are guaranteeing a 95
  probability of a budget overrun!

6
More Consequences

• Consider the following exchange between our
  decision-maker and his cost estimator one year
  later
• Mr. OKeefe Joe, the contractor says our
  program is now projected to cost 135M! You told
  me it would be 115M! YOURE FIRED!
• Joe Hamaker But Sir, its all Carpios fault!

7
Andersons Law

• Cost estimators should explain the probabilistic
  nature of their estimate, and
• Acquisition decision-makers should set budgets so
  that they have a reasonable probability of
  budgetary success.

8
A Different Narrative

• Heres how that initial exchange should have
  gone
• Mr. OKeefe So, Joe, how much will our future
  system cost?
• Joe Hamaker Sir, weve developed the following
  cost probability distribution (shows the
  distribution to the decision-maker). This
  distribution has an expected value of 120M, but
  notice, sir, that there is a fair amount of
  variability in this estimate. With a standard
  deviation of 35M, the cost could easily exceed
  155M.
• Mr. OKeefe Well, Joe, thats too high. The
  contractor says it will only cost 72M. Well
  just have to manage to that number.

Continued
9
A Different Narrative

• Joe Hamaker Yes, sir, I understand what the
  contractor says, but we have confidence in this
  cost estimate based on our knowledge of history.
  The contractor might be right, but the odds are
  against it. If you really want this new system,
  then we need to be prepared to budget more than
  72M for it. In fact, 72M falls at the 5th
  percentile of this estimate. That means there is
  only a 5 chance that 72M will be enough, with a
  corresponding 95 chance that the actual cost
  will exceed 72M.
• Mr. OKeefe HmmI see what you mean. So where
  should we set the budget to have, say, an 80
  probability of avoiding a budget overrun?
• Joe Hamaker Well, sir, you should set the
  budget at the 80th percentile of the estimate, or
  146M.
• Mr. OKeefe Okay, Joe, thanks for the insight.
  Ill see what I can do about getting the budget
  you recommend.

10
Better Consequences

• Suppose, because of the high priority of this
  system, Mr. OKeefe was able to budget to Joes
  recommendation of 146M.
• Now consider the following exchange one year
  later
• Mr. OKeefe Joe, the contractor says our program
  is now projected to cost 135M! But because of
  what you told me about budget risk, Ive got
  enough money in the budget to cover it. Thanks
  again for explaining that cost probability
  distribution to me. IM PUTTING YOU IN CHARGE OF
  CRLs!
• Joe Hamaker Aye, aye, sir!

11
Tenet 7

• NASA cost risk assessment ensures cost estimates
  are for likely-to-be programs vice as
  specified for optimum credibility.

12
Likely-To-Be vs. As Specified

• As Specified
• No engineering issues
• No uncertainties
• This time well get it right
• No weight growth
• Optimistic assumptions
• No requirements changes
• No ECPs
• No schedule creep

• Likely-To-Be
• Test failures
• Technical uncertainty
• As usual, we made mistakes
• Unanticipated weight growth
• We were too optimistic
• Some requirements change
• Some ECPs
• Some schedule creep

13
All Programs Have Experienced Problems

• Listed below are all of the perfectly-planned
  NASA programs that had no cost increases, weight
  growth, schedule creep, ECPs, etc.
• We are continually seeking programs to add to
  this list, but havent found any to date.

NOTE This list is empty.
14
Estimating Likely-To-Be

• History shows that no space acquisition goes
  along as specified
• Weight growth
• Software growth
• Schedule creep
• Cost growth
• Everyone wants NASA programs to be successful.
• But a key measure of success is getting the cost
  right!
• Underestimating cost with the expectation that
  things will go as specified is a recipe for
  disappointment.

15
Tenet 6

• correlation levels are based on actual cost
  history to the maximum extent possible.

16
Correlation Based on Actual History

• I agree, and Ill defer comments on this to Steve
  Book.
• However, this is not always possiblethen what?
• May be necessary to subjectively assess
  approximate correlation between WBS elements.
• Better than nothing.

17
Tenet 11

• NASA decision-makers need to know
• How much money is in the estimate to cover risk
  events
• To which WBS elements are they allocated and
• The likelihood of an overrun.

18
NRO Cost Group Risk Process

• Step 1 Define likely-to-be program.
• Rather than the optimistic program that the
  contractor and program manager are planning for.
• Step 2 Quantify the probability distributions
  describing the modeling uncertainty of all CERs,
  cost factors, and other estimating methods.
• Specifically, the type of distribution (normal,
  triangular, lognormal, beta, etc.)
• The mean and variance of the distribution
• Step 3 Quantify the correlation between all WBS
  elements that are estimated using CERs and other
  methods.
• Correlation affects the overall cost variance

19
NRO Cost Group Risk Process

• Step 4 Set up and run the cost estimate in a
  Monte Carlo framework (e.g., Crystal Ball,
  _at_RISK), resulting in a baseline estimate.
• This will provide a probability distribution of
  the cost based on cost estimating model
  uncertainty only.
• Report the MEAN as the baseline expected cost.
• Step 5 Now incorporate technical uncertainty and
  discrete risks.
• Step 5a Set up a new baseline estimate which
  also contains any discrete risk events that are
  to be guarded against.
• Quantify appropriate modeling uncertainties and
  correlations, as in Steps 2 and 3, for these
  discrete risks.
• Step 5b Define the probability distributions for
  all CER input variables.
• Also may need to quantify correlation between CER
  input variables.

20
NRO Cost Group Risk Process

• Step 6 Re-run the Monte Carlo simulation with
  random CER input variables and discrete risk
  events, resulting in a final risk-adjusted
  estimate.
• Results in a new risk-adjusted cost probability
  distribution.
• Wider and shifted to the right.

Baseline Estimate
Risk-Adjusted Estimate
21
NRO Cost Group Risk Process

• Step 7 Assess risk dollars.
• Difference between the risk-adjusted mean and
  the baseline mean represents the estimate of
  risk dollars.
• Risk dollars can be allocated downward to any
  level of WBS using a variety of simple approaches.

22
NRO Cost Group Risk Process

• Step 8 Assess budget risk.
• Area under the PDF to the right of the budget
  represents budget risk.

23
NRO Cost Group Risk Process

• Step 9 Assess management reserve.
• Difference between mean of risk-adjusted estimate
  and budget represents management reserve.