Setting Up, Getting, and Using CostRisk Feedback

1
Setting Up, Getting, and Using Cost-Risk
Feedback

• by
• David R. Graham
• Senior Engineering Specialist
• The Aerospace Corporation

2002 National Conference and Educational
Workshop Chaparral Suites Conference
Hotel Scottsdale, AZ 13 June 2002
2
Motivation
3
Program Manager is in Charge

• USD(ATL) Goal 1 establish the credibility
  and the effectiveness of the acquisition and
  logistics support process
• Congress is responsible to the taxpayer for
  well-run programs
• Program Managers run programs
• Congress helps out when programs run into
  trouble
• Program managers need tools to help themselves,
  for example
• Setting up, getting, and using cost-risk feedback

4
Outline

• The Cost-Risk Continuum
• Requirements/Mission Allocation
• Organize the Estimate
• Develop the Point Estimate
• Analyze Risk
• Translate Risk into Cost-Risk
• Communicating Cost-Risk Requirements
• Contractor Selection
• Cement Risk Management Plans (IBR)
• Monitor Cost Performance
• 10 Update Cost and Risk
• 11 Use Past Data for Future Efforts
• 12 Evaluate Cost Models

Setting Up Cost-Risk Feedback
Getting Cost-Risk Feedback
Using Cost-Risk Feedback
5
The Cost-Risk ContinuumEstimating EVMS
Data Analysis
2. Organize Estimate Rqmts/Functions/WBS Allocati
on
1. Mission/Requirements Allocation
3. Develop Cost Schedule Estimate
Set Up
Build system function/WBS matrix (CARD)
Develop Reference Point Cost Schedule Estimate
4. Analyze Risk
Cost Performance Trades
Assess WBS element risk
6. Communicating Cost Schedule Requirements
5. Translate Risk into Cost Schedule Impacts
7. Choose Contractor
Get
Develop RFP EVM LCCE contract update
requirements
Cost/Schedule risk assessment
Review Cost Proposal during Source Selection
9. Monitor Cost/Schedule Performance
10. Update Cost, Schedule and Risk
8. Cement Risk Management Plans with Chosen
Contractor
Cost estimate cost/sched-risk
assessment updates
Participate in IBR (Integrated Baseline Review)
CPR analysis
Key Bridging Event
12. Perform VV
Use
11. Use Past Data for Better Future Estimates
Assist in end-of- contract EVM data analysis
Evaluate Cost Models
6
Setting Up Cost-Risk FeedbackContinuum Steps 1-5
7
Cost Performance Trades
8
Requirements
Missions
Costs include risk margin
Benefit Measure
9
Alternative Requirements Level Allocation Costs
Constrained at 17,300
10
Cost Performance Trades
Optimization Curve with Knee
11
Build system function/WBS matrix (CARD)
12
Build system function/WBS matrix (CARD)
13
Continuum Step 3Develop the Point Estimate
Develop Reference Point Cost Schedule Estimate

• Apply cost model/methodology by WBS element
• Model examples Unmanned Spacecraft Cost Model
  Passive Sensor Cost Model Small Satellite Cost
  Model SEER Price
• Methodology examples Analogy grass-roots
  parametric BOGSAT SWAG
• Necessary for establishing a reference, best,
  or most likely estimate

14
Continuum Step 4 Analyze/Assess WBS Element Risk
Assess WBS element risk
WBS Technology Risk Scale
Note This is one of many risk category
templates (e.g., complexity design/engineering,
interactions/dependencies, schedule, etc.)
15
Continuum Step 5Translate Risk into Cost
Impacts
TECHNLGY DES/ENG COMPLEXITY
SCHEDULE
TOTAL
WBS ELEMENT PROFILES 0.35 0.25 0.2 0.2
1.0
1. PESSIMISTIC PROFILE HIGH VERY
HIGH VERY HIGH HIGH 5.9 2.
REFERENCE PROFILE MOD MOD MOD MOD 2.9 3.
OPTIMISTIC PROFILE LOW MOD LOW MOD MOD 2.0
5.9
HIGH END RISK FACTOR FOR S/C
PESSIMISTIC SCORE


2.0

2.9
REFERENCE SCORE
THESE FACTORS ARE THEN APPLIED TO THE RPE TO
OBTAIN THE LOW AND HIGH END COSTS
Cost/Schedule risk assessment
RPE
2.0RPE (HIGH ENDCOST)
0.7RPE (LOW ENDCOST)
16
Sum WBS Cost Distributions
Cost/Schedule risk assessment
SUMMARY COST DISTRIBUTION
WBS COST DISTRIBUTIONS
BELL CURVE
PROBABILITY DENSITY

RPE

åRPE
COST

100


RPE
S-CURVE
85
70

CONFIDENCE LEVEL

50

20

RPE
åRPE
COST
17
Getting Cost-Risk FeedbackContinuum Steps 6-8
18
The Cost-Risk ContinuumEstimating EVMS
Data Analysis
2. Organize Estimate Rqmts/Functions/WBS Allocati
on
1. Mission/Requirements Allocation
3. Develop Cost Schedule Estimate
Set Up
Build system function/WBS matrix (CARD)
Develop Reference Point Cost Schedule Estimate
4. Analyze Risk
Cost Performance Trades
Assess WBS element risk
6. Communicating Cost Schedule Requirements
5. Translate Risk into Cost ScheduleImpacts
7. Choose Contractor
Get
Develop RFP EVM LCCE contract update
requirements
Cost/Schedule risk assessment
Review Cost Proposal during Source Selection
9. Monitor Cost/Schedule Performance
10. Update Cost, Schedule and Risk
8. Cement Risk Management Plans with Chosen
Contractor
Cost estimate cost/sched-risk
assessment updates
Participate in IBR (Integrated Baseline Review)
CPR analysis
Key Bridging Event
12. Perform VV
Use
11. Use Past Data for Better Future Estimates
Assist in end-of- contract EVM data analysis
Evaluate Cost Models
19
Continuum Step 6Communicating Cost-Risk
Requirements
Develop RFP EVM LCCE contract update
requirements

• RFP EVM LCCE CDRL Instructions
• Necessary to inform the contractor
• Specifies high-risk WBS elements
• Specifies reporting levels
• Specifies update to S-curve at significant
  milestones
• Specifies cost growth reporting by categories

20
Continuum Step 7Contractor Selection
Review Cost Proposal during Source Selection

• Source Selection
• Proposals received from bidders
• Evaluate cost proposals
• Models/methodologies
• Cost-risk
• Response to EVMS CDRL instructions
• Begins the implementation of insight vs oversight
  
• Ultimately to achieve a partnership with
  contractor

21
From Oversight to Insight
Review Cost Proposal during Source Selection
Where weve been
Where were going
Observe
Observe
Manage
Track
Manage 20 of risks that drive 80 of cost
Manage

• Contractor Control of Baseline
• Trust and Teaming Critical Condition

22
Continuum Step 8Integrated Baseline Review (IBR)
Participate in IBR (Integrated Baseline Review)
Key Bridging Event

• Project managers receive pre-IBR training
• Correlate proposal cost-risk estimates with
  baseline plans
• Review high/medium risk WBS elements for CPR
  reporting
• Interview contractor engineers responsible for
  risk management
• Ensure adequate resources are planned
• Ensure Control Account Manager (CAM) understands
  reporting responsibilities
• Reporting on high/medium risk WBS elements
• No matter WBS level
• Reporting on risk-driven and externally-driven
  cost growth
• Ensures Oversight-to-Insight-to-Partnership IPT

23
Using Cost-Risk FeedbackContinuum Steps 9-12
24
The Cost-Risk ContinuumEstimating EVMS
Data Analysis
2. Organize Estimate Rqmts/Functions/WBS Allocati
on
1. Mission/Requirements Allocation
3. Develop Cost Schedule Estimate
Set Up
Build system function/WBS matrix (CARD)
Develop Reference Point Cost Schedule Estimate
4. Analyze Risk
Cost Performance Trades
Assess WBS element risk
6. Communicating Cost Schedule Requirements
5. Translate Risk into Cost ScheduleImpacts
7. Choose Contractor
Get
Develop RFP EVM LCCE contract update
requirements
Cost/Schedule risk assessment
Review Cost Proposal during Source Selection
9. Monitor Cost/Schedule Performance
10. Update Cost, Schedule and Risk
8. Cement Risk Management Plans with Chosen
Contractor
Cost estimate cost/sched-risk
assessment updates
Participate in IBR (Integrated Baseline Review)
CPR analysis
Key Bridging Event
12. Perform VV
Use
11. Use Past Data for Better Future Estimates
Assist in end-of- contract EVM data analysis
Evaluate Cost Models
25
Relative Timescales for Cost Schedule
Management Continuum Activities
Concept Tech Development Mid-Term Review
7. Choose Contractor
10. Update Cost, Schedule and Risk
10. Update Cost, Schedule and Risk
9. Monitor Cost/Schedule Performance
Review Cost Proposal during Source Selection
5. Translate Risk into Cost ScheduleImpacts
Cost estimate cost-risk assessment updates
Cost estimate cost-risk assessment updates
3. Develop Cost Schedule Estimate
1. Requirements/Functional Allocation
Cost/Schedule risk impact assessment
Good CPR's
Develop Reference Point Cost Estimate
Cost Performance Trades
2. Organize Estimate
4. Analyze Risk
6. Communicating Cost Schedule Requirements
8. Cement Risk Management Plans with Chosen
Contractor
Build system function/WBS matrix (CARD)
Participate in IBR (Integrated Baseline Review)
Assess WBS element risk
Develop RFP EVM LCCE contract update
requirements
9. Monitor Cost/Schedule Performance
Key Bridging Event
Good CPR's
Months to Years
6 Months Max
24-48 Months
6-12 Months
Main Point Longest time period involves CPR
analysis, cost estimate, and cost and schedule
risk updating and analysis - SPO must be actively
engaged to have successful risk management
26
Format 5 FEEDBACK!
Continuum Step 9Monitor Cost-Risk Performance
CPR analysis

• Separate discussion of CV, SV (current and
  cum) and VAC for
• each control account-identified medium and
  high-risk WBS element
• Trends, EACs, cost/performance trades, effects
  on other WBS elements

• Clear description of reason for variance
• Technical risk-driven problems design too
  ambitious performance
• too hard to achieve dependency on other WBS
  late outputs
• External risk-driven problems budget cuts
  requirements changes

• Corrective actions described and statused
• New subcontractor added design simplified
  back off objective requirement
• to threshold level

• Technical, schedule, and cost impacts

• Impact to estimate at completion

• Should be written by responsible manager,
  e.g.,CAM/WPM

27
Where Are the Significant Problems?
sorted by CV
28
EAC realism
29
How Should We Do EAC Estimates?
CPR analysis

• Treat every EAC estimating task as a Cost-Risk
  Analysis
• Recognize uncertainty inherent in WBS-Element
  EACs
• Construct cost-at-completion probability
  distribution for each WBS element
• Sum WBS-Element costs-at-completion statistically
  (via Monte-Carlo or analytic approximation)
• Avoid meaningless outcome of roll-up procedure
• Get mean, median, mode of total-system cost at
  completion
• Get all cost-at-completion percentiles as
  products of a Cost-Risk Analysis

By Dr Stephen A. Book. This, and the next 5
charts, were developed by Dr. Book when he was
with The Aerospace Corporation
30
EACs by Cost-Risk Methods
CPR analysis

• Construct cost-at-completion probability
  distribution for each WBS element
• Use CPI-Based EACs as lower bounds
• Use SCI-Based EACs as upper bounds
• Sum WBS-Element costs-at-completion statistically
  (via Monte-Carlo or analytic approximation)
• Apply in-house-developed Monte Carlo Tool or
  commercial software such as Crystal BallTM or
  _at_RiskTM
• Obtain mean, median, mode of total-system cost at
  completion, as well as all cost-at-completion
  percentiles, as standard outputs of Monte Carlo
  software

Dr. David S. Christensen
31
CPR analysis
Proposed Right-Triangular Distribution
of WBS-Element Cost-to-Complete (inferred from
behavior of sample EV data )
EAC Floor The smaller of (1) Element
CPI-based EAC (2) Element T.P.CPI-based EAC
EAC Ceiling The larger of (1) Element
SCI-based EAC (2) Element T.P.SCI-based EAC
32
Sum WBS Cost Distributions
CPR analysis
SUMMARY COST DISTRIBUTION
WBS COST DISTRIBUTIONS
BELL CURVE
PROBABILITY DENSITY

Ref EAC

åRPE
COST

100


S-CURVE
Ref EAC
85
70

CONFIDENCE LEVEL

50

20

Ref EAC
åRPE
COST
33
Continuum Step 10Update Cost and Risk
Cost estimate cost/sched-risk
assessment updates

• Contractor(s) tasked, via contract cost CDRLs, to
  update proposal and LCCE at major contract
  milestones
• Partnership paradigm provides framework for
  Government and contractor to work together
• Especially in evaluating risk
• Use same methodology as initial estimate and
  compare
• Government/contractor engineers participate in
  risk assessment update
• Re-accomplish initial risk assessment and cost
  impacts
• Develop updated cost estimate CDF
• Compare results with EVMS EACs

34
Cost-Risk Updates Over Time
Cost estimate cost/sched-risk
assessment updates
Near End-of-Contract
Mid-Contract
Contract End - 4th Update
Proposal
100
Confidence Level
50
Initial Government Point Estimate and Budget
25
Contract Target Cost
Actual
Could Have Picked This Low Confidence Level
Initially!!
Cost
35
Cost-Risk Feedback Analysis Crosscheck Methods
Cost estimate cost/sched-risk
assessment updates

• Advanced methods to augment S-curve updates
• Relative Efficiency Weighting (REW)
• Future vs past perspective of performance
  efficiencies
• Headlight vs Taillight metric
• Requires engineering input
• Bayesian Conditional Probabilities
• Explicit identification of initial probabilities
  and modified probabilities based on new
  information
• Headlight vs Taillight metric

36
Relative Efficiency Weighting (REW)Efficiency
Risk Profile (ERP) Development
Cost estimate cost/sched-risk
assessment updates

• Identify weight risk categories influencing
  future efficiency
• Quantify pessimistic, reference, and optimistic
  efficiency cases for each WBS element
• Rate cases against future efficiency risk
  categories
• Reference case is defined by cum-to-date trend
• Build (Pessimistic/Reference)
  (Optimistic/Reference) relative efficiency ratio
  factors
• Apply factors to cum-to-date CPI or SCI SPI
• Identifies optimistic and pessimistic CPI, SCI,
  SPI

37
Cost estimate cost/sched-risk
assessment updates
REW Rating Step
TECHNOLGY DES/ENG COMPLEXITY
SCHEDULE
TOTAL
WBS ELEMENT PROFILES 0.2103 0.3863
0.2618 0.1416 1.0
1. PESSIMISTIC ERP Low Eff
Mlow Eff Vlow Eff Low Eff
0.184 2. REFERENCE ERP Hi Eff Low
Eff Mlow Eff Mlow Eff 0.339 3.
OPTIMISTIC ERP Mhi Eff Mhi Eff
HiEff Mod Eff 0.587
38
Cost estimate cost/sched-risk
assessment updates
Ratio Factor Step
0.587
OPTIMISTIC ERP SCORE
OPTIMISTIC ERP RISK FACTOR TO APPLY TO ACTUAL CPI


1.73

REFERENCE ERP SCORE
0.339
39
Application Steps
Cost estimate cost/sched-risk
assessment updates
1. THESE FACTORS ARE THEN APPLIED TO THE
REFERENCE CPI TO OBTAIN THE OPTIMISTIC AND
PESSIMISTIC CPIs. 2. THE RESULTING CPIs ARE
THEN APPLIED TO EAC FORMULAS TO OBTAIN THE
OPTIMISTIC AND PESSIMISTIC EACs.
1
0.54 RCPI Pessimistic CPI (PCPI)
Reference CPI
1.73 RCPI Optimistic CPI (OCPI)
2
ACWPC BCWR/OCPI Optimistic EAC
ACWPC BCWR/PCPI Pessimistic EAC
Reference EAC
40
REW Matrix SummaryRisk-to-EAC Efficiency
Translation
Cost estimate cost/sched-risk
assessment updates
TECHNLGY DES/ENG COMPLEXITY
SCHEDULE
( NOTE All rating intensities (e.g., Low Eff)
have associated ratio-scale weighted values for
use in producing weighted average totals)
TOTAL
Rating Step
WBS ELEMENT PROFILES 0.2103 0.3863 0.2618
0.1416 1.0
1. PESSIMISTIC ERP Low Eff Mlow
Eff Vlow Eff Low Eff
0.184 2. REFERENCE ERP Hi Eff Low Eff Mlow
Eff Mlow Eff 0.339 3. OPTIMISTIC ERP Eff Mhi
Eff Mhi Eff Eff 0.587
0.587
OPTIMISTIC ERP SCORE
OPTIMISTIC ERP RISK FACTOR TO APPLY TO ACTUAL CPI


1.73

Ratio Factor Step
0.339
REFERENCE ERP SCORE
1. THESE FACTORS ARE THEN APPLIED TO THE
REFERENCE CPI TO OBTAIN THE OPTIMISTIC AND
PESSIMISTIC CPIs. 2. THE RESULTING CPIs ARE
THEN APPLIED TO EAC FORMULAS TO OBTAIN THE
OPTIMISTIC AND PESSIMISTIC EACs.
Application Steps
1
2
ACWPC BCWR/OCPI Optimistic EAC
Reference EAC
0.54 RCPI Pessimistic CPI (PCPI)
Reference CPI
1.73 RCPI Optimistic CPI (OCPI)
ACWPC BCWR/PCPI Pessimistic EAC
41
Sum WBS Cost Distributions
Cost estimate cost/sched-risk
assessment updates
SUMMARY COST DISTRIBUTION
WBS COST DISTRIBUTIONS
BELL CURVE
PROBABILITY DENSITY

Ref EAC

åRPE
COST

100


S-CURVE
Ref EAC
85
70

CONFIDENCE LEVEL

50

20

Ref EAC
åRPE
COST
42
Bayesian Statistics Crosscheck
Cost estimate cost/sched-risk
assessment updates

• Estimate pessimistic, optimistic, and reference
  costs
• Develop and utilize estimates of probabilities
  for risk scenarios (i.e., a priori
  probabilities) and derive initial expected
  monetary value (EMV)
• Use risk scenario indicators (i.e., conditional
  events), that give us more information about the
  true scenario, to update a priori probabilities
• Apply the Bayesian statistics formula to derive
  a posteriori probabilities and produce an
  updated EMV

43
Cost estimate cost/sched-risk
assessment updates
a priori Scenario Probabilities
44
Cost estimate cost/sched-risk
assessment updates
a posteriori Indicator Scenario Probabilities
45
Cost estimate cost/sched-risk
assessment updates
Calculate EMVs Using a priori a posteriori
Probabilities
46
Continuum Step 11Use Past Data for Better
Future Estimates
Assist in end-of- contract EVM data analysis

• Cost-risk performance documented
• CVs, SVs, EACs, LREs, VACs,
• Technical risk-driven cost growth
• External risk-driven cost growth
• Periodic contract and life cycle cost updates
• Updates to statistical variance
• Implies increase/decrease of risk/cost-risk
• Future acquisitions and source selections
• Cost data collection less expensive
• Ad hoc cost data collection is expensive

47
Continuum Step 12Perform Verification/Validation
of Cost Models
Evaluate Cost Models

• Cost models/methodologies being continually
  updated
• Data from documented EVM risk/cost-risk can be
  used in updating
• Data from EVM reporting valuable in checking
  assumptions
• Leads to improved cost estimating models