Using Process Improvement and Knowledge Management for Better Predictive Analysis Capability

1
Using Process Improvement and Knowledge
Management for Better Predictive Analysis
Capability

• Rick Hefner Marilee J. Wheaton 310.812.7290 31
  0.813.6510 rick.hefner_at_trw.com marilee.wheaton_at_t
  rw.com TRW One Space Park -
  R2/2130 Redondo Beach, CA 90278

Presented to Sixteenth International Forum On
COCOMO and Software Cost Modeling
2
Agenda

• Motivation for Improvement
• CMMI Level 4 Requirements and Expectations
• Organizational Metrics
• Process Database Design and Implementation
• Lessons Learned
• Systems Architecture Vision

3
Motivation for Improvement to Level 4

• Level 3 ensures well-defined, repeatable
  processes
• Competition demands better quantitative
  understanding of the contributors to cost and
  schedule
• Process productivity
• Product quality (which drives rework)
• Needed better measures, data, and analytic
  techniques for critical process and product
  characteristics
• Determine whether processes are behaving
  consistently or have stable trends (i.e., are
  predictable)
• Identify improvement in TRWs standard processes
• Identify project practices which may be best
  practices
• Understand the cost-quality-schedule tradeoffs

4
How Does Six Sigma Fit with ISO 9001 and CMMI?
Process Improvement
Best-Practices
Quality Mgmt.
Six Sigma
ISO 9000
SW CMM
Business Measures
Voice of the Customer
Change Management
Process Management
DMAICDFSS
Methods Tools
ISO 9001
CMMI

• Capability Maturity Model Initiative (CMMI) and
  ISO 9001 establish the change management and
  process management framework needed for Six Sigma
  
• Six Sigma methods and tools assist in the
  quantitative analysis needed at CMMI Levels 4/5

5
CMMI-SE/SW Staged Representation
Focus
Process Areas
Level
Causal Analysis and Resolution Organizational
Innovation and Deployment
5 Optimizing
Quantitative management
4 Quantitatively Managed
Quantitative Project Management Organizational
Process Performance
Organizational Process Focus Organizational
Process Definition Organizational Training
Integrated Project Management Risk
Management Decision Analysis and
Resolution Requirements Development Technical
Solution Product Integration Verification Validati
on
Process standardization
3 Defined
Requirements Management Project Planning Project
Monitoring and Control Supplier Agreement
Management Measurement and Analysis Process and
Product Quality Assurance Configuration Management
Basic project management
1 Performed
6
Level 4 Changes in the CMMI

• SW-CMM process areas are split by process and
  product quality
• Quantitative Process Mgmt Identify and correct
  special causes of process variation
• Software Quality MgmtDevelop a quantitative
  understanding of the quality of the project's
  software products
• CMMI process areas are split by organization and
  project
• Organizational Process PerformanceMaintain a
  quantitative understanding of the performance of
  the organizations set of standard processes, and
  provide the process performance data, baselines,
  and models to quantitatively manage the
  organizations projects.
• Quantitative Project ManagementQuantitatively
  manage the projects defined process to achieve
  established quality and process performance
  objectives.

7
Organizational Process Performance (CMMI)

• Establish and maintain a quantitative
  understanding of the performance of the
  organizations set of standard processes, and to
  provide the process performance data, baselines,
  and models to quantitatively manage the
  organizations projects.
• Required Goals
• SG 1 Establish Performance Baselines and
  ModelsBaselines and models that characterize the
  expected process performance of the
  organization's set of standard processes are
  established and maintained.
• GG 3 Institutionalize a Defined ProcessThe
  process is institutionalized as a defined
  process.

ExpectedImplementation Practices
ExpectedInstitutionalization Practices
8
Expected Implementation Practices

• SP 1.1 Select ProcessesSelect the processes or
  process elements in the organization's set of
  standard processes that are to be included in the
  organization's process performance analyses.
• SP 1.2 Establish Process Performance
  MeasuresEstablish and maintain definitions of
  the measures that are to be included in the
  organization's process performance analyses.
• SP 1.3 Establish Quality and Process Performance
  ObjectivesEstablish and maintain quantitative
  objectives for quality and process performance
  for the organization.
• SP 1.4 Establish Process Performance
  BaselinesEstablish and maintain the
  organization's process performance baselines.
• SP 1.5 Establish Process Performance
  ModelsEstablish and maintain the process
  performance models for the organization's set of
  standard processes.

9
Approach

• Our Level 3 process database supported cost
  estimation and process improvement
• Surveyed management team to establish business
  drivers
• Defined measures needed to characterize process
  performance and quality at the organizational
  level
• Defined measures needed to characterize project
  satisfaction of organizational goals
• Identified sub-processes amenable to quantitative
  management
• Defined project measures needed for quantitative
  management of those sub-processes
• Examined improvements in the organization
  standard process needed to stabilize the process
  or make measures meaningful
• Examined improvements in the projects defined
  processes

10
Example Organizational Metrics Collected and
Derived

• Collect base measures
• Size
• Effort by activity
• Cost by activity
• Number of defects (By phase)
• Derive other measures
• Defect density (defects/size)
• Productivity (size/effort)
• Defect containment (defects saves/escapes from
  defects by phase)
• Rework effort
• SPI/CPI (planned vs. actual effort)

11
How Projects Use theOrganizational Process Assets
CMM
Organizational Standard Process Organizational
Procedures
Process Asset Library (PAL)
OrganizationalDatabase
OrganizationalTraining Office
Organizational Policies
Senior ManagementReview
Organization
Project
Project Schedules Budgets
Project Plans
Project Results
Project Defined Process Procedures
12
Project Comparative Data Base (PCDB)

• PCDB contains
• Cost data from a certified accounting system
• Project validated technical characteristics data
• PCDB relates
• Cost data to standardized (WBS) work elements
• Cost data to the technical characteristics of
  work performed
• The Office of Cost Estimation uses the PCDB to
• Calibrate the parametric models
• Derive cost estimating relationships
• Provide historical data for proposals, project
  planning/replanning
• Define risk affordability/analysis
• Characterize process performance and quality

13
What is the PCDB ?

• Accounting Data Project Inputs
• PCDB Standard WBS JN Mapping
• PCDB Standard WBS Alternate Hours/Cost Mapping
• Accounting Data
• Labor hours, dollar costs
• Non-labor costs
• Breakdown by PCDB WBS element, by labor category
• S/W Development Descriptive Data Examples
• SLOC, other size measures
• ESLOC (derived)
• Labor Required to Develop SLOC/ESLOC
• Software Development
• Other Project Disciplines
• Cost Model Parameters

• Accounting Descriptive Data
• Project Management
• Systems Engineering
• Hardware Development
• Software Development
• Systems Integration Test
• Site Activation
• Integrated Logistics
• Configuration Management
• Data/Documentation Management
• Quality Assurance
• Development Support Facility
• Operations and Maintenance
• Specialty Customer Services
• Other Activities

14
PCDB Support to Cost Modeling and Proposals
RFP
Project and Proposal Team Support
Functional Requirements
Final Estimates
First Order Estimates
Advanced Pricing System (APS)
Model Calibration
Project Comparative Data Base (PCDB)
Cost Volume
Sanity Check
Final Estimates
Project Technical Description Data Metrics
Certified Accounting Data
BOEs
Sanity Check
BOE Generation Support
Actuals Management Information System (AMIS)
Project Data
15
PCDB Data Submittal Process
No
Project Prepares and Submits Descriptive Data
Inputs
Yes
Project Prepares and Submits Accounting Data
Inputs
Proj OCE OK?
No
Yes
Perform Parametric Validation if Applicable
16
Parametric Validation/Calibration

• Additional selective validation for software
  development History Data Points
• Creation of parametric cost model baseline
• Input from Project
• Descriptive Data Forms Software Development
• OCE provides guidance and assistance
• Provided to Project
• Software Parametric Model (i.e., Costar (COCOMO
  II), SEER-SEM, Price S, or SLIM) cost estimation
  validation
• Iteration with project to achieve validation
  within 10 - 20 of actual effort
• Results in calibrated baseline for future
  estimates

17
Lessons Learned - 1

• Determining a common set of metrics is tough
  because there are different needs (and frequency,
  granularity, accuracy, etc.)
• Project management decisions
• Customer insight
• Senior management oversight
• Organizational process performance
  characterization
• Process improvement
• Proper support for metrics collection requires
  changing the culture to management-by-data
• Project managers must use the data to manage
  their projects
• Senior managers must use the data to meet
  organizational business objective
• Data collectors must be confident that data will
  not be used against them

18
Lessons Learned - 2

• Understanding variation in process performance
  allows more insight into estimation
• Whats the likely cost of this work?
• Whats the probability we can perform the work
  for ____?
• Project resistance to data collection is
  primarily due to the time and effort required to
  collect and report the data
• Must be integrated and consistent with the
  process
• Data collection mechanisms require clear
  instructions, to ensure the desired information
  is captured and validated
• Collection should be a combination of automated
  and manual methods for cost and accuracy

19
Systems Architecture Vision
Current Situation
Project Risk Assessment
Organizational Metrics Data
What if Modeling
Predictive Analysis
Financial Data
Project Risk Assessment
What if Modeling
Risk Radar
Project Mgmt Data
Decision Support Layer Organizational Metrics Data
Group Common Data Repository
Financial Data
Proposal Data
Project Mgmt Data
The End Objective
Integrate Metrics with Financial, Project,
proposal and other information to support trend
and risk analysis
Proposal Data
20
Conclusions

• The process database developed at Level 3 is a
  key asset in achieving Level 4
• The many uses of metrics places additional
  emphasis on innovative database design and usage
• Characterizing the organizations process
  performance requires
• Definitizing your business goals
• Selecting the right metrics
• Stabilizing the organization and projects
  processes
• Collecting and analyzing the metrics
• Management and decisions that are data driven
  result in better predictive analysis capability