Gary Stanley

1
Manufacturing Readiness Assessment Acquisition
Training
Gary Stanley Manufacturing Technology Division
AFRL/RXMT Phone [phone]gary.stanley_at_wpafb
.af.mil
Public Release Case 88ABW-2008-0329
2
Defense Policy Insertion PlansPolicy Targets

• DoD documents
• 1) Defense Acquisition Guidebook
• 2) DoDI 5000.2
• 3) DoDD 5000.1
• Air Force documents
• AFI 63-1201, Disciplined Systems Engineering
• Secy of the Air Force for Acquisition Policy Memo
  
• AFRL Advanced Technology Demonstrations
• Program Baseline Development
• AFMC Instruction 61-102, ATD Tech Transition
  Planning

3
What Will The Policy Look Like?

• MRLs are linked very closely with TRLs
• MRAs will be performed prior to each Milestone
  Decision
• M/S A MRL 4
• M/S B MRL 6
• M/S C MRL 8
• Full Rate Production MRL 9

4
PRR and MRA

• The areas evaluated in a PRR and MRA process are
  nearly the same
• Major differences
• Timing - PRRs usually occur in the SDD phase
  versus MRAs that occur throughout the Acquisition
  and ST Phases
• PRR will evaluate the total program's readiness
  to proceed into production vs MRA focusing on MRL
  ratings and MMPs
• PRR focuses on a total program risk assessment
  versus MRA providing an objective score on the
  manufacturing maturity of the program and how to
  achieve required MRLs
• Bottom Line if you have done a PRR, you can do
  an MRA
• Requires same skill base

5
Manufacturing Readiness Assessment Acquisition
Training
Jim Morgan Manufacturing Technology Division
AFRL/RXMT Phone [phone]jim.morgan_at_wpafb.a
f.mil
Public Release Case 88ABW-2008-0329
6
Session Outline

• What is a Manufacturing Readiness Assessment
  (MRA)?
• Why Manufacturing Readiness?
• What are Manufacturing Readiness Levels (MRLs)
  and how do they pertain to the Acquisition Life
  Cycle?
• How to do an MRA
• Sample Outputs and Deliverables
• Finding and Conclusions
• Additional Information

7
What Is A Manufacturing Readiness Assessment?

• An MRA is
• An Assessment of a Programs Readiness to
  Manufacture and Produce Its Intended Design
• A Tool to Develop and Implement -
• Manufacturing Risk Mitigation Plans
• Business Strategies
• Effects of Design Changes (Planned Upgrades,
  Spiral)
• Pricing Agreements (Long Term vs. Single Lot)
• Capital Investment Plans (Contractor and/or
  Government)
• An MRA
• Assigns Manufacturing Readiness Levels (MRLs) to
  Key System Components
• Analogous to Technology Readiness Levels (TRLs)

8
Session Outline

• What is a Manufacturing Readiness Assessment
  (MRA)?
• Why Manufacturing Readiness?
• What are Manufacturing Readiness Levels (MRLs)
  and how do they pertain to the Acquisition Life
  Cycle?
• How to do an MRA
• Sample Outputs and Deliverables
• Finding and Conclusions
• Additional Information

9
Why Manufacturing Readiness? Manufacturing
Industrial Base Challenge

• Consensus among Congress, OSD, CSAF, GAO
• Advanced weapon systems cost too much, take too
  long to field, and are too expensive to sustain
• GAO study of 54 weapons programs
• Core set of 26 programs RDTE costs up by 42
  and schedule slipped by 20
• 42.7B total cost growth
• 2.5 years slip on average
• Characteristics of successful programs (GAO)
• Mature technologies, stable designs, production
  processes in control
• ST organization responsible for maturing
  technologies, rather than program or product
  development manager
• To mitigate impact of diminishing manufacturing
  infrastructure
• People, policy, programs gutted
• Lost recipe on how to manage manufacturing risk
• Wont get infrastructure back but still need to
  manage manufacturing risk

10
Why Manufacturing Readiness? Acquisition Health
and Manufacturing Readiness

• Manufacturing risk/maturity is not the only
  cost/schedule/performance driver, but we need to
  manage manufacturing readiness integral to the
  overall acquisition process
• Products made by immature manufacturing processes
  generally
• Cost more
• Are prone to quality problems
• May not all perform the same
• Are less reliable in service
• Have a hard time delivering on schedule

11
Technology Readiness Levels (TRLs)

• Provide a common language and widely-understood
  standard for
• Assessing the performance maturity of a
  technology and plans for its future maturation
• Understanding the level of performance risk in
  trying to transition the technology into a weapon
  system application
• TRLs leave major transition questions unanswered
• Is the technology producible?
• What will these cost in production?
• Can these be made in a production environment?
• Are key materials and components available?

12
Session Outline

• What is a Manufacturing Readiness Assessment
  (MRA)?
• Why Manufacturing Readiness?
• What are Manufacturing Readiness Levels (MRLs)
  and how do they pertain to the Acquisition Life
  Cycle?
• How to do an MRA
• Sample Outputs and Deliverables
• Finding and Conclusions
• Additional Information

13
Manufacturing Readiness Levels(MRL)

• Common language and standard for
• Assessing the manufacturing maturity of a
  technology or product and plans for its future
  maturation
• Understanding the level of manufacturing risk in
  trying to produce a weapon system or transition
  the technology into a weapon system application
• Designed to help set the agenda for manufacturing
  risk mitigation

14
MRL Relationships
Relationship to System Acquisition Milestones
Production Deployment
System Development Demonstration
Technology Development
Concept Refine-ment
Pre-Concept Refinement
C
B
A
MRL 3 Mfg Proof of Concept Developed
MRL 4 Manufacturing Processes In Lab Environment
MRL 5 Components In Production Relevant
Environment
MRL 6 System or Subsystem In Production Relevant
Environment
MRL 7 System or Subsystem In Production Represent
ative Environment
MRL 8 Pilot Line Demonstrated Ready for LRIP
MRL 9 LRIP Demonstrated Ready for FRP
MRL 10 FRP Demonstrated Lean Production
Practices in place
MRL 2 Mfg Concepts Identified
MRL 1 Basic Mfg Implications Identified
TRL 8 System Qual
TRL 7 Prototype in Ops Environment
TRL 6 Prototype in Rep Environment
TRL 5 Breadboard in Rep Environment
TRL 4 Breadboard in Lab
TRL 3 Proof of Concept
TRL 2 Concept Formulation
TRL 1 Basic Principles Observed
TRL 9 Mission Proven
Relationship to Technology Readiness Levels
15
MRL Definitions

• Production relevant environment An environment
  normally found during MRL 5 and 6 that contains
  key elements of production realism not normally
  found in the laboratory environment (e.g. uses
  production personnel, materials or equipment or
  tooling, or process steps, or work instructions,
  stated cycle time, etc.). May occur in a
  laboratory or model shop if key elements or
  production realism are added.
• Production representative environment An
  environment normally found during MRL 7 (probably
  on the manufacturing floor) that contains most of
  the key elements (tooling, equipment,
  temperature, cleanliness, lighting, personnel
  skill levels, materials, work instructions, etc)
  that will be present in the shop floor production
  areas where low rate production will eventually
  take place.
• Pilot line environment An environment normally
  found during MRL 8 in a manufacturing floor
  production area that incorporates all of the key
  elements (equipment, personnel skill levels,
  materials, components, work instructions,
  tooling, etc.) required to produce production
  configuration items, subsystems or systems that
  meet design requirements in low rate production.
  To the maximum extent practical, the pilot line
  should utilize rate production processes.

16
9 MRL Evaluation Criteria(Threads)

• Technology and Industrial Base
• Technology maturity, technology transition to
  production, ManTech development
• Design
• Producibility program, design maturity
• Cost and Funding
• Production cost knowledge (cost modeling), cost
  analysis, mfg investment budget
• Materials (raw matls, components, subassys,
  subsystems)
• Maturity, availability, supply chain management,
  special handling
• Process Capability and Control
• Modeling Simulation (product process), mfg
  process maturity, process yields/rates
• Quality Management, to include supplier quality
• Manufacturing Personnel, to include
  specialization, training, certification
• Facilities, to include capacity and plant layout
  design
• Manufacturing Management
• Manufacturing planning and scheduling
• Materials planning
• Tooling and special test equipment

17
Milestone B Key Manufacturing Considerations
A

• Industrial Base capabilities identified for key
  technologies and key processes
• Producibility Manufacturability assessment of
  design concepts completed
• Establishment/validation of manufacturing
  capability and management of manufacturing risk
  for the product lifecycle
• Initial Key Performance Parameters (KPPs)
  identified
• Producibility cost risks assessed
• Survey completed to determine if materials have
  been used before
• Lead times identified for all materials
• Survey completed for potential supply chain
  sources
• Special handling requirements identified
• Survey completed to determine the current state
  of proposed processes
• Yield and Rates assessed on proposed processes
• Quality strategy developed
• Manufacturing skill sets identified
• Specialized facility requirements/needs
  identified
• Special Tooling/Special Test Equipment (STE)
  requirements are considered

18
Milestone B Key Manufacturing Considerations
B

• Industrial capability in place to support
  manufacturing of development articles
• Required manufacturing technology development
  solutions demonstrated in a production relevant
  environment
• Producibility assessments of key
  technologies/components and producibility trade
  studies completed
• Key Characteristics and tolerances established
• Lead times have been identified for all materials
• Cost model inputs include design requirements,
  material specifications, tolerances, integrated
  master schedule, results of system/subsystem
  simulations and production relevant
  demonstrations
• Material maturity verified through technology
  demonstration articles
• Availability issues addressed to meet technology
  demonstration articles
• Supply chain plans in place
• Plans to address special handling requirements
  complete
• Initial simulation models developed at the
  technology, sub-system or system level
• Manufacturing processes demonstrated in
  production relevant environment
• Yields and Rates evaluated in production relevant
  environment
• Initial Quality Plan and Quality Management
  System is in place
• Manufacturing workforce skills available for
  production in a relevant environment
• Manufacturing facility and facility development
  plans adequate to support SDD or Technology
  insertion
• Manufacturing risk mitigation approach for SDD or
  Technology insertion Programs defined.
• Most material decisions made (make/buy), material
  risk identified and plans made to mitigate
• Prototype tooling concepts demonstrated in
  relevant manufacturing environment.

19
Milestone C Key Manufacturing Considerations
C

• Industrial Capability Assessment (ICA) for MS C
  has been completed. Industrial capability is in
  place to support LRIP.
• Required manufacturing technology solutions
  validated on a pilot line
• Known producibility issues have been resolved and
  pose no significant risk for LRIP
• Detailed design of product features and
  interfaces is complete
• Major product design features are sufficiently
  stable such that key LRIP manufacturing processes
  will be representative of those used in FRP
• Engineering cost model driven by detailed design
  and validated with data from relevant environment
• Cost analysis of proposed changes to requirements
  or configuration
• Yields and Rates evaluated in production relevant
  environment
• Materials proven and validated on System
  Demonstration and Development (SDD) production as
  adequate to support LRIP
• Long Lead procurement initiated for LRIP.
  Availability issues pose no significant risk for
  LRIP
• Most material decisions made (make/buy), material
  risk identified and plans made to mitigate
• Prototype tooling concepts demonstrated in
  relevant mfg environment

20
Manufacturing Considerations forFull Rate
Production Decision (MRL 9)

• Industrial capability is in place to support
  start of FRP
• Producibility issues/risks discovered in LRIP
  have been mitigated and pose no significant risk
  for FRP
• Known producibility issues have been resolved and
  pose no significant risk for LRIP
• Major product design features are stable and LRIP
  produced items are proven in product testing
• Major product design features are sufficiently
  stable such that key LRIP manufacturing processes
  will be representative of those used in FRP
• Variability experiments conducted to show FRP
  impact and potential for continuous improvement
• Program has budget estimate for lean
  implementation during FRP
• Special handling procedures demonstrated in LRIP
• Manufacturing processes procedures are
  established and controlled in production to
  3-sigma or other appropriate quality level
• Yield and rate targets achieved, yield
  improvements on-going
• Quality targets verified on production line
• FRP personnel requirements identified
• Capacity plans adequate to support FRP decision
• All manufacturing risks have been validated and
  mitigated using LRIP articles
• All tooling, test and inspection equipment proven
  on LRIP

21
Session Outline

• What is a Manufacturing Readiness Assessment
  (MRA)?
• Why Manufacturing Readiness?
• What are Manufacturing Readiness Levels (MRLs)
  and how do they pertain to the Acquisition Life
  Cycle?
• How to do an MRA
• Sample Outputs and Deliverables
• Finding and Conclusions
• Additional Information

22
MR Assessment Process

• Assessment Lead Briefs PM on Manufacturing
  Assessment Efforts/Expectations
• Works with PM to
• Determine appropriate level for Manufacturing
  Readiness Assessment(s) (MRAs) -- System may
  contain several critical technologies/components/m
  anufacturing cells
• Schedule on-site MRA with contractor(s)
• Send Orientation Package to contractor(s)
• Define Assessment Team Membership
• Define Deliverables of Assessment Results
• Conduct on-site assessment with contractor(s)
• Deliver final report/briefing

23
Manufacturing Readiness ImplementationApproach
(ACATs)
INTRODUCE
Meet with Wing/Program Management Team And Other
Stakeholders
TRAIN

• Define Objectives
• - Yield Improvement
• New Variant (e.g. Spiral)
• Increased Capacity (Surge)

OBJECTIVE STATEMENT DEFINED
ASSESS

• Decompose the Problem Space
• By Technology (i.e. Component)
• By Supplier
• - Handle Assembly Test

INCORPORATE
Wing/PM Team owns the plan
MANAGE
24
Preparations

• Contact Defense Contract Management Agency (DCMA)
  to gather information on the contractors current
  and past performance.
• Notify companies and send orientation package
• Purpose, approach, questions, strawman agenda
• MRL definitions/threads
• Address contract issues if any
• Self-Assessment
• Select Assessment team(s)
• Typically 2-6 members per team
• Appropriate members (include Govt customer)
• Specialists for key technologies (if needed)
• Schedule On-site assessments
• Months prior to key milestone decisions to
  establish a baseline and allow time to
  develop/implement risk mitigation plans
• Team OrientationMeet prior to on-site assessment

25
On-site MRA Process Review

• Contractor welcome, review of agenda and
  orientation to facility
• Introduction of assessment team and contractor
  personnel
• Government team lead briefing to contractor
  describing objectives and expectations for the
  on-site visit
• Contractor overview and discussion of the results
  of their self-assessment
• Shop-floor visits to key areas by individuals or
  small groups
• One-on-one or small group discussions between
  assessment team members and contractor subject
  matter experts focused on key areas
• Private meeting of Government assessment team to
• Prepare feedback and identify any action items
• Initial assessment of current MRL (their area or
  overall)
• Key strengths/risks/issues
• Key missing data (if any)
• Proposed action items
• Outbriefing by Government team to contractor

26
Example Process FlowGeneric Aircraft

• Large programs can require multiple MRAs

Deliver to Facility B
Install Fuselage Fuel System
Install Cables
Composite Fabrication
Composite Assembly
Install Oil System
Assemble Tails
Install Avionics
Engine Delivery
Cable Fabrication
INS Delivery
Landing Gear Build-up
Engine Build-up
Deliver to Facility C
Initial Power-up Checks
Install Brake System
Install Landing Gear
Install Power plant
Assemble Wings
Install Wings/Tails
Install Engine Cowls
Move A/C To Test
Colors represent supplier/facility location
System Tests
Final Inspection
27
Supplier MRA Plan

• Identify and prioritize critical suppliers
• Develop common SOW for distribution to suppliers
• Scope of MRA detailed
• Method of MRA detailed
• Output defined
• Developed detailed MRA execution plan with each
  supplier (Schedule, format, personnel)
• Execute MRA
• Define/Plan/Execute MRL mitigations
• Measure mitigation effectiveness, update
  assessment

28
Follow-on Activities

• Gather any key missing data
• Convene team meeting -- Typically within 2 weeks
  of on-site assessment
• Discuss and finalize assessment
• Examine current program and manufacturing risk
  reduction plans
• Agree on likely MRL at completion of milestone if
  current plan is followed
• Share results with contractor
• Identify the specific risk reduction activities
  necessary to reach the next milestone
• Identify the funding, time-phasing and approach
  to carrying out each activity
• Prepare and submit final report

29
Session Outline

• What is a Manufacturing Readiness Assessment
  (MRA)?
• Why Manufacturing Readiness?
• What are Manufacturing Readiness Levels (MRLs)
  and how do they pertain to the Acquisition Life
  Cycle?
• How to do an MRA
• Sample Outputs and Deliverables
• Finding and Conclusions
• Additional Information

30
MRA Process Outputs

• Baseline Manufacturing Readiness Levels (MRL)
• Key Factors Where Manufacturing Readiness Falls
  Short Of Target
• Define MRL Driving Issues
• Identify Programs And Plans To Reach Target MRL
• Identify Existing Investments and Additional
  Needs
• Summarize Improvement In Manufacturing Plan
• Assess Risk to Manufacturing Cost, Schedule, and
  Performance
• Implement and Execute the Manufacturing Plan
• Assess Effectiveness Of the Manufacturing Plan
• Address Right Issues?
• Timely? Adequately Funded?
• Probability Of Success?
• Options For Increased Effectiveness

31
SAMPLE SUMMARY ROLL-UP OF COMPONENTS
SubSystem Top Level MRL Observations Most Critical
Guidance 3 Lacking detailed process information Key suppliers identified Need key performance parameters Need detailed process plans Detector from supplier A Design production issues No alternate source
Data Processor 3 New processor architecture Immature design tools New attachment processes needed Board Supplier cant test at their site Low yields on initial run
Propulsion 6 Same as other systems in use New component scheme Re-validate manufacturing process Supplier handle increased rate
Air Vehicle 7 Same supplier as system X Need to test new mating and assembly processes at the prime No critical items
Test Plan 6 Several instances of re-design work and new test processes New test strategy and plan What will new design incorporate? Manufacturing experience vital
32
SAMPLE SUMMARY (Drill down)
Guidance Sub systems Top Level MRL Observations Most Critical
Front End Sensor 3 Lacking details on builds Process procedures need more work Test and assembly procedures have not been verified in manufacturing environment Detector from supplier A Design production issues No alternate source
Data Processing PWB 3 New processor architecture Awaiting Design for Manufacturing and Assembly (DFMA) results Low yields on initial build Working process controls Looking at re-design for easier fabrication
Cables For Power Data 3 Using same suppliers other weapon systems Have not received prototypes, awaiting supplier delivery Re-validate manufacturing process as seen on past programs Need new process plan
Housing 4 New supplier limited experience Need new assembly processes at the prime Need supplier management process need new process plans
Cooling 3 - Form, fit factors for new cooling design not in place Initial process plan for build in place Final cooling plan will be defined after front end is stable
Integration Process that includes assembly and test 3 - Several new test processes need development for new components New test strategy and plan New special test equipment must be ordered
33
MRA Risk Management

• Assessing Risk is independent of the MRL value
  assigned
• Higher MRL value may be highest risk
• Eg. Requires new equipment, high cost process
• Risk Assessment should consider
• Time needed to reach target MRL
• Require new personnel, training, capital, or more
  POM samples to flush out the process
• Leverage other programs
• Captive or Merchant Supplier Dependency??
• Part of a companys core business
• Leads into an industrial base assessment
• Effective of use of Design for Manufacturing
  Tools and other simulation techniques.

34
Session Outline

• What is a Manufacturing Readiness Assessment
  (MRA)?
• Why Manufacturing Readiness?
• What are Manufacturing Readiness Levels (MRLs)
  and how do they pertain to the Acquisition Life
  Cycle?
• How to do an MRA
• Sample Outputs and Deliverables
• Finding and Conclusions
• Additional Information

35
AFRL/RXM MRA Deskbook

• The how-to of MRAs
• First draft completed in March 07
• Modeled after TRA Deskbook
• Similarities
• Achieving levels of readiness for risk reduction
• Selection process for assessment areas
• Differences
• Readiness in ST and Acquisition world
• Rigorous assessment process
• Draft revised based on lessons learned from
  Reaper MRA
• Dec 07, Public releasable, on DAU website

36
Some MRL Thoughts

• MRLs are not a report card
• MRL 7 might not be good
• MRL 3 might not be bad
• MRLs are a tool to manage and mitigate
  manufacturing risk
• A common language used to assess manufacturing
  maturity
• Provide insight not oversight

37
Some MRA Lessons Learned

• Process is more effective if company is actively
  engaged in the assessment
• System integration and test operations are often
  ripe for maturation efforts
• Resources required to conduct an MRA will vary
  significantly
• Subject matter expertise is needed to do it
  right

38
Findings and Conclusions

• Looking at transitioning technology to production
  
• Must incentivize good decision processes
• Unlike TRLs, going backwards on MRLs might be a
  good thing
• A low MRL number may be ok
• Is there time to raise the level?
• Is there a new manufacturing process being
  pursued?
• Replacing a manual process with an automated
  process
• Encouraging repeatability, faster cycle time,
  etc.
• Identify opportunities to validate manufacturing
  processes
• Avoid accepting analogous process claims during
  the design phase and claiming fabrication is
  maturing
• May never build enough units to reach MRL 10
• Achieve a six sigma or equivalent process

39
MRA Thoughts

• MRA process highlights areas needing attention to
  lower production risk
• Easy Tracking for Prime contractor and Government
  as manufacturing proceeds
• Detailed analysis rank ordered Can be an
  investment strategy
• Accomplished the goal as an acquisition test case

Manufacturing maturity through the MRA process
enables efficient, cost effective manufacturing
40
Session Outline

• What is a Manufacturing Readiness Assessment
  (MRA)?
• Why Manufacturing Readiness?
• What are Manufacturing Readiness Levels (MRLs)
  and how do they pertain to the Acquisition Life
  Cycle?
• How to do an MRA
• Sample Outputs and Deliverables
• Finding and Conclusions
• Additional Information

41
Additional Information

• DoD MRL web site
• http//www.dodmrl.com
• - MRA link to DAU CoP
• - Contains MRL definitions, MRL criteria
  matrix, MRA Deskbook
• and more
• TRA Deskbook
• http//www.dod.mil/ddre/doc/tra_deskbook.p
  df
• - MRLs contained in Appendix I
• DAU PQM Community of Practice
• https//acc.dau.mil/pqm
• - Manufacturing Readiness folder
• Look for MR definitions
• Look for MR matrix
• Look for MRL tutorial
• Look for MRA Deskbook