Applying Systems Engineering SYSE 511

1
Applying Systems EngineeringSYSE 511

• Module 10 Verification and Validation Testing

2
Objectives

• Understand Test Verification and Validation
• Define System Verification and where it fits into
  the Systems Engineering process
• Understand the Methods of Verification
• Understand Technical Reviews
• Review the Verification Timeline
• Chapter 6 in Text book

3
Where are we in theSE Process?

• Objectives
• Test and demonstrate possible solutions
• Test, inspect, simulate, etc. the physical
  architecture of preferred system solutions
• Identify and resolve any non-compliance with
  requirements

4
The ProblemCorrection Costs
108
106
FCA/ PCA
Cost of Fixing Problems
ATR
104
TRR
102
CDR
PDR
SCDR
SRR
100
Reqts
Concept Design
Title Design
Test
Build
Production
System Life Cycle
5
System Verification Planning Resource Allocation
6
Key Terminology

• Analysis
• Test and Demonstration
• Inspection
• Verification and Validation
• Critical Design Review (CDR)
• Preliminary Design Review (PDR)
• System Requirements Review (SRR)
• Test and Evaluation Master Plan (TEMP)
• Acceptance Testing
• Alpha and Beta Testing

7
What is System Verification?
System Verification is a testing process which
ensures that the selected solution meets
specified requirements and properly integrates
with interfacing products. Martin, p. 20
8
Verification (INCOSE Handbook)

• Verification is the confirmation that the model
  does what it was designed to do.
• A proper development process will have direct
  correspondence between the definition/decompositio
  n activities and the integration/verification
  activities. For every specification, there should
  be an independent integration and verification
  activity. For every level in the hierarchical
  decomposition, there should be an independent
  level of integration and verification.
• Verification tasks are test, analysis,
  inspection, demonstration, or simulation also
  includes manufacturing process proofing, quality
  assurance and acceptance, and development test
  and evaluation (DTE).
• Not INCOSE Plain and simple, if you buy
  something, you should have a plan to test that
  what you receive is what you ordered. I bought
  1GB DRR Memory, I get two strips. I plug it in
  and start the computer. I open
  MyComputer/Properties/General. It says there is
  1GB RAM. At least my computer thinks there is
  1GB. Im happy.

9
Validation (INCOSE Handbook)

• Validation confirms that the system, as built (or
  as it will be built), will satisfy the users
  needs, and,
• Validation.
• 1. The determination that the requirements for
  a product are sufficiently correct and complete.
  (ARP 4754, 1996) 
• 2. The effort required of the contractor or
  preparing activity, during which the technical
  data product is tested for technical adequacy,
  accuracy, and compliance with the provisions of
  the specifications and other contractual
  requirements. Validation is accomplished by
  comparing the data product with the actual
  systems or equipment for which the data product
  was prepared. Validation is normally conducted at
  the preparing activity or venders facility. In
  extenuating circumstances, validation may be
  conducted at an alternative site. (MIL-HDBK-59A,
  App A, Para 30.8.6)

Recall Functional, complete, correct, accurate,
traceable, verifiable
10
Methods of Verification

• Inspection
• Analysis
• Demonstration (Simulation)
• Testing

Requirements Verification Matrix
11
System Verification Through Testing
FMA Failure Modes Analysis FRACAS Failure
Reporting, Analysis and Corrective Action System
12
Verification Validation Examples

• You buy a new printer for your computer.
• How would you Verify it?
• How would you Validate it?
• Your ski boat goes 44 mph with your current
  aluminum propeller. You buy a new SS prop that
  promises to go 52 mph. You buy a a new stainless
  steel Raptureª Michigan BT25146, propeller . A
  highly raked, progressive blade pitch provides a
  perfect blend of acceleration, bow lift and top
  end speed. Cupped trailing blade edge grabs water
  cleanly and pushes it back with minimal drag or
  energy loss. A cushioned rubber hub mounted in a
  ribbed cavity provides maximum bushing grip and
  protects your shaft, motor and gearbox.
• How would you Verify it?
• How would you Validate it?
• Suppose instead that you are the Michigan design
  engineer. How might you test it, i.e., Verify
  and or Validate your design?       

13
System Verification Through Technical Reviews

• System Requirements Review (SRR)
• System Conceptual Design Review (SCDR)
• Preliminary Design Review (PDR) or Title I (30)
• Critical Design Review (CDR) or Title II (90)
• Test Readiness Review (TRR)
• Formal Qualification Review (FQR) or Acceptance
  Test Review (ATR)
• Operational Readiness Review
• Functional Configuration Audit (FCA)
• Physical Configuration Audit (PCA)

14
Technical Review Team Members

• Project Management
• Systems Engineering
• Technical Specialists
• Manufacturing
• Quality Assurance
• Finance
• Suppliers
• Customers

15
Timing of Reviews
108
106
FCA/ PCA
Cost of Fixing Problems
ATR
104
TRR
102
CDR
PDR
SCDR
SRR
100
Reqts
Concept Design
Title Design
Test
Build
Production
System Life Cycle
16
How are Technical Reviews Performed?

• Checklists
• Peer Reviews
• Components
• References
• Slowly and deliberately.
• Avoid mass meeting reviews (the standard
  approach.)
• NRC budgets 10 pages per day per man.

17
Review Techniques

• Review for
• Complete
• Accurate
• Functional
• Traceable
• Verifiable
• Consistent
• Readable
• Pick a key topic and work the problem yourself.
• Check a reference document find it, study it.
• Pull the thread pick a promise or conclusion,
  This document will prove, It is recommended
  that and then try to validate that statement
  based on what is in the document.
• If you find a virgule (/), parenthetical
  expression (), quotation () test it.
• With an electronic document, pick a word, phrase,
  or acronym, follow it throughout the document.
• Read a paragraph and try to explain it to a
  co-worker.

18
Verification Planning

• Verify and Validate Requirements and Solution
  Performance
• Use preplanned Peer Reviews and Inspections
• Use Models to Demo Feasibility
• Explicitly identify and check all model
  assumptions
• Plan the verification for every requirement
  (perhaps there are too many then what?)
• Predict the results before testing.
• Include success/fail criteria

19
Test Plans, Procedures, Reports and MTEMP (TEMP)

• See Army TEMP Guide Pamphlet 73-2
• 64 page guideline.
• Basically Plan everything.
• See http//www.acomtech.com/testplan.html for a
  set of test plan, test specification and test
  report outlines, per IEEE
• See http//www.southwestern.edu/bucheles/SE/VVPla
  n-outline.html for a partially annotated outline

20
Types of Verification and Validation or Testing

• Tests vary from initial theoretical
  considerations to re-engineering
• Type of tests depend on customer objectives
• DOE and DOD budget for and manage Basic Science
  studies, developmental programs, project
  development, and then the gambit of tests and VV
  throughout a program or project
• What one does depends on the Plan again, it
  begins with a problem definition, mission needs,
  requirements, functional analysis, architecture,
  etc.
• Discuss various development programs in these
  terms ONR, DARPA (Defense Advanced Research
  Projects Agency), NASA, DOE

21
Development Testing

• Recall the DOD acquisition model.
• Demonstrate feasibility of concepts
• Testing of concepts to determine engineering
  parameters.
• Testing to find improvements.

22
The 5000 Model
Technology Opportunities User Needs

• Process entry at Milestones A, B, or C
• Entrance criteria met before entering phases
• Evolutionary Acquisition or Single Step to Full
  Capability

B
A
C
IOC
FOC
Concept Exploration
Technology Development
System Integration
System Demonstration
LRIP
Full-Rate Prod Deployment
Sustainment
Disposal
Critical Design Review
FRP Decision Review


Concept Tech Development
System Development Demonstration
Operations Support
Production Deployment
Sustainment
Pre-Systems Acquisition
Systems Acquisition (Demonstration, Engineering
Development, LRIP Production)
IOC Initial Operational Capability FOC Full
Operational Capability
Validated approved by operational validation
authority
Initial Capability Document (ICD)
Capability Development Document (CDD)
Capability Production Document (CPD)
Relationship to Requirements Process
23
Acceptance Test Procedure

• A planned sequence of operations to demonstrate
  to the customer that the facility is ready to
  turn over to operations.
• Classic Title Design final test.
• In a residential or business building
  construction project, this is typically the
  walk-through.
• In automobile sale this is the drive-around-the-bl
  ock.

24
Operational Readiness Review

• A NASA construct adopted by DOE
• Since DOE owns the facility, and the MO operates
  the facility, the DOE is responsible for
  accidents.
• Therefore, DOE wants detail assurance that the
  plant is safe to operate prior to initiating
  start up.
• The higher the hazard level (category) the
  greater the detail of the ORR.
• See also
• http//books.usapa.belvoir.army.mil/cgi-bin/bookmg
  r/books/p73_5/CONTENTS6.4
• DOE ORR Standard

25
Phased Start Up

• THORP Thermal Oxide Reprocessing Plant in
  England. Last nuclear reprocessing plant to be
  built. Extreme safety issues.
• Carefully planned phased start up to ensure that
  plant operated as intended before adding the most
  hazardous material.
• One definition required GWd/tU, a.k.a.
  MWD/MTIHM, (Giga or Mega Watt Day per Metric ton
  Irradiated heavy metal (Uranium), is a measure of
  the nuclear fuel burnup, hence a measure of the
  amount of fission products, and hence a measure
  of the hazard of the spent nuclear fuel.

26
(No Transcript)
27
(No Transcript)
28
Alpha and Beta Testing

• Page 143 Text
• Alpha testing is product use in-house users
• Beta testing is product testing by a select set
  of friendly external users
• Alpha and beta testing is applicable to many of a
  kind, of consumer products

29
Objectives Review

• Define System Verification and where it fits into
  the Systems Engineering process
• Understand the Methods of Verification
• Understand Technical Reviews
• Review the Verification Timeline

30
Boeing 777 A Video Illustration if there is time