Software Independent Verification and Validation (IV

1
Software Independent Verification and
Validation(IVV)NASA IVV FacilityFairmont,
West VirginiaJudith N. BrunerActing
Director304-367-8202judith.n.bruner.1_at_gsfc.nasa.
gov
2
Content

• Why are we discussing IVV?
• What is IVV?
• How is IVV done?
• IVV process
• Why perform IVV?
• Summary
• Points of Contact

3
Why are we discussing IVV?

4
Setting the Stage

• In the 90s, the Commanding General of theArmys
  Operational Test and Evaluation Agency noted that
  90 percent of systems that were not ready for
  scheduled operational tests had been delayed by
  immature software.

5
Software Chaos
The Standish Group examined 8,380 Software
Projects.
16 Successful
53 Challenged
- In Budget
-
Over budget
by 189
- On Time
- Late by 222
- Meets Requirements
- Missing 39 of Capabilities
- User involved
31 Cancelled
Note For Large Companies
- Development
- 9 were Successful
- 61.5 Challenged
-
Over budget
by 178
- Late by 230
- Missing 58 of Capabilities
- 29.5 were Cancelled
6
Error Densities
Design Implementation
23
Requirements Specification
Installation Commissioning
68
9
7
Increasing Cost of Changes
The cost to correct an software error multiplies
during the development lifecycle.
Cost scale factor
(Normalized to Requirements Phase)

8
What is IVV?

9
Independent Verification and Validation (IVV)

• Independent
• Technical IVV prioritizes its own efforts
• Managerial Independent reporting route to
  Program Management
• Financial Budget is allocated by program and
  controlled at high level such that IVV
  effectiveness is not compromised
• Verification (Are we building the product
  right?)
• The process of determining whether or not the
  products of a given phase of the software
  development cycle fulfill the requirements
  established during the previous phase
• Is internally complete, consistent and correct
  enough to support next phase
• Validation (Are we building the right product?)
• The process of evaluating software throughout its
  development process to ensure compliance with
  software requirements. This process ensures
• Expected behavior when subjected to anticipated
  events
• No unexpected behavior when subjected to
  unanticipated events
• System performs to the customers expectations
  under all operational conditions

10
Independent Verification Validation

• Software IVV is a systems engineering process
  employing rigorous methodologies for evaluating
  the correctness and quality of the software
  product throughout the software life cycle
• Adapted to characteristics of the
  target program

11
How is IVV done?

12
IVV Activities Throughout Lifecycle
13
IVV Life Cycle Functions

• IVV Process provides tools and analysis
  procedures appropriate to each phase of the
  software development life cycle
• Formulation Phase
• Is development process sound, repeatable, and
  managed?
• Requirements Phase
• Verify that system and software requirements are
  correct, complete, traceable and testable
• Analyze system-level requirements Are test
  plans and acceptance criteria sufficient to
  validate system requirements and operational
  needs?
• Are testing methods sufficient to verify and
  validate software requirements?
• Are the correct software development, management,
  and support processes in place?
• Design Phase
• Does the design support the requirements?
• Are test plans and test environments sufficient
  to verify and validate software and operational
  requirements?
• Does the design have any characteristics that
  will cause it to fail under operational
  scenarios? What solutions are appropriate?

14
IVV Life Cycle Functions (cont.)

• Typical IVV functions by Software life-cycle
  phase (cont.)
• Coding Phase
• Does the code reflect the design?
• Is the code correct?
• Verify that test cases trace to and cover
  software requirements and operational needs
• Verify that software test cases, expected
  results, and evaluation criteria fully meet
  testing objectives
• Analyze selected code unit test plans and results
  to verify full coverage of logic paths, range of
  input conditions, error handling, etc.
• Test Phase
• Analyze correct dispositioning of software test
  anomalies
• Validate software test results versus acceptance
  criteria
• Verify tracing and successful completion of all
  software test objectives
• Operational Phase
• Verify that regression tests are sufficient to
  identify adverse impacts of changes

15
IVV Testing Involvement

• IVV identifies deficiencies in programs test
  planning
• Program changes their procedures to address
  deficiencies vice IVV independently test
• IVV may independently test highly critical
  software using an IVV testbed
• Whitebox
• Stress
• Endurance
• Limit
• Developer motivated to show software works
• IVV attempts to break software

16
IVV Process

17
IVV Process
Integrates IVV into program Provides IVV
funding Resolves Exception issues
Reflects IVV in program mgmt plan Agrees to data
transfer plan Reflects agreement in subcontracts
IVV in phase with development
18
IVV Scope

• Scope is determined so as to minimize the risk
  within the Programs IVV budget. Effort is
  based on
• Criticality and risk of system functions
  performed/managed by software
• Budget limitations

Programs IVV budget
19
CARA Scoring Methodology
20
CARA Criticality
Sample Criticality Evaluation Criteria
21
CARA Risk
Sample Risk Driver Criteria
22
Requirements Analysis IALs
23
Design Analysis IALs
24
Code Analysis IALs
25
Test Analysis IALs
26
IVV Is Process As Well As Product Oriented
27
IVV Increases Program Awareness
28
Staffing Paradigm
29
Why perform IVV?

30
IVV Benefits
Technical
Management
31
Summary

32
IVV Key Points

• IVV works with the Project
• Goal is project success
• IVV is an engineering discipline
• IVV processes are defined and tailored to the
  specific program
• Mission, operations and systems knowledge is used
  to perform engineering analyses of system
  components
• IVV is most effective when started early
• 70 of errors found in testing are traceable to
  problems in the requirements and design
• IVV works problems at the lowest possible level
• Primarily work via established informal
  interfaces with the development organization -
  working groups, IPTs, etc.
• Elevate issues only when necessary

33
IVV Approach Efficiently Mitigates Risk

• It is not necessary or feasible to perform all
  IVV analyses on all software functions
• IVV resources allocated to reduce overall
  exposure to operational, development, and
  cost/schedule risks
• Software functions with higher cirticality and
  development risk receive enhanced levels of
  analysis (CARA process)
• Systems analyses performed to reduce costly
  interface and integration problems
• Process analyses performed to verify ability to
  produce desired result relative to program plans,
  needs and goals
• IVV working interfaces promote timely problem
  resolution
• Proactive participation on pertinent development
  teams
• Emphasis on early identification of technical
  problems
• Engineering recommendations provided to expedite
  solution development and implementation

34
Analyses Are Value Added and Complementary- Not
Duplicative

• Analyses performed from a systems perspective
  considering mission needs and system use, hazards
  and interfaces
• Discipline experts assigned to perform analysis
  across all life cycle phases
• Horizontal specialty skills are matrixed across
  IVV functional teams to verify correct systems
  integration
• Specialized tools and simulations perform complex
  analyses
• IVV testing activities complement developer
  testing enhancing overall software confidence
• Developer testing focuses on demonstrating
  nominal behavior, IVV testing activities try to
  break the software
• Overall program integration, test and
  verification approach analyzed for completeness,
  integrity and effectiveness

35
Why use NASA IVV Facility?

• Software IVV, as practiced by the NASA Software
  IVV Facility, is a well-defined, proven, systems
  engineering discipline designed to reduce the
  risk in major software developments.

36
NASA IVV FacilityPoints of Contact

• Judy Bruner
• Acting Director
• 304-367-8202
• judith.n.bruner.1_at_gsfc.nasa.gov
• Bill Jackson
• Deputy Director
• 304-367-8215
• bill.jackson_at_ivv.nasa.gov