Requirement Discovery during the Testing of Safety-Critical Software

1
Requirement Discovery duringthe Testing of
Safety-Critical Software

• Robyn R. Lutz
• Inés Carmen Milkulski

2

• Purpose
• Introduction
• Approach
• Results and Analysis
• Conclusion
• Relevance To course
• Comment

3
Purpose

• To understand how requirements discovery
  occurs during testing and how such discoveries
  are resolved ( or not resolved ) prior to
  deployment.
• Four Mechanisms of Requirement discovery and
  resolutions are identified in the paper

4
Introduction

• Difficulties with requirements have been
  implicated as the main source of testing defects
  in any software development process.
• The same is true for Safety critical systems due
  to the complexity and risks involved in them.

5
Approach

• 171 Problem reports (PRs) are collected and
  analyzed during integration and system testing of
  the Mars Exploration Rovers (MER)
• - MER software 300K LOC
• implementing of more than
• 400 software requirements

6

• Orthogonal Defect Classification (ODC) (Same as
  Medical labs)
• - Activity
• - Trigger
• - Target
• - Type

7
Integration Test
System Test
Fig1 Types of Corrections for Testing Reports
8
Results And Analysis

• Four Mechanisms of Requirement discovery and
  resolutions are identified in the paper
• 1/ Incomplete requirements, resolved by
  changes to the software ex. Interface issues

9

• - 65 of 171 resolved by a change to the flight
  software, Target Flight Software.
• 23 of 65 had Type Assign/Init
• ex. max ?
• Another 23 had Type Func / Algor
• 10 of 23 was resolved by requiring new
  function.
• ex. parameter validation checks,
• checks on pre and post conditions.

10

• 2/ Unexpected requirements interaction,
  resolved by changes to operational procedures
• ex. sequencing of activities
• (first calibrate before use)
• 13 PRs has an ODC Type Information
  Development and ODC Type missing or incomplete
  procedures
• - special case of first mechanism

11

• 3/ Requirements confusion by the testers,
  resolved by changes to the documentation
• ex. Erroneous requirements assumptions by
  the testers
• 14 of 171 PRs were resolved by changes to
  the documentation.
• Target Info. Development and
• Type Documentation.

12

• 4/ Requirements confusion by the testers,
  resolved by a determination that no change was
  needed
• -similar to the previous one
• -no fix was made
• ex. false positives , incorrect
  assumptions about the effect of specific commands
  on the system
• 30 of 171 PRs had Target Unknown
• and Type Nothing fixed

13
Fig2 Fixing Testing Problem Reports
14
Conclusion

• Four common mechanisms for requirements discovery
  and resolution during integration and system
  testing of a safety critical software system.
• And these discoveries can be better used to
  reduce future requirements anomalies during
  operation by changing either the software ,
  operational procedure or proper documentation and
  training.

15
Relevance To Course

• The ideas in the paper is relevant to the course
  in that difficulties with requirements are the
  source of defects not only in testing but also in
  other phases of the software development process.
  

16
Comment

• Requirements discovery and resolutions during
  testing gives significant insights on what would
  happen during operation of safety critical
  systems. To avoid operational anomalies ,
  requirements confusion and false positives
  during testing should be properly documented
  regardless of availability of project resources
  as a mishap in operation could cause loss of
  lives.

17
Questions?