28 October

1
28 October

• Software Quality and Testing

2
Why do we care?

• Therac-25 (1985)
• Multiple space fiascos (1990s)
• Ariane V exploded after 40 seconds (conversion)
• Mars Pathfinder computer kept turning itself off
  (system timing)
• Patriot missile misquided (floating point
  accuracy)
• Millenium bug (2000)
• Microsoft attacks (2003)
• NIST cost to US, 59 billion

3
References

• Therac-25 http//courses.cs.vt.edu/cs3604/lib/Th
  erac_25/Therac_1.html
• Patriot missile http//www.fas.org/spp/starwars/g
  ao/im92026.htm
• Ariane 5 http//www.esa.int/export/esaCP/Pr_33_19
  96_p_EN.html
• NIST
• http//www.nist.gov/director/prog-ofc/report02-3.
  pdf

4
Quality and testing

• Errors should be found and fixed as close to
  their place of origin as possible. Fagan
• Trying to improve quality by increasing testing
  is like trying to lose weight by weighing
  yourself more often. McConnell

5
Tools for Improving Quality

• Reviews and inspections
• Formal specification
• Program verification and validation
• Self-checking (paranoid) code
• Testing
• Deploy with capabilities to repair

6
Reviews and Inspections

• Why?
• Developer cant correct unseen errors
• More eyes to catch problems
• Earlier is cheaper
• Integration fix typically 3-10 times the cost at
  design
• Difference in terms
• Review implies completed work, often reviewed by
  someone at a different level
• Inspection implies peer review of work in progress

7
Software Inspections

• Disciplined engineering practice for detecting
  and correcting defects
• Introduced at IBM by Fagan in the 1970s
• More formal than walkthroughs or peer reviews
• Roles, statistics
• Used for specs, code, test plans,

8
Uses

• Early detection of errors
• Major escapes cost 2-10 times as much minor 2-4
• Identification of excellence indicators
• Completeness (requirements to code)
• Correctness (specification to code)
• Style (consistency)
• Exit criteria for life cycle phases

9
Additional Benefits

• Programmer finds errors and types of errors that
  he is apt to make immediately
• Awareness means focus on those types of errors
  and therefore improved skills
• Designers get feedback on quality of their
  designs
• Using statistical anomalies to recode

10
Why do inspections work?

• More eyes
• Focused activity
• Structure
• Timely
• Measurable criteria for passing and rework
• Required follow-up

11
Why Arent Inspections Used?

• Rigorous and formal (requires training)
• Time consuming
• 4-5 people over multiple 2 hour sessions
• 250-500 lines of code per hour
• 5-10 errors detected per session
• Boring, low tech
• Egos

12
References

• Fagan, Design and code inspections to reduce
  errors in program development, IBM Systems
  Journal (reprinted 99)
• Porter, Siy and Votta, A Review of Software
  Inspections, 1995

13
Formal Methods and Specifications

• Mathematically-based techniques for describing
  system properties
• Used to show completeness, consistency,
  unambiguity
• Able to be used without executing the program
  (inference systems)

14
Inference Systems

• Proving something about the specification not
  already stated
• Formal proofs
• Mechanizable
• Examples theorem provers and proof checkers

15
Uses of Specifications

• Requirements analysis
• rigor
• System design
• Decomposition, interfaces
• Verification
• Specific sections
• Documentation
• System analysis and evaluation
• Reference point, uncovering bugs

16
Properties of Specifications

• Unambiguous
• Maps to a single specificand set
• Consistency
• Maps to a non-empty specificand set
• Completeness
• Not required!
• Balance between underspecification and
  overspecification

17
Examples of Specification Languages

• Abstract data types
• Algebras, theories, and programs
• VDM (Praxis UK Civil aviation display system
  CDIS), Z (Oxford and IBM CICS), Larch (MIT)
• Concurrent and distributed systems
• State or event sequences, transitions
• Hoares CSP, Transition axioms, Lamports
  Temporal Logic
• Programming languages!

18
References

• J.M. Wing, A Specifier's Introduction to Formal
  Methods. IEEE Computer, 23(9)8-24, September
  1990.
• Clarke et al, Formal methods state of the art
  and future directions, ACM Computing Surveys,
  28(4) 626--643, 1996.

19
Life Testing

• Used regularly in hardware
• Addresses normal use
• n specimens put to test
• Test until r failures have been observed
• Choose n and r to obtain the desired statistical
  errors
• As r and n increase, statistical errors decrease
• Expected time in test mu0 (r / n)
• Where mu0 mean failure time

20
Butler and Finelli

• The Infeasibility of Experimental Quantification
  of Life-Critical Software Reliability
• In order to establish that the probability of
  failure of software is less than 10-9 in 10
  hours, testing required with one computer is
  greater than 1 million years

21
Types of Testing Purpose

• Conformance testing
• Usability testing
• Performance testing
• Acceptance testing
• Reliability testing

22
Other classifications

• Scope
• Unit, component, system, regression,
• Access to code
• Black box vs. white box
• (Note that black box testing still assumes
  knowledge of coding and development in general)