Dimensions of Testing

1
Dimensions of Testing 2

• EE599
• Software VV
• Winter 2006
• Diane Kelly Terry Shepard

2
Dimensions of Testing Part 2
3
 Purposes of testing ...

• Why is the testing being done?
• find failures (not bugs!)
• certification
• safety criticality
• usability
• performance
• acceptance
• load
• porting
• compatibility
• security

4
 More purposes of testing...

• Why is the testing being done?
• fault tolerance and recovery
• measure reliability
• estimate defects remaining
• decide when to release
• assess problems in software development process
• confirm that functionality is not broken
  following modification
• all the rest of the ilities!

5
Qualities (Ilities)

• Assessing a property (ility) of a PUT may be a
  purpose of testing e.g. correctness, reliability,
  usability, performance,
• may affect choices of strategies or techniques
• assessment of some ilities, such as reliability
  or performance, is difficult without testing
  automation

6
Large Number of Qualities (Ilities)

• Usability, Flexibility, Adaptability,
  Testability, Readability, Portability, Device
  independence, Self containedness, Reusability,
  Interoperability, Efficiency, Device efficiency,
  Integrity, Security, Accuracy, Robustness,
  Completeness, Consistency, Accountability,
  Accessibility, Human Engineered, Self
  descriptive, Fault tolerant, Structuredness,
  Conciseness, Legibility, Augmentability,
  Modifiability, Understandability, Clarity,
  Resilience, Validity, Generality, Minimality,
  Modularity, Expandability, Timeliness,
  Functionality, Cost, Correctness, Reliability,
  Safety, Maintainability, ...

7
Different Viewpoints on the Purpose of Testing

• Quality - at a high level
• Common Reasons for Why test?
• Kaner, Falk, Nguyen 1
• Beizer 3
• Myers 4
• Pfleeger 5
• Quality is system specific
• Ariane 5
• example of a missed purpose
• References

8
Quality - at a high level

• 1 If the customer is not happy, the quality is
  not high
• A programs quality depends on
• the features that make the customer want to use
  the program
• the flaws that make the customer wish hed bought
  something else
• 2 many testing decisions should ultimately
  be based on customer satisfaction.

9
Common Reasons for Why Test? (1)

• To see if it works
• working correctness
• correctness ? comparing software to
  specification of intended behaviour
• Bug prevention is the primary goal of testing.
• Program testing can be used to show the presence
  of bugs, but never their absence.
• Dijkstra (Dahl, et al, 1972)
• to show that particular classes of faults are
  not present

10
Common Reasons for Why Test? (2)

• Testing is any activity aimed at evaluating an
  attribute or capability of a program or system.
  Testing is the measurement of quality.
• Hetzel (1985)
• Definition of testing by a prominent British
  researcher
• Testing is just sampling.

11
Kaner, Falk, Nguyen 1

• The purpose of testing a program is to find
  problems in it.
• testers shouldnt want to verify that a program
  runs correctly
• the program doesnt work correctly, so you CANT
  verify that it does
• The purpose of finding problems is to get them
  fixed.
• fixing bugs results in improved quality
• testers beat up the program in the service of
  making it stronger

12
Beizer 3

• The penultimate objective of testing is to
  gather management information.
• The highest goal of testing is to support
  quality assurance to gather information that,
  when fed back to programmers, results in
  avoidance of past mistakes and in better future
  software.

13
Myers 4

• when one tests a program, one wants to add some
  value to the program. Adding value means raising
  the quality or reliability of the program.
  Raising the reliability of the program means
  finding and removing errors one should start
  with the assumption that the program contains
  errors and then test the program to find as many
  errors as possible.

14
Pfleeger 5

• We test a program in order to demonstrate the
  existence of an error.
• new programmers are not accustomed to viewing
  testing as a discovery process
• students use testing to demonstrate programming
  skill to their instructor
• testing shows their programs work correctly and
  thus demonstrates their ability as a programmer
• a critique of their program is a critique of
  their ability

15
Quality is system specific

• every software system has several key quality
  concerns
• eg. wordprocessor
• functionality, responsiveness, usability
• eg. telephone switching software
• reliability, modifiability
• eg. nuclear operations software
• safety, verifiability
• eg. gaming software
• usability, responsiveness, robustness

16
Ariane 5 Flight 501Equipment involved in failure

• Duplicated inertial reference systems (SRI)
• identical hardware and software
• one SRI is active, one in hot stand-by
• if active unit fails, immediately switches
  control to stand-by unit
• design of Ariane 5 SRI practically same as that
  of Ariane 4

17
Ariane 5 Flight 501Failure Events(1)

• Internal SRI software exception (operand error)
  caused by a data conversion from 64 bit floating
  point to 16 bit signed integer
• error occurred in part of software that computes
  meaningful results only before lift-off
• function continues calculations for 40 seconds
  after liftoff
• time sequence based on a requirement for Ariane 4
• not required for Ariane 5
• values calculated for Ariane 5 different than
  that for Ariane 4 due to differences in the early
  part of trajectory

18
Ariane 5 Flight 501Failure Events(2)

• SRI 1 switched to SRI 2
• SRI 2 suffered same operand error
• Diagnostic bit pattern interpreted as flight data
• Full nozzle deflection of solid boosters and
  Vulcain main engine
• Angle of attack more than 20 degrees causing high
  aerodynamic loads
• Boosters separated from main stage, triggering
  self-destruct system of launcher

19
Ariane 5 Software Testing

• No test was performed to verify the SRI would
  behave correctly when subjected to count-down,
  flight time, and trajectory of Ariane 5
• not possible to test SRI in flight environment
• ground testing can be done
• simulated accelerometric signals with predicted
  flight parameters
• SRI specification did not contain Ariane 5
  trajectory data as a functional requirement
• systems specification for SRI did not contain
  limits of operation

20
Ariane 5Tests for SRIs

• Open loop compliance of SRI and on-board
  computer
• electrical integration tests
• bus communication compliance tests
• Separate equipment qualification
• assumed fully qualified at this level
• Assumed validation by previous use on Ariane 4

21
Ariane 5Possible Simulated Test for SRIs (1)

• Closed loop system test that included both SRIs
• simulate analog output of accelerometers and
  gyros via dedicated test input produced by
  simulation
• depends on accuracy of simulation
• precision not considered high enough for accurate
  calculations by On-Board computer

22
Ariane 5Possible Simulated Test for SRIs (2)

• Decision not to do this test
• precision of navigation software in On-Board
  computer critically depends on precision of SRI
  measurements
• base period of SRI was 1 millisecond simulation
  was 6 milliseconds
• reduces precision of simulation
• Failure to clearly see the purposes of the tests
• emphasis on precision of calculations
• lack of emphasis on system functionality

23
References

• 1 Kaner, Falk, Nguyen, Testing Computer
  Software, Wiley, 1999
• 2 Edward Kit, Software Testing in the Real
  World, Addison-Wesley, 1995
• 3 Boris Beizer, Black-Box Testing, Wiley, 1995
• 4 Glenford Myers, The Art of Software Testing,
  Wiley, 1979
• 5 Shari Pfleeger, Software Engineering Theory
  and Practice, Prentice-Hall, 2001
• 6 ARIANE 5 Flight 501 Failure, Report by the
  Inquiry Board, July 19, 1996