Analysis of Test Results CFICSE VV09 October 1999

1
Analysis of Test ResultsCFICSE VV09
October 1999

• Dr. Terry Shepard
• Royal Military College of Canada
• Electrical and Computer Engineering
• 613-541-6000 ext. 6031

2
Analysis of Test Results Topics

• Issues related to expected results
• Sources of Errors found during testing
• Tracking test results and failure/fault counts

3
Issues related to expected results Slide 1

• Finding an independent source (oracle) for
  expected results may be difficult
• eg. In a scientific calculation, the only place
  to get the results may be from the program that
  is being tested
• in some cases, assertions and internal
  consistency checks among actual results may be
  the only options

4
Issues related to expected results Slide 2

• Determination of test pass or fail should be
  automated in general
• human observation is tedious, expensive and error
  prone
• in some cases, human observation is valuable, or
  even necessary
• acceptance testing
• verification of operational procedures
• usability testing

5
Issues related to expected results Slide 3

• In the case of a difference between actual and
  expected results, testing tools should report
  this, and should facilitate debugging actions as
  needed
• Representation may mean that comparison between
  actual and expected values is not exact
• floating point
• screen contents e.g. trailing blanks on a line

6
Issues related to expected results Slide 4

• Expected or actual result (or both) may be an
  exception
• special handling may be needed
• coverage measures may be disrupted if an
  exception causes a block to be left prior to
  completion
• Results may not be predictable or repeatable in
  testing distributed, concurrent systems

7
Sources of Errors found during testing

• are testing checklists different from inspection
  checklists?
• Beizer 2nd Ed. Ch. 2, pp. 27-58 Appendix (pp.
  460-476) A Taxonomy of Bugs
• IBM work on Orthogonal Defect Classification Don
  Hamiltons thesis

8
Test results and failure/fault counts need to be
tracked to answer questions such as

• When can we stop testing?
• How many bugs can we expect?
• Which test technique is most cost- effective?
• Are we testing hard or are we testing smart?
• Do we have a strong program or a weak test suite?