MSc Software Testing and Maintenance MSc Pr

1
MSc Software Testing and MaintenanceMSc Prófun
og viðhald hugbúnaðar

• Fyrirlestrar 35 36
• What is the expected result ?

of hratt -gt
2
oracle/véfrétt
Case StudyDæmisaga
Reference A Taxonomy for Test Oracles, Douglas
Hoffman, Software Quality Week (QW98), 1998,
8pp. http//www.softwarequalitymethods.com/Papers/
OracleTax.pdf
3
The oracle

• The oracle is the mechanism used to generate
  expected results.
• High machine speeds, cheap memory, and test
  automation has meant that it is easy to generate
  very large amounts of test data.
• Correspondingly, the oracle being used has to be
  capable of generating all the expected results.

4
Oracle methods used by Hoffman

• Manual verification of results
• human oracle
• Separate program implementing the same algorithm.
• Simulator of the software system to produce
  parallel results.

5
Oracle methods used by Hoffman

• Debugged hardware simulator to emulate hardware
  and software operations.
• Earlier version of the software.
• Same version of software on a different hardware
  platform.

6
Oracle methods used by Hoffman

• Verification of consistency of generated values
  and end points.
• Sampling of values against independently
  generated expected results.

7
Background

• Many organizations rely on a human oracle but the
  volume of data from automated tests is often
  overwhelming.
• It is not enough to regard program termination as
  a successfully executed test case. Output results
  must be verified.
• very few errors cause noticeable abnormal
  termination

8
Background

• Verifying mathematical subroutines is relatively
  straightforward by using a different algorithm,
  language, compiler, etc.
• Verifying the interrupt handling of an operating
  system kernel is far more difficult.
• Generating complete sets of expected results is
  often not practical.
• It is particularly difficult to generate
  expected information for file directories,
  machine registers, system tables, memory, etc.

9
Input-Process-Output testing model
System Under Test
Test Inputs
Test Results

• SUTs very rarely fit this simple model.
• There are often multiple complex input and
  outputs.
• Outputs can include values left in memory, the
  program state for the SUT (and other software),
  data base values, ...

10
Expanded Testing Model
Test Inputs
Test Results
Precondition Data
Postcondition Data
System Under Test
Precondition Program State
Postcondition Program State
Environmental Inputs
Environmental Results

• Test designers select what they regard as the
  most relevant inputs and results and choose a
  subset to use in verifying program behaviour.
• Environmental inputs are rarely specified and
  often only some preconditions are specified.

11
Patient monitoring
extra

• Suppose the SUT is software used to monitor a
  patients vital signs and provide intelligent
  advice.
• Suppose the SUT has access to the electronic
  patient record.

12
Patient monitoring
extra

• What data would you use in the electronic patient
  record for the purposes of testing ?
• age, weight, sex, allergies, smoker/non-smoker,...
  
• Should the electronic patient record be updated
  after executing a test ?
• What data would you use to represent the current
  state of the patient ?
• drug dosage, heart rate, level of consciousness,
  intubated or not, blood alcohol level, ...
• Should the current state of the patient be
  updated after executing a test ?
• if the advice is to intubate, change drug dosage,
  ...

13
Patient monitoring
extra

• A likely scenario is that the electronic patient
  record is accessed over a local or a wide-area
  network.
• Should network load be an input to the testing ?
• Should size of electronic patient record be an
  input to the testing ?
• Does the SUT download a complete copy or send SQL
  queries to a third party ?

14
Patient monitoring
extra

• Just prior to connecting the patient to the
  monitoring system, what should the patient data
  be?
• A blank patient record? An average patient
  record?
• Just prior to connecting the patient to the
  monitoring system, what should the state of
  intelligent advice be?
• No advice?
• Some monitoring devices may provide data at
  intervals rather than in real-time...
• So initial values at t0 may be important.

15
Observations

• Several oracles may be needed for one program.
• functional behaviour
• screen layout and navigation
• memory use

... Only the SUT running in the target
environment will process all the inputs and
provide all the results. No matter how meticulous
we are in creating an oracle, we will not achieve
both independence and completeness.
16
Characteristics of oracles

• Completeness of information
• A complete oracle that duplicates all the
  results categories can be regarded as a second
  implementation of the SUT.
• Accuracy of information
• A complete oracle that is as accurate as the SUT
  is at least as complex as the SUT.
• Differences detected by a complex oracle may be
  as a result of faults in the oracle rather than
  the SUT.
• Faults will be missed if both the SUT and a
  complex oracle contain the same fault.

17
Characteristics of oracles

• Independence of oracle from SUT
• different algorithms
• different libraries
• different system platform
• different operating environment
• Speed of predictions
• is the oracle slow ?
• Time of execution
• in parallel with the SUT ?

18
Characteristics of oracles

• Usability of results
• An expected result calculated using pencil and
  paper will have to be manually copied to the test
  environment.
• Scripts may have to be written to transfer
  expected results from a simulation to the test
  environment.
• Maintainability of the oracle in response to
  changes in the SUT.

19
Manual testing

• To calculate an expected result, the manual
  human tester, in addition to paper and pencil,
  might use
• books
• tables
• calculators
• programs...

20
Automated testing

• Automated testing does not mean mechanical
  reproduction of manual tests.
• Some kind of oracle is used.

21
True oracle
Types of oracles
Categories are based on oracle outputs, not the
methods used.

• A true oracle faithfully reproduces all relevant
  results for a SUT using independent platform,
  algorithms, processes, compilers, code, etc.
• Building a true oracle for a function can be
  relatively straightforward.
• The sin() function can be implemented using
  different hardware and software.
• A true oracle can accept any inputs that the SUT
  can and there is a good chance of finding faults.

22
True oracle
Types of oracles
Categories are based on oracle outputs, not the
methods used.

• The less an oracle has in common with the SUT the
  more confident we are that the results of testing
  are correct.
• A true oracle is as complicated as the SUT and
  may have its own faults...
• Common hardware and software (operating system,
  compiler,...) may inject errors that effect the
  oracle and the SUT in the same way.
• Remember the Pentium bug...

23
The Pentium Division Bug 1994
extra

• There was a fault in a lookup table used to
  perform division.
• Example 1
• Inputs x 4195835, y 3145727
• Calculation z x - (x/y)y
• Expected result 0
• Pentium answer 256
• Example 2
• Inputs x 5505001, y 294911
• Calculation x/y
• Expected result 18.66665197
• Pentium answer 18.66600093

24
Stochastic oracle
Types of oracles
Categories are based on oracle outputs, not the
methods used.

• When resources are limited, a relatively small
  sample of inputs are used in testing.
• A pseudo-random number generator may be used to
  select the input values which are fed to the
  oracle and the SUT.
• A statistically random sample suffers no bias.
• The oracle must be capable of accepting any
  randomly generated value or it must be specially
  built for the values selected.

25
Heuristic oracle
Types of oracles
Categories are based on oracle outputs, not the
methods used.
leiðsagnarregla

• Selected results are reproduced.
• Remaining values are checked for consistency
  using a heuristic.
• The sin() function can be checked for selected
  values such as 90o (1), 180o (0), 270o (-1), and
  360o (0).
• Using small increments of the inputs, the outputs
  from the SUT can be checked for consistency
• progressively greater/progressively less
• A heuristic oracle is very easy to implement and
  runs much faster than a true oracle.

26
Heuristic oracle
Types of oracles
Categories are based on oracle outputs, not the
methods used.
extra
leiðsagnarregla
1
0
-1
0o
90o
180o
270o
360o

• The sin() function increases between 0 and 90
  degrees, decreases from 90 to 270 degrees, and
  increases again to 360 degrees.
• The heuristic oracle for sin() can detect various
  kinds of faults.

27
Heuristic oracle
Types of oracles
Categories are based on oracle outputs, not the
methods used.
extra
leiðsagnarregla
1
0
-1
0o
90o
180o
270o
360o

• The values are increasing and decreasing as
  appropriate.
• The heuristic oracle for sin() would not detect
  that the function is Triangle.

28
Sampling oracle
Types of oracles
Categories are based on oracle outputs, not the
methods used.

• A sampling oracle involves using a selected (not
  random) set of values.
• Boundary values, midpoints, minima, and maxima
  are typically chosen.
• A sampling oracle is created to provide the
  expected results for the selected values.

29
Consistent oracle
Types of oracles
Categories are based on oracle outputs, not the
methods used.

• A consistent oracle uses the results from a
  previous test run as the oracle for the next test
  run.
• A useful way for evaluating changes.
• Historic faults may remain undetected.

30
Five types of oracles Table 2
True oracle Stochastic Heuristic Sampling Consistent
Definition Independent generation of expected results Verify a randomly selected sample Verify selected points, use a heuristic for remainder Verify a specially selected sample Compare run n results with n-1
Example of use Algorithm validation Operational verification Algorithm verification Boundary testing Regression test
Advantages Possibility for exhaustive testing Can automate tests with a simple oracle Easier than true oracle Very fast verification possible with simple oracle Fastest can generate and verify large amounts of data
Disadvantages Expensive implementation. Possibly long execution times May miss systematic and specific errors. Can be time consuming to verify Can miss systematic errors and incorrect algorithms May miss systematic or specific errors Original run may include unknown errors
31
Other remarks on oracles

• Oracle data can be generated before, parallel to,
  or after a test case is run.
• If generated before, inputs to the test case must
  be known beforehand.
• If the test case performs comparisons with the
  expected results, the oracle must run before or
  in parallel with the test case.
• Parallel running of the oracle assumes the oracle
  is as fast as the software under test.

32
Other remarks on oracles

• Manually comparing outputs with expected results
  is limited by human processing capabilities.
• Plan the method of results comparisons to be
  used.