Advanced Software Engineering: Software Testing COMP 3702(L2)

1
Advanced Software Engineering Software
TestingCOMP 3702(L2)

• Anneliese Andrews
• Sada Narayanappa
• Thomas Thelin
• Carina Andersson

2
News Project

• News
• Updated course program
• Reading instructions
• The book, deadline 23/3
• Project Option 1
• IMPORTANT to read the project description
  thoroughly
• Schedule, Deadlines, Activities
• Requirements (7-10 papers), project areas
• Report, template, presentation
• Project Option 2 date!

3
Lecture

• Some more testing fundamentals
• Chapter 4 (Lab 1)
• Black-box testing techniques
• Chapter 12 (Lab 2)
• Statistical testing
• Usage modelling
• Reliability

4
Terminology

• Unit testing testing a procedure, function, or
  class.
• Integration testing testing connection between
  units and components.
• System testing test entire system.
• Acceptance testing testing to decide whether to
  purchase the software.

5
Terminology (2)

• Alpha testing system testing by a user group
  within the developing organization.
• Beta testing system testing by select customers.
• Regression testing retesting after a software
  modification.

6
Test Scaffolding

• Allows us to test incomplete systems.
• Test drivers test components.
• Stubs test a system when some components it uses
  are not yet implemented.
• Often a short, dummy program --- a method with an
  empty body.

7
Test Oracles

• Determine whether a test run completed with or
  without errors.
• Often a person, who monitors output.
• Not a reliable method.
• Automatic oracles check output using another
  program.
• Requires some kind of executable specification.

8
Testing StrategiesBlack Box Testing

• Test data derived solely from specifications.
• Also called functional testing.
• Statistical testing.
• Used for reliability measurement and prediction.

9
Testing TheoryWhy Is Testing So Difficult?

• Theory often tells us what we cant do.
• Testing theory main result perfect testing is
  impossible.

10
An Abstract View of Testing

• Let program P be a function with an input domain
  D (i.e., set of all integers).
• We seek test data T, which will include selected
  inputs of type D.
• T is a subset of D.
• T must be of finite size.
• Why?

11
We Need a Test Oracle

• Assume the best possible oracle --- the
  specification S, which is function with input
  domain D.
• On a single test input i, our program passes the
  test when
• P(i) S(i)
• Or if we think of a spec as a Boolean function
  that compares the input to the output S(i, P(i))

12
Requirement For Perfect Testing Howden 76

• 1. If all of our tests pass, then the program is
  correct.
• ?xx ? T ? P(x) S(x)
• ? ?yy ? D ? P(y) S(y)
• If for all tests t in test set T, P(t) S(t),
  then we are sure that the program will work
  correctly for all elements in D.
• If any tests fail we look for a bug.

13
Requirement For Perfect Testing

• 2. We can tell whether the program will
  eventually halt and give a result for any t in
  our test set T.
• ?xx? T ? ? a computable procedure for
  determining if P halts on input x

14
But, Both Requirements Are Impossible to Satisfy.

• 1st requirement can be satisfied only if T D.
• We test all elements of the input domain.
• 2nd requirement depends on a solution to the
  halting problem, which has no solution.
• We can demonstrate the problem with Requirement 1
  Howden 78.

15
Other Undecidable Testing Problems

• Is a control path feasible?
• Can I find data to execute a program control
  path?
• Is some specified code reachable by any input
  data?
• These questions cannot, in general, be answered.

16
Software Testing Limitations

• There is no perfect software testing.
• Testing can show defects, but can never show
  correctness.
• We may never find all of the program errors
  during testing.

17
Why test techniques?

• Exhaustive testing (use of all possible inputs
  and conditions) is impractical
• must use a subset of all possible test cases
• want must have high probability of detecting
  faults
• Need processes that help us selecting test cases
• Different people equal probability to detect
  faults
• Effective testing detect more faults
• Focus attention on specific types of fault
• Know youre testing the right thing
• Efficient testing detect faults with less
  effort
• Avoid duplication
• Systematic techniques are measurable

18
Dimensions of testing

• Testing combines techniques that focus on
• Testers who does the testing
• Coverage what gets tested
• Potential problems why you're testing (risks /
  quality)
• Activities how you test
• Evaluation how to tell whether the test passed
  or failed
• All testing should involve all five dimensions
• Testing standards (e.g. IEEE)

19
Black-box testing
20
Equivalence partitioning
mouse picks on menu
Partitioning is based on input conditions
output format requests
user queries
responses to prompts
numerical data
command key input
21
Equivalence partitioning

• Input condition
• is a range
• one valid and two invalid classes are defined
• requires a specific value
• one valid and two invalid classes are defined
• is a boolean
• one valid and one invalid class are defined

22
Test Cases

• Which test cases have the best chance of
  successfully uncovering faults?
• as near to the mid-point of the partition as
  possible
• the boundaries of the partition and
• Mid-point of a partition typically represents the
  typical values
• Boundary values represent the atypical or unusual
  values
• Usually identify equivalence partitions based on
  specs and experience

23
Equivalence Partitioning Example

• Consider a system specification which states that
  a program will accept between 4 and 10 input
  values (inclusive), where the input values must
  be 5 digit integers greater than or equal to
  10000
• What are the equivalence partitions?

24
Example Equivalence Partitions
25
Boundary value analysis
output domain
26
Boundary value analysis

• Range a..b ? a, b, just above a, just below b
• Number of values max, min, just below min, just
  above max
• Output bounds should be checked
• Boundaries of externally visible data structures
  shall be checked (e.g. arrays)

27
Some other black-box techniques

• Risk-based testing, random testing
• Stress testing, performance testing
• Cause-and-effect graphing
• State-transition testing

28
Black Box TestingRandom Testing

• Generate tests randomly.
• Poorest methodology of all Meyers.
• Promoted by others.
• Statistical testing
• Test inputs from an operational profile.
• Based on reliability theory.
• Adds discipline to random testing.

29
Black Box TestingCause-Effect Analysis

• Rely on pre-conditions and post-conditions and
  dream up cases.
• Identify impossible combinations.
• Construct decision table between input and output
  conditions.
• Each column corresponds to a test case.

30
Error guessing

• Exploratory testing, happy testing, ...
• Always worth including
• Can detect some failures that systematic
  techniques miss
• Consider
• Past failures (fault models)
• Intuition
• Experience
• Brain storming
• What is the craziest thing we can do?
• Lists in literature

31
Black Box TestingError Guessing

• Some people have a knack for smelling out
  errors Meyers.
• Enumerate a list of possible errors or error
  prone situations.
• Write test cases based on the list.
• Depends upon having fault models, theories on the
  causes and effects of program faults.

32
Usability testing

• Characteristics
• Accessibility
• Responsiveness
• Efficiency
• Comprehensibility

• Environments
• Free form tasks
• Procedure scripts
• Paper screens
• Mock-ups
• Field trial

33
Specification-based testing

• Formal method
• Test cases derived from a (formal) specification
  (requirements or design)

Specification
Test case generation
Test execution
Model (state chart)
34
Model-based Testing
Specification
VALIDATION
Top-levelDesign
Integration
Detailed Design
Unit Test
Test phase
Coding
35
Need For Test Models

• Testing is a search problem.
• Search for specific input state combinations
  that cause failures.
• These combinations are rare.
• Brute force cannot be effective.
• Brute force can actually lead to overconfidence.
• Must target test specific combinations.
• Targets based on fault models.
• Testing is automated to insure repeatable
  coverage of targets target coverage.

36
Model-Based Coverage

• We cannot enumerate all state/input combinations
  for the implementation under test (IUT).
• We can enumerate these combinations for a model.
• Models allow automated target testing.
• Automated testing replaces the testers
  slingshot with a machine gun.
• The model paints the target casts the bullets.

37
Test Model Elements

• Subject the IUT.
• Perspective focus on aspects likely to be buggy
  based on a fault model.
• Representation often graphs (one format is UML)
  or checklists.
• Technique method for developing a model and
  generate tests to cover model targets.

38
The Role of Models in Testing

• Model validation does it model the right thing?
• Verification implementation correct?
• Informal checklist
• Formal proof
• Consistency checking is representation instance
  of meta model?
• Meta-model UML, graphs, etc technique
• Instance model representation constructed

39
Models Roles in Testing

• Responsibility-based testing.
• Does behavior conform to the model
  representation?
• Implementation-based testing.
• Does behavior conform to a model of the
  implementation.
• Product validation.
• Does behavior conform to a requirements model,
  for example, Use Case models?

40
Models That Support Testability

• Represent all features to be tested.
• Limit detail to reduce testing costs, while
  preserving essential detail.
• Represents all state events so that we can
  generate them.
• Represents all state state actions so that we
  can observe them.

41
Model Types

• Combinational models
• Model combinations of input conditions.
• Based on decision tables.
• State Machines
• Output depends upon current past inputs.
• Based on finite state machines.
• UML models model OO structures.

42
Combinational Model - Spin Lock Binder Fig. 6.1
8
5
2
7
4
0
6
3
9
43
Combinational Model Use a Decision Table

• 1 of several responses selected based on distinct
  input variables.
• Cases can be modeled as mutually exclusive
  Boolean expressions of input variables.
• Response does not depend upon prior input or
  output.

44
Decision Table Construction
Combinational Models

• 1. Identify decision variables conditions.
• 2. Identify resultant actions to be selected.
• 3. Identify the actions to be produced in
  response to condition combinations.
• 4. Derive logic function.

45
Auto Insurance Model
Combinational Models

• Renewal decision table Table 6.1.
• Column-wise format Table 6.2.
• Decision tree Fig. 6.2.
• Truth table format Fig. 6.3.
• Tractability note
• Decision table with n conditions has a maximum of
  2 variants.
• Some variants are implausible.

n
n
46
Insurance Renewal Decision TableBinder Table
6.1
Condition Sec. Condition Sec. Action Section Action Section Action Section
Variant Claims Age Prem. Incr. Send Warning Cancel
1 0 lt26 50 no no
2 0 gt25 25 no no
3 1 lt26 100 yes no
4 1 gt25 50 no no
5 2 to 4 lt26 400 yes no
6 2 to 4 gt25 200 yes no
7 5 or gt Any 0 yes yes
47
Insurance Renewal Decision Table Column-wise
Format
Variant
1 2 3 4 5 6 7
Condition Section Number of Claims 0 0 1 1 2 to 4 2 to 4 5 or more
Insured age 25 or younger 26 or older 25 or younger 26 or older 25 or younger 26 or older Any
Action Section Premium increase() 50 25 100 50 400 200 0
Send warning No No Yes No Yes Yes No
Cancel No No No No No No Yes
Yes
48
Implicit Variants

• Variants that can be inferred, but not given.
• In the insurance example, we dont care about age
  if there are five or more claims. The action is
  to cancel for any age.
• Other implicit variants are those that cannot
  happen --- one cannot be both under 25 yrs old
  and over 25.

49
Test Strategies

• All explicit variants at least once
• When decision boundaries are systematically
  exercised
• weak, if cant happen conditions or undefined
  boundaries result in implicit variants

50
Non binary Variable Domain Analysis

• Exactly 0 claims, age 16-25
• Exactly 0 claims, age 26-85
• Exactly 1 claim, age 16-25
• Exactly 1 claim, age 26-85
• Two, 3, or 4 claims, age 16-25
• Two, 3, or 4 claims, age 26-85
• Five to 10 claims, age 16-85

51
Test Cases Test Cases Test Cases Test Cases Test Cases Test Cases Test Cases
Variant 1 Boundaries Variant 1 Boundaries Variant 1 Boundaries 1 2 3 4 5
0 On 0
Number Of Claims Off (above) 1
Off (below) -1
Typical In 0 0 0 0
gt 16 On 16
Insured Age Off 15
lt 25 On 25
Off 26
Typical In 20 20 20
Expected result Expected result Expected result Accept V3 Reject Accept Reject Accept V2
Premium increase Premium increase Premium increase 50 100 50 50 25
Send warning Send warning Send warning No Yes No No No
Cancel Cancel Cancel No No No No No
52
Additional Heuristics

• Vary order of input variables
• Scramble test order
• Purposely corrupt inputs

53
Statistical testing /Usage based testing
Usage specification Test case generation Test execution Failure logging Reliability estimation

54
Usage specification models
Algorithmic models Grammar model State hierarchy model
lttest_casegt ltno_commandsgt _at_ ltcommandgt ltselectgt ltno_commandsgt ( ltunif_intgt(0,2) prob(0.9) ltunif_intgt(3,5) prob(0.1)) ltcommandgt (ltupgt prob(0.5) ltdowngt prob(0.5))
55
Usage specification models
Domain based models Operational profile Markov model

56
Operational profiles
57
Operational profiles
58
Statistical testing / Usage-based testing
Usage model
Random sample
59
Usage Modelling
Invoke

• Each transition corresponds to an external event
• Probabilities are set according to the future use
  of the system
• Reliability prediction

Click on OK with non-valid hour
Right-click
Move
Dialog Box
Main Window
Resize
CANCEL or OK with Valid Hour
Close Window
Terminate
60
Markov model

• System states, seen as nodes
• Probabilities of transitions
• Conditions for a Markov model
• Probabilities are constants
• No memory of past states

Transition matrix Transition matrix To Node To Node To Node To Node
Transition matrix Transition matrix N1 N2 N3 N4
From Node N1 P11 P12 P13 P14
From Node N2 P21 P22 0 P24
From Node N3 P31 0 P33 P34
From Node N4 P41 0 0 P44
61
Model of a program

• The program is seen as a graph
• One entry node (invoke) and one exit node
  (terminate)
• Every transition from node Ni to node Nj has a
  probability of Pij
• If no connection between Ni and Nj, then Pij 0

P21
N1
N2
P12
Input
P24
P31
P14
P13
Output
N4
P34
N3
62
Clock Software
63
Input Domain Subpopulations

• Human users keystrokes, mouse clicks
• System clock time/date input
• Combination usage - time/date changes from the OS
  while the clock is executing
• Create one Markov chain to model the input from
  the user

64
Operation modes of the clock

• Window main window, change window, info
  window
• Setting analog, digital
• Display all, clock only
• Cursor time, date, none

65
State of the system

• A state of the system under test is an element of
  the set S, where S is the cross product of the
  operational modes.
• States of the clock
• main window, analog, all, none
• main window, analog, clock-only, none
• main window, digital, all, none
• main window, digital, clock-only, none
• change window, analog, all, time
• change window, analog, all, date
• change window, digital, all, time
• change window, digital, all, date
• info window, analog, all, none
• info window, digital, all, none

66
Top Level Markov Chain

• Window operational mode is chosen as the primary
  modeling mode

Rules for Markov chainsEach arc is assigned a
probability between 0 and 1 inclusive,The sum of
the exit arc probabilities from each state is
exactly 1.
67
Top Level Model Data Dictionary
Arc Label Input to be Applied Comments/Notes for Tester
invoke Invoke the clock software Main window displayed in full Tester should verify window appearance, setting, and that it accepts no illegal input
options.change Select the Change Time/Date... item from the Options menu All window features must be displayed in order to execute this command The change window should appear and be given the focus Tester should verify window appearance and modality and ensure that it accepts no illegal input
options.info Select the Info... item from the Options menu The title bar must be on to apply this input The info window should appear and be given the focus Tester should verify window appearance and modality and ensure that it accepts no illegal input
options.exit Select the Exit option from the Options menu The software will terminate, end of test case
end Choose any action and return to the main window The change window will disappear and the main window will be given the focus
ok Press the ok button on the info window The info window will disappear and the main window will be given the focus
68
Level 2 Markov Chain
Submodel for the Main Window
69
Data Dictionary Level 2
Arc Label Input to be Applied Comments/Notes
invoke Invoke the clock software Main window displayed in full Invocation may require that the software be calibrated by issuing either an options.analog or an options.digital input Tester should verify window appearance, setting, and ensure that it accepts no illegal input
options.change Select the Change Time/Date... item from the Options menu All window features must be displayed in order to execute this command The change window should appear and be given the focus Tester should verify window appearance and modality and ensure that it accepts no illegal input
options.info Select the Info... item from the Options menu The title bar must be on to apply this input The info window should appear and be given the focus Tester should verify window appearance and modality and ensure that it accepts no illegal input
options.exit Select the Exit option from the Options menu The software will terminate, end of test case
end Choose any action (cancel or change the time/date) and return to the main window The change window will disappear and the main window will be given the focus Note this action may require that the software be calibrated by issuing either an options.analog or an options.digital input

70
Data Dictionary Level 2
Arc Label Input to be Applied Comments/Notes
ok Press the ok button on the info window The info window will disappear and the main window will be given the focus Note this action may require that the software be calibrated by issuing either an options.analog or an options.digital input
options.analog Select the Analog item from the Options menu The digital display should be replaced by an analog display
options.digital Select the Digital item from the Options menu The analog display could be replaced by a digital display
options.clock-only Select the Clock Only item from the Options menu The clock window should be replace by a display containing only the face of the clock, without a title, menu or border
options.seconds Select the Seconds item from the Options menu The second hand/counter should be toggled either on or off depending on its current status
options.date Select the Date item from the Options menu The date should be toggled either on or off depending on its current status
double-click Double click, using the left mouse button, on the face of the clock The clock face should be replaced by the entire clock window

71
Level 2 Markov Chain
Submodel for the Change Window
72
Data Dictionary
Arc Label Input to be Applied Comments/Notes for Tester
options.change Select the Change Time/Date... item from the Options menu All window features must be displayed in order to execute this command The change window should appear and be given the focus Tester should verify window appearance and modality and ensure that it accepts no illegal input
end Choose either the Ok button or hit the cancel icon and return to the main window The change window will disappear and the main window will be given the focus Note this action may require that the software be calibrated by issuing either an options.analog or an options.digital input
move Hit the tab key to move the cursor to the other input field or use the mouse to select the other field Tester should verify cursor movement and also verify both options for moving the cursor
edit time Change the time in the new time field or enter an invalid time The valid input format is shown on the screen
edit date Change the date in the new date field or enter an invalid date The valid input format is shown on the screen
73
Software Reliability

• Techniques
• Markov models
• Reliability growth models

74
Dimensions of dependability
75
Costs of increasing dependability
C
o
s
t
Dependability
L
o
w
M
e
d
i
u
m
H
i
g
h
V
e
r
y
U
l
t
r
a
-
h
i
g
h
h
i
g
h
76
Availability and reliability

• Reliability
• The probability of failure-free system operation
  over a specified time in a given environment for
  a given purpose
• Availability
• The probability that a system, at a point in
  time, will be operational and able to deliver the
  requested services
• Both of these attributes can be expressed
  quantitatively

77
Reliability terminology
78
Usage profiles / Reliability
Removing X of the faults in a system will not
necessarily improve the reliability by X!
79
Reliability achievement

• Fault avoidance
• Minimise the possibility of mistakes
• Trap mistakes
• Fault detection and removal
• Increase the probability of detecting and
  correcting faults
• Fault tolerance
• Run-time techniques

80
Reliability quantities

• Execution time
• is the CPU time that is actually spent by the
  computer in executing the software
• Calendar time
• is the time people normally experience in terms
  of years, months, weeks, etc
• Clock time
• is the elapsed time from start to end of computer
  execution in running the software

81
Reliability metrics
82
Nonhomogeneous Poisson Process (NHPP) Models

• N(t) follows a Poisson distribution. The
  probability that N(t) is a given integer n is

• m(t) ?(t) is called mean value function, it
  describes the expected cumulative number of
  failures in 0,t)

83
The Goel-Okumoto (GO) model

• Assumptions
• The cumulative number of failures detected at
  time t follows a Poisson distribution
• All failures are independent and have the same
  chance of being detected
• All detected faults are removed immediately and
  no new faults are introduced
• The failure process is modelled by an NHPP model
  with mean value function ?(t) given by

84
Goel-Okumoto

• The shape of the mean value function (?(t)) and
  the intensity function (?(t)) of the GO model

?(t)
?(t)
t
85
S-shaped NHPP model

• ?(t)a1-(1bt)e-bt, bgt0

86
The Jelinski-Moranda (JM) model

• Assumptions
• Times between failures are independent,
  exponential distributed random quantities
• The number of initial failures is an unknown but
  fixed constant
• A detected fault is removed immediately and no
  new fault is introduced
• All remaining faults contribute the same amount
  of the software failure intensity

87
Next weeks

• Next week (April 11)
• Lab 1 Black-box testing