SE 754NT

1
SE 754-NT Object-Oriented Testing and
Reliability Robert Oshana Lecture
15 For more information, please
contact NTU Tape Orders NTU Media
Services [phone]
oshana_at_airmail.net
tapeorders_at_ntu.edu
2
Statistical Testing Techniques for
Software-Intensive Systems
3
Agenda

• Introduction
• Stochastic Processes
• Markov Models
• Usage Profiles
• Example
• Modeling Limitations

4
Do you believe that...

• All bugs are equally bad?
• All bugs should be fixed?
• Should you find all the bugs, or just the bad
  ones?

5
Adequate quality? (Bach)

• Failures may not be noticed
• Faults may not result in failures
• Faults noticed may not be reported
• Software development may be described as the
  final proof of Murphy's Law - "If anything can
  possibly go wrong, it will, at the worst possible
  time"

6
We often chase the wrong bugs

• 33 of all faults failed less than once in every
  5000 execution years
• 2 of all faults caused the common failures (more
  than once every 5 execution years)

Source Reliable Software Technologies
7
The Problems with Testing

• Testing is a large part of the overall project
  budget (up to 80).
• Look at programming as an activity dominated by
  testing with occasional lapses into design and
  documentation!
• Most testing produces no quality measurements.
• Most testing tools are designed to accommodate
  code coverage.

8
Failures and Faults

• Software testing is the process of executing
  software to determine correctness with respect to
  the product spec.
• Failures are deviations from the specification
  (observed vs intended).
• Faults are what causes failures (the things you
  find and fix).

9
An unbounded process

• Software testing is an unbounded process
• you can never know with total certainty whether
  the software
• contains no faults
• will never fail
• Any testing process is a form of sampling. Most
  of this sampling today is ad hoc

10
Problems with coverage testing

• Many errors occur in software despite complete
  coverage testing.
• Many sources of failures are design errors
• Failures found when executing s/w under the right
  usage scenarios
• Failures defined by usage, not code
• Executing a line of code does not mean that it is
  correct

11
Maximize the MTTF

• Some execution failures will occur frequently,
  others infrequently.
• Coverage testing likely to find the infrequent as
  well as frequent failure.
• If the goal of testing is to maximize the MTTF a
  strategy that concentrates on the failures that
  occur frequently is more effective than one that
  has equal probability of finding high/low
  failures.

12
Another focus on testing

• Test the S/W the way the users use it.
• Allows development of tests without having to
  wait for the code to be complete.
• Find failures that impact the user most.
• Provide maximum quality for the testing dollar.
• Focus on quality from the users point of view

13
Find the high frequency faults

• Some failures occur more than others.
• Users determine which faults occur and how often.
• Locate high frequency faults early (from a users
  point of view).
• Reliability increases when high frequency faults
  are eliminated early
• Reliability is determined by the user !!

14
Can we do this statistically?

• When a population is too large for exhaustive
  study (software systems) a statistically correct
  sample must be drawn as a basis for inferences
  about the population
• Treat testing as an engineering problem to be
  solved by statistical methods

15
Components of a statistical experiment

• Population the set of items in which we wish to
  make a statement.
• Strata useful subsets of the population.
• Sample subset of the population actually used in
  the experiment.
• Inference process of estimating population
  statistics based on results of the sample.

16
Sampling a population
Usage population
reliability inference
3
stratum 1
2
samples from the population
usage sample
17
Parallel between a classical statistical design
and statistical software testing
18
Benefits of statistical testing

• Statistical testing used to determine when to
  release software
• Predict failures
• Estimate testing costs
• Observe measures about software quality
• Acceptance decisions on impartial basis

19
Binomial Experiments (and how they relate to
software testing)

• Two outcomes (true or false)
• Software tests either pass or fail
• Identical trials
• Outcomes of the trials must be consistent (by
  testers and evaluators)
• Common setting
• Same version of software must be used in each
  test case (a new version marks the beginning of a
  new experiment)

20
Binomial Experiments (and how they relate to
software testing)

• Trials are independent
• Each software test should be independent of
  previous tests
• Probability of success is the same in all trials
• The pass/fail criteria does not change from test
  to test

21
Stochastic Processes

• A stochastic process is a random process or
  experiment that takes place in stages
• The outcome of any preceding experiment does not
  affect predictions of the next experiment
• Stochastic processes can be used to study the
  evolution of a system state as a function of time

22
Stochastic software

• Software use is stochastic. S/W has many
  different uses for different missions starting
  with different initial conditions and different
  input data.

23
Stochastic Processes

• The stochastic process most commonly used is the
  Markov property Given the state of a process
  at any moment in time, its subsequent behavior is
  independent of its past history

24
A wrong approach to statistical testing
select inputs at random
A
D
C
B
The sample is only correct when the probability
of selecting the next stimuli is not dependent
on the current state of the system.
25
Markov Models

• A better model encodes the usage space (input
  domain) as a Markov chain.
• States of the model represent usage history
  (state of the s/w from users point of view)
• Arcs are state transitions caused by stimuli
  (human, h/w, other s/w, etc)
• Transition probabilities simulate how a typical
  user is likely to apply stimuli

26
Markov chain - digraph
D,q(D)
state 3
A,p(A)
state 1
C,p(C)
B,p(B)
B,q(B)
state 4
state 2
C,p(C)
D,q(D)
27
Markov Chain

• A Markov chain is a discrete time, finite state
  machine
• A Markov chain can be represented by a directed
  graph (digraph)
• Points correspond to states and arcs correspond
  to its possible transitions between the states
• This is called a transition diagram

28
States, probabilities, and arcs
Outcome at
Outcome at
Weighting
Stage 1
Stage 2
C
0.5 0.75 .375
A
C
0.75
A
0.25
D
0.5 0.25 .125
A
D
0.5
Invoke
0.5
E
B
B
E
0.5
0.5 0.5 0.25
0.2
F
B
F
0.3
0.5 0.2 0.1
G
0.5 0.3 0.15
B
G
------------------------- SUM 1.0
29
Arcs, states, and probabilities
1/3
state 3
state 1
6/7
1/3
1/7
state 4
1
3/4
1/4
States and arcs can be expanded like macros
state 2
1/3
30
Transition Matrix
state 1 state 2 state 3 state 4
state 1 state 2 state 3 state 4
0 0 0 0
1/7 0 1/3 3/4
0 1 1/3 0
6/7 0 1/3 1/4
T
all rows sum to 1
31
Add Invocation and Termination States
32
Properties of Markov Chains

• Time homogeneous Probabilities on the arcs do
  not change with time
• Finite state Finite number of states
  represented in the model
• Discrete parameter The state transition
  probabilities are discrete

33
Advantages of Markov Chains

• State diagram is a common and familiar tool
• Graph theoretic modeling techniques available
• Mathematical analysis available
• Sequences look more like typical use
• Generalizes well

34
Developing Usage Profiles

• Define the usage models
• Stratify the models
• Define the models for each stratum
• Analyze the models
• Update and finalize the models

35
The Statistical Testing Process
Operational Usage Modeling
Model Analysis and Validation
Test Planning
Testing
Product and Process Measurement
Software certification
36
Developing Usage Profiles

• Software systems may have multiple users or
  classes of users that need to be analyzed.
• Different modes of operation are considered as
  well in different usage strata.
• A finite state model of software usage is
  developed based on the operational states of the
  software system.

37
Developing Usage Profiles

• Test developer must understand what the S/W is
  intended to do and how it is to be used.
• No knowledge about how the software is designed
  and constructed is required.

38
Special purpose models

• Usage models represent conditions under which S/W
  is used.
• In general, expected usage conditions are
  modeled.
• Other usage conditions may be of interest.

39
Special purpose models

• Models for special purposes
• Hazardous usage conditions for safety critical
  software
• Malicious usage conditions for special security
  requirements.
• Usage should be characterized in whatever terms
  are important in the testing context.

40
Attempting to Model Reality
usage model and profile
real usage
sample test cases
Usage models characterize the infinite population
of scenario of use at the chosen level of
abstraction
41
SE 754-NT Object-Oriented Testing and
Reliability Robert Oshana End of
lecture For more information, please
contact NTU Tape Orders NTU Media
Services [phone]
oshana_at_airmail.net
tapeorders_at_ntu.edu