Advanced Software EngineeringCleanroom Software Engineering

1
SE 754-NT Object-Oriented Testing and
Reliability Robert Oshana Lecture
16 For more information, please
contact NTU Tape Orders NTU Media
Services [phone]
oshana_at_airmail.net
tapeorders_at_ntu.edu
2
Statistical Testing Techniques for
Software-Intensive Systems
3
Agenda

• Introduction
• Stochastic Processes
• Markov Models
• Usage Profiles
• Example
• Modeling Limitations

4
Define the usage probabilities

• Approach 1 Uninformed approach
• assign uniform probability distribution across
  the exit arcs for each state.
• Approach 2 Informed approach
• used when some actual user sequences are
  available (prototype, prior version, etc).
• Approach 3 Intended approach
• sequences obtained by hypothesizing runs of the
  S/W by a careful and reasonable user.

5
Engineering Steps to Developing Usage Models

• Identify the system boundary (H/W, S/W, Users)
  and the stimuli they can send
• Define the structure of the usage model based on
  chosen level of abstraction
• Define the important environments for use
  (routine, hazardous, maximum capacity, etc)
• Define the transition probabilities

6
Conduct the statistical test

• Execute tests until required acceptance goals are
  reached or until enough failures are seen to
  warrant stopping.
• Determine what is an acceptable and unacceptable
  outcome for each test.
• acceptable means the S/W is ready to use.
• unacceptable requires determining the level of
  rework
• should testing be stopped?

7
Order of statistical and other testing

• Reliability estimate pertains to a specific
  version of software. If changes are made, the
  estimate no longer applies.
• Perform non-statistical tests prior to
  statistical tests.
• Performing non-statistical tests after
  statistical tests may invalidate the reliability
  certification if the s/w is changed.

8
ATM example
eject card
wrong type
account closed
closed
power on
card inserted
valid type
s/w not invoked
stolen
wait for card
check card
PIN entry
stolen retain
terminate
wrong pin
valid pin
invalid PIN
PIN entry
select
valid pin
with drawal
balance
exit
deposit
9
ATM example - Test planning

• Determine what requires testing (stratification
  of the input domain)
• How the budget will be allocated to testing
  (creating the sampling plan)

10
ATM example - Test planning

• Stratification of the input domain
• user level stratification
• who or what can stimulate the system?
• usage level stratification
• what can the system under test do?

11
ATM example - Test planning

• User and usage level stratification of the ATM
  example

User type
Classes for this type
Normal User
Experienced user, inexperienced user
Adverse User
Thief, Absent minded user, unintelligent user
Usage type
Classes for this type
Security Mode
PIN check, Security check
Normal mode
Withdrawal, Deposit, Balance, Multiple
transactions
12
ATM example - Test planning

• Develop a sampling plan

Stratum
Sampling Percentage
Typical use (80)
Adverse use (20)
Experienced user
30
Inexperienced user
30
Thief
10
Absent minded user
20
Unintelligent user
20
PIN check
20
Security check
30
Withdrawal
10
Deposit
10
Balance
10
Multiple transactions
10
Total
100
100
13
Planning the types of tests to run

• Model coverage tests
• minimal covering set (least cost sequence) used
  to reveal any immediate problems
• Mandatory tests
• required for contract, policy, moral, or ethical
  reasons
• Nonrandom regression tests
• used to discover redundancy in the test suite

14
Planning the types of tests to run

• Critical but unlikely use
• having low likelihood of arising in a random
  sample
• Importance tests
• management constraints for added value
• Partition testing
• usage model can be used to define partitions for
  sampling efficiency

15
ATM example - Markov model
1.0
eject card
1.0
0.1
account closed
0.1
1.0
1.0
0.9
0.1
s/w not invoked
1.0
wait for card
check card
PIN entry
stolen retain
terminate
0.2
0.2
0.6
1.0
invalid PIN
PIN entry
0.1
select
0.8
0.6
0.1
0.2
with drawal
balance
exit
deposit
1.0
1.0
1.0
1.0
16
A test case is any traversal of the usage model
Random test cases based on state transitions that
are randomly selected from the usage distribution
1.0
eject card
1.0
0.1
account closed
0.1
1.0
1.0
0.9
0.1
s/w not invoked
1.0
wait for card
check card
PIN entry
stolen retain
terminate
0.2
0.2
0.6
1.0
invalid PIN
PIN entry
0.1
select
0.8
0.6
0.1
0.2
with drawal
balance
exit
deposit
1.0
1.0
1.0
1.0
17
Algorithm for generating tests
0.1
do lt for a single test casegt bring in state
A draw random number select transition perform
transition to state A1 add stimuli to test
sequence until lttest case readygt
x
0.5
y
A
z
0.4
18
ATM example
eject card
Card ejected
wrong type
account closed
Expired card
Second attempt failed
Valid card type
Stolen card
power on
card inserted
s/w not invoked
Card retained
wait for card
check card
PIN entry
stolen retain
terminate
wrong PIN
First att. failed
valid PIN
PIN entry try 2
invalid PIN
Balance inq. select
Trans. select
With. select
valid pin
exit select
Deposit select
with drawal
Balance inq.
exit
deposit
Transaction complete
19
Usage Model in Tabular Form
20
(No Transcript)
21
SE 754-NT Object-Oriented Testing and
Reliability Robert Oshana End of
lecture For more information, please
contact NTU Tape Orders NTU Media
Services [phone]
oshana_at_airmail.net
tapeorders_at_ntu.edu