Reachability Testing An Approach to Testing Concurrent Software'

1
Reachability Testing An Approach to Testing
Concurrent Software.
Advanced Software Testing

2
Organization of the talk

• Introduction to the problem.
• Existing solutions.
• Reachability Testing
• How to achieve our goal.
• Generating race-variants.
• Empirical results.
• Conclusions

3
Introduction

• Consider the following program

1. Result is 55
2. Result is 30
4
Introduction

• Different executions of a concurrent program may
  produce different results.
• Unpredictable progress of concurrent processes.
• Use of non-deterministic statements.
• Possible that some executions of P with input X
  may produce correct results and some may not.

5
Introduction

• What does this mean?
• A single execution of a program P on an input X
  is not sufficient to test correctness of P on X.
• Non-determinism further adds to our woes.

6
Properties of Concurrent Programs

• Set of shared variables accessed by different
  processes.
• Exercising of a sequence of synch events called a
  SYN-sequence.
• Format of SYN-sequence depends the concurrent
  constructs used.
• E.g. Read Write, Wait Signal

7
Existing Testing techniques

• Non-deterministic approach
• Execute P with X many times.
• Deterministic approach
• Force execution of P with input X for a given S.
• Prefix based approach
• Replay technique (?)

8
Existing Testing techniques

• Non-Deterministic Testing (NDET)
• Select a set of inputs X for P
• For each x ? X, execute P with x many times and
  examine the result.
• Intuition multiple executions may exercise
  different feasible SYN seq. and hence detect
  errors.
• Introduce more non-determinism.
• Control scheduling, Random Sleep
• What are the problems?

9
Existing Testing techniques

• Deterministic Testing (DET)
• Select a set of input and corresponding
  SYN-sequences (X, S).
• Force execution of P for each (X,S) set.
• Intuition forced execution determines if S is
  feasible in P with input X.
• Selection based on implementation and
  specification.
• What are the problems?

10
Existing Testing techniques

• Prefix Based testing.
• Prefix of feasible SYN-sequence is said to be
  prefix-feasible for P.
• Hybrid approach combines both NDET and DET.
• Perform DET of P with (X,S) until the end of S.
• Perform NDET after the end of S is reached.
• If S is prefix feasible for P, this test can be
  called as prefix-based replay.

11
Reachability Testing

• Reachability testing of P with input X and
  SYN-sequence S involves
• Use S to generate a set of prefixes of other
  feasible SYN-sequences (called race-variants).
• Perform prefix-based replay of P with X and each
  race-variants to get S.
• Repeat above steps with S.

12
Reachability Testing

• If,P with input X contains a finite
  SYN-sequences, reachability testing assures
• Exhaustive testing of P with input X
• Determine the correctness of P with X
• Obviously a problem not handled in paper.
  Mention that a limit is needed.
• Now, how do you find out the race-variants?

13
Reachability Testing

• Example Algorithm to generate race-variants based
  on Read-Write.
• Each variable has a version.
• Read event denoted as R(U, V),
• U is the name of the variable.
• V is the version of the variable.
• Write event denoted as W(U, V).
• Only event to modify the version number.

14
Reachability Testing

• Generate RV diagram to get race-variants of S.
• It essentially is a tree with each node
  representing a prefix or a race-variant of S.
• Each node has Index vector (Iv) and Version
  vector (Vv) and a flag.
• Root has each element in Iv and Vv set to 0 and
  the flag is not set.

15
Reachability Testing

• Consider a program with 2 processes and 2 shared
  variables, such that
• P1 R(A),R(B),W(A)
• P2 W(A),W(B)
• A feasible SYN-sequence S for this can be
• P1 R(A,0),R(B,0),W(A,2)
• P2 W(A,1),W(B,1)
• Loosely based on happens before

16
Reachability Testing

• Algorithm to generate RV diagram(RVD)
• For an node with the flag not set, construct a
  child with incremented index.
• Change the version of the variable in the child,
  if the operation was a write.
• If the child is a prefix node (that has already
  been generated) or a race variant node then set
  the flag.

17
Reachability Testing

• Algorithm to generate RV diagram.
• A node is a prefix if
• For Index I,
• The version of the variable in S and the child
  node are the same
• It is a race-variant other-wise.
• Repeat the process until the flags of all the
  nodes are set.

18
Reachability Testing
(0,0) (0,0)
(1,0) (0,0)
(0,1) (0,1)
PS Not Exhaustive!!!
(2,0) (0,0)
(1,1) (1,0)
(3,0) (1,0)
(2,1) (1,0)
(2,1) (1,0)
(1,2) (1,1)
P1 R(A),R(B),W(A) P2 W(A), W(B)
S1 R(A,0),R(B,0),W(A,2) S2 W(A,1),W(B,1)
(2,2) (1, 1)
(3,1) (2,0)
(2,2) (1,1)
19
Reachability Testing

• Execute P with X, to collect one or more unmarked
  SYN-sequences.
• For each unmarked SYN-sequences, derive
  race-variants.
• If the race-variant is not already in the set of
  RVs, add it.
• Generate more SYN-sequences for each unmarked rv
  E RVs.
• Repeat the above steps for the new sequences.

20
Test Results

• Tested using two producers and a consumer.
• Producer produces 2 items each
• Consumer consumes all 4 items.
• Totally 420 possible sequences.
• NDET finds only about 10 sequences.
• Reachability testing identifies all 420 possible
  cases. (How wonderful !!!)

21
Test Results

• Separately tested NDET using RR scheduling and
  random selection w/ time slicing.
• NDET with RR gives about 10 sequences only.
• Random selection manages to get all the sequences
  when executed about 10000 times!!!
• The time taken by reachability testing about 250
  sec (3 processors).

22
Conclusion

• Exhaustively tests P for an input X.
• It is cost-effective when compared to NDET since
  the time taken is only a few tens of seconds
  more.
• Strategy to handle infinite sequences has to be
  developed.
• Algorithm needs to be improved for time and space
  requirements.