Determination of Test Configurations for Pairwise Interaction Coverage

1
Determination of Test Configurationsfor
Pair-wise Interaction Coverage

• Alan Williams
• School of Information Technology and
  EngineeringUniversity of OttawaOttawa ON Canada
• awilliam_at_site.uottawa.ca
• www.site.uottawa.ca/awilliam

2
Outline

• Context Testing of heterogeneous systems
• Interaction Coverage
• Design of statistical experiments, and the
  application to system testing
• Test configuration generation
• Some results

3
Testing of Configurable Systems
telephone switch
called phone
calling phone
Type regular wireless pay phone
Type regularwirelesspager
Market Canada US Mexico
Call type local long distance toll free

• The goal Verify reliability and
  interoperability by testing as many system
  configurations as possible, given time and budget
  constraints.

4
Electronic Commerce Example
Payment Server
BusinessWeb Server
Client browser
Type MasterCard,Visa,American Express
Business Database
Type Netscape,Explorer
Type WebSphere,Apache,BackOffice
Type DB/2,Oracle,Access
5
Testing Java Interface Implementations
Message parameters
DataInput
DataOutput
Serializable
DataOutputStream
DataInputStream
Serializable
RandomAccessFile
RandomAccessFile
...
Implementations of DataInput
Implementations of DataOutput

• Sender, receiver, and message parameters could be
  one of various implementations of a public
  interface

6
Interaction Coverage Example

• Suppose that we have three parameters.
• For each parameter, there are two possible
  values.
• types are
• A, B for parameter 1
• J, K for parameter 2
• Y, Z for parameter 3
• Degree of interaction coverage is 2.
• We want to cover all potential 2-way interactions
  among parameter values.

7
Interaction test coverage goal
Goal using a subset of all test configurations...
...cover all possible 2-way interactions
8
Statistical Experimental Design

• Used in many fields other than computer science.
• Objective
• create an experiment to test several parameters
  at once
• individual effect of each parameter
• interactions among parameter
• minimize the number of experiments needed
• facilitate result analysis
• Application to software system testing
• Can be used in any situation where there are a
  set of parameters, each of which have a set of
  (discrete) values.
• Assumption each parameter can be set
  independently
• when there are dependencies, enumerate legal
  combinations

9
Orthogonal Arrays

• Orthogonal arrays are a standard construction
  used for statistical experiments.
• Strength 2 select any 2 columns and all ordered
  pairs occur the same number of times
• covers all 2-way interactions

• Orthogonal arrays can be found in statistical
  tables, or can be calculated from algebraic
  finite fields.
• Many existence restrictions
• if you have n values, there can be, at most, n
  1 parameters

10
Adaptation to Software Testing

• If we are testing strictly for software
  interactions, we can use a smaller experimental
  design.
• Why?
• If each component has previously been through a
  unit testing process, we can eliminate the need
  for testing for the effect of a single parameter.
• Software testing yields a discrete test result
  (pass or fail), rather than requiring result
  analysis of real valued results.
• The result
• Each interaction needs to be covered at least
  once, instead of the same number of times.
• Fewer configurations are required
• The construction for this purpose is called a
  covering array.

11
Covering Arrays

• Definition of covering array
• if we select d columns, all possible ordered
  d-tuples occur at least once
• A covering array of strength d will ensure than
  any consistent interaction problem caused by a
  particular combination of two parameter values is
  detected. Problems caused by an interaction of d
  1 (or more) values may not be detected.
• Choosing the degree of coverage defines the
  trade-off in risk we are making
• fewer test configurations versus potential
  uncovered interactions

12
Recursive Covering Array Construction

• Problem
• if the range of values is 1, , n, then an
  orthogonal array can handle at most n 1
  parameters
• existence of suitable orthogonal arrays
• Goal
• generate covering arrays for problems of
  arbitrary size
• Method
• assemble larger covering array from smaller
  building blocks
• no heuristics

13
Building Blocks for Large Covering Arrays

• Entire orthogonal array is denoted O.

Parameter
Configuration
1
2
3
4

• Call the array with the first row dropped a
  basic (B) array.

• Call the lower right 6 rows and 3 columns a
  reduced (R) array.
• The reduced array covers pair-wise combinations
  other than (x,x).

R
B
O
14
Constructing Larger Covering Arrays
With 3 values per parameter, an orthogonal array
can handle up to 4 parameters.
15
Constructing Larger Covering Arrays
Duplicate orthogonal array three times for 12
parameters
16
Constructing Larger Covering Arrays
Check coverage so far For the first column...
17
Constructing Larger Covering Arrays
we have pair-wise coverage with the rest of the
orthogonal array (by definition)
18
Constructing Larger Covering Arrays
but we also have pair-wise coverage with the
corresponding columns in the duplicate orthogonal
arrays.
19
Constructing Larger Covering Arrays
1
1
1
1
1
1
2
2
2
2
2
2
3
3
3
3
3
3
We have also covered the (x,x) combinations in
the identical columns, but not the (x,y)
combinations.
20
Constructing Larger Covering Arrays
1
1
1
1
Use reduced array, which covers only the
(x,y) combinations
1
1
2
2
2
2
2
2
3
3
3
3
3
3
21
Constructing Larger Covering Arrays
1
1
1
1
and add new configurations to cover
missing combinations.
1
1
2
2
2
2
2
2
3
3
3
3
3
3
22
Constructing Larger Covering Arrays
1
1
1
1
This covers the remaining combinations for the
first column.
1
1
2
2
2
2
2
2
3
3
3
3
3
3
23
Constructing Larger Covering Arrays
1
1
The same scheme applies to other columns.
1
1
1
1
2
2
2
2
2
2
3
3
3
3
3
3
All pair-wise combinations have now been covered
24
Constructing Larger Covering Arrays
O
O
O
R ? 4 (columns duplicated 4 times consecutively)
25
Another recursive construction
O
O
O
O

• This construction handles up to 16 parameters

B ? 4
26
Multistage covering arrays
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
B ? 4
B ? 4
B ? 4
B ? 4
B ? 16
27
Some results

• Results from the recursive construction example
• 13 parameters, 3 values for each parameter
• Number of potential test configurations
  1,594,323
• Number of possible pair-wise interactions 702
• Minimum number of configurations for 100
  coverage of pair-wise interactions 15
• Achieving coverage of pair-wise interactions
  results in a number of test configurations that
  is proportional to
• The logarithm of the number of parameters.
• The square of the maximum number of values for
  any parameter

28
Number of configurations to coverpair-wise
interactions
29
TConfig

• A prototype software tool has been constructed to
  calculate test configurations for pair-wise
  interaction coverage.

Demo
30
Further work

• Handling parameters with varying numbers of
  values
• Some work has been done in this area, but not
  with the recursive constructions
• Handling required configurations
• existing regression test suite
• preferred customer configurations
• Recursive construction of covering arrays to
  cover 3 way interactions