Simplified Modeling of Combinatorial Test Spaces

1
Simplified Modeling of Combinatorial Test Spaces

• Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick
• IBM Haifa Research Labs

2
Agenda

• Introduction
• Counters
• Properties
• Implementation
• Summary

3
Background Restrictions

• Restrictions are a crucial part of real-life
  combinatorial models
• Effort and expertise required for capturing the
  restrictions is high
• Existing work concentrates on the ability to
  state complex restrictions
• Typically as propositional formulas or predicates
• Can be captured using classification trees

4
The Problem

• Restrictions are typically
• Cumbersome to read and review
• Not very generalized (exclude specific
  conditions)
• Hard to maintain

5
Introduction

• An approach for handling some complex
  restrictions
• We introduce two new constructs to the CTD model
• Counter parameters
• Value properties
• Real-life case studies

6
Counters
7
Counters Example

• Consider testing an upgrade process for a
  virtualized server
• Traverses the host and VMs
• Upgrades various out-of-date components
• Ten VMs, each has one of four OS versions RedHat
  5.7, RedHat 6.1, AIX 6.1.3, AIX 6.1.6
• 2 versions are marked preferred
• Triggered when at least half are out of date

8
Counters Example Cont.

• Parameters and values
• VM1 RedHat5.7 / RedHat6.1 / AIX6.1.3 /
  AIX6.1.6
• VM10 RedHat5.7 / RedHat6.1 / AIX6.1.3 /
  AIX6.1.6

9
Counters Example Cont.
Triggered when at least half are out of date ?!
10
Counters Example Cont.

• (Bad) option I explicitly exclude all
  combinations in which less than half are out of
  date
• All 653,312 of them !

11
Counters Example Cont.

• (Bad) option II choose 5 VMs, and force them to
  be outdated
• VM1 ! AIX6.1.6 VM1 ! RedHat6.1 VM2
  ! AIX6.1.6 VM2 ! RedHat6.1 VM5
  ! AIX6.1.6 VM5 ! RedHat6.1
• 252 such (terrible) restrictions

12
Counters

• A counter is a parameter that counts values in
  other parameters
• Automatically evaluated in each test
• Can be used in restrictions and in coverage
  requirements

13
Counters Back to the example

• Define counter preferred
• Counts appearances of RedHat6.1 and AIX6.1.6
  in parameters VM1 VM10
• One restriction that excludes preferred gt 4

14
Auxiliary Variables

• The notion can be extended to auxiliary variables
• Parameters that do not define the test
• but are rather a function of the defining
  parameters
• Examples
• Sum of other numeric parameters
• Count the number of duplicate values

15
(Touching on) Implementation

• Naïve implementation Add the counter to the
  model, and restrictions that correctly map its
  values to the counted values
• Much better Represent as functions on other
  parameters
• More details in the paper

16
Properties
17
Properties Example

• Consider testing a Storage Area Network (SAN)
• A server (one of four types two x86-based, two
  PowerPC-based)
• An operating system (one of 17 versions two
  Windows-based, three AIX-based, one virtualized,
  others Linux-based)
• HBA (one of 11 types, of different manufacturers)

18
Properties Example Cont.

• Some OSs do not run on some processors (e.g., AIX
  on x86, Windows on PowerPC)
• Some HBAs do not have drivers for some OSs
• Some HBAs cannot connect to some processors
• A total of 299 pairs to be excluded
• Tedious and error-prone

19
Properties

• Properties may be defined for parameters
• A value to the property is assigned for each
  value of the parameter
• Can be used in restrictions and in coverage
  requirements
• Represent different abstractions of the parameters

20
Properties Back to the example

• Define property Architecture for parameter
  Server
• The two PowerPC servers will have value PowerPC
• The two x86 servers will have value x86
• Define property OSFamily for parameter OS,
  with values Windows, AIX, Linux

21
Properties Back to the example cont.

• Define properties X_Compatible, P_Compatible,
  AIX_Compatible and Win_Compatible for the HBA
  parameter
• Give property values to the parameter values
  according to their compatibility with x86,
  PowerPC, AIX and Windows, respectively

22
Properties Back to the example Cont.

• The restrictions now disallow
• Server.Architecture PowerPC
  OS.OSFamily Windows
• Server.Architecture x86 OS.OSFamily
  AIX
• Server.Architecture PowerPC !
  HBA.P_Compatible
• Server.Architecture x86 !
  HBA.X_Compatible
• OS.OSFamily AIX ! HBA.AIX_Compatible
• OS.OSFamily Windows ! HBA.Win_Compatible

23
Simplified Modeling Benefits

• More readable restrictions
• Easier to review
• More general
• Preserves time, reduces risk of errors
• More maintainable
• e.g., a new Windows-based OS is introduced

24
Implementation

• Our symbolic BDD-based implementation is agnostic
  to test-space representation
• It merely requires one to symbolically represent
  the set of valid tests
• Simple to extend to support counters properties
• More details in the paper

25
Summary

• Correctly capturing restrictions is a significant
  obstacle in combinatorial testing deployment
• We introduced two constructs that significantly
  simplify restrictions in some cases
• Examples based on real-life cases

26
Thank You For Listening