Orthogonal Array Testing Strategy

1
Orthogonal Array Testing Strategy
2
Abstract

• Orthogonal Array Testing Strategy (OATS) is
  a systematical, statistical way of testing
  pair-wise interactions by deriving suitable small
  set of test cases from a large number of
  scenarios. The testing strategy can be used to
  reduce the number of test combinations and
  provide maximum coverage with a minimum number of
  test cases. OATS utilizes an array of values
  representing variable factors that are combined
  pair-wise rather than representing all
  combinations of factors and levels.

3
Abstract

• Being intelligent about which test cases are
  selected can make the difference between
  endlessly executing over executing a concise,
  defined set of test cases. OATS steps involve
  identifying, determining, finding, mapping,
  choosing, and transcribing. As an end result,
  orthogonal array testing provides a means to
  select a test set that guarantees, creates,
  exercises, and is simplistic, deriving a suitable
  small set of test cases from a large number of
  scenarios with broad coverage.

4
Outline

• Definition for Orthogonal Array Testing Strategy
  (OATS)
• The OAT Theory
• Simple OAT Example
• Pairwise Combination of Parameters
• OATS Example 2
• Why Use Orthogonal Array Testing?
• Advantages of OATS
• Disadvantages of OATS

5
Outline

• Selecting the Right Test Set with OATS
• Mistakes to Avoid
• Applying OATS manually
• Focusing OATS on the wrong areas of applications
• Using OATS for minimal testing efforts
• Using OATS for High-Risk Applications
• Picking the Wrong Parameters to Combine

6
Outline

• How to Utilize and Apply OATS?
• What Can OATS Provide?
• Definition of Pairwise
• Pairwise Facts
• Where to get Pairwise?
• Summary
• References
• Resourceful Websites

7
Definition of Orthogonal Array Testing Strategy

• Orthogonal Array Testing Strategy (OATS) is a
  systematical, statistical way of testing pairwise
  interactions by deriving a suitable small set of
  test cases from a large number of scenarios.

8
The OAT Theory

• OAT can be used to reduce the number of
  combinations and provide maximum coverage with a
  minimum number of test cases.
• OAT is an array of values in which each column
  represents a variable - factor that can take a
  certain set of values called levels. Each row
  represents a test case.
• In OAT, the factors are combined pairwise rather
  than representing all possible combinations of
  factors and levels.
• Orthogonal arrays are two dimensional arrays of
  numbers which possess the interesting quality
  that by choosing any two columns in the array you
  receive an even distribution of all the pairwise
  combinations of values in the array.

9
Simple OAT Example

• Consider three parameters A, B, and C.
• Parameters A, B, and C each have positive values
  1, 2, and 3.
• Testing all combinations of the three parameters
  would involve executing a total of 27 test cases.

10
Simple OAT Example (contd)

• With the way programming works, a fault will most
  likely depend on the values of two parameters,
  not three.
• The fault might occur for each of these three
  test cases A1, B1, C1, A1, B1, C2, and
  A1, B1, C3.
• Not necessary to run all 27 scenarios, 9 test
  cases is suffice.
• The 9 scenarios outlined in Table 1 (on the next
  screen) address all possible pairs within the
  three parameters.

11
Pair-wise Combination of Parameters

• All possible pairwise combinations between
  parameters A and B, B and C, and C and A are
  displayed in Table 1.

12
Orthogonal Array Example 2(More Complex)

• Orthogonal Array has 3 columns, representing 3
  variables.
• Utilize a three level-alphabet Red, Green, and
  Blue.
• Next, organize combinations into a table (next
  slide).

13
Orthogonal Array Example 2
All Combinations for Three Variables of Three Levels Each All Combinations for Three Variables of Three Levels Each All Combinations for Three Variables of Three Levels Each All Combinations for Three Variables of Three Levels Each
  A B C
1 Red Red Red
2 Red Red Green
3 Red Red Blue
4 Red Green Red
5 Red Green Green
6 Red Green Blue
7 Red Blue Red
8 Red Blue Green
9 Red Blue Blue
10 Blue Red Red
11 Blue Red Green
12 Blue Red Blue
13 Blue Green Red
14 Blue Green Green
15 Blue Green Blue
16 Blue Blue Red
17 Blue Blue Green
18 Blue Blue Blue
19 Green Red Red
20 Green Red Green
21 Green Red Blue
22 Green Green Red
23 Green Green Green
24 Green Green Blue
25 Green Blue Red
26 Green Blue Green
27 Green Blue Blue
14
Orthogonal Array Three Variables of Three
Levels Each

• For each pair of columns, AB, AC, and BC, each
  pair of colors appears exactly three times.
• To save testing effort, the appearance of each
  pair has been reduced to once.

15
Orthogonal Array Example 2
All-Pairs Array, Three Variables of Three Levels Each All-Pairs Array, Three Variables of Three Levels Each All-Pairs Array, Three Variables of Three Levels Each All-Pairs Array, Three Variables of Three Levels Each
  A B C
2 Red Red Green
4 Red Green Red
9 Red Blue Blue
12 Blue Red Blue
14 Blue Green Green
16 Blue Blue Red
19 Green Red Red
24 Green Green Blue
26 Green Blue Green
16
Why Use Orthogonal Array Testing?

• Being intelligent about which test cases you
  choose can make all the difference between
• Endlessly executing tests that just arent likely
  to find bugs and dont increase your confidence
  in the system.
• Executing a concise, well-defined set of tests
  that are likely to uncover most (not all) of the
  bugs and that give you a great deal more comfort
  in the quality of your software.

17
Advantages of OATS

• Utilizes variable pair combination
• Provides uniformly distributed coverage
• Useful for integration testing of software
  components
• Useful for testing configurable options on an
  application
• web page allowing font style
• background color
• page layout selections

18
Disadvantages of OATS

• Can only be applied at the initial stage of the
  product/process design system.
• There are some situations whereby OA techniques
  are not applicable, such as a processes involving
  influencing factors that vary in time and cannot
  be quantified exactly.
• Can be overwhelming to use the first time
  improves with use on each project.
• Arrays can be difficult to construct.

19
Selecting the Right Test Set with OATS

• OATS provides a means to select a test set that
• Guarantees testing the pair-wise combinations of
  all the selected variables.
• Creates an efficient and concise test set with
  many fewer test cases than testing all
  combinations of all variables.

20
Selecting the Right Test Set with OATS (contd)

• 3. Creates a test set that has an even
  distribution of all pair-wise combinations.
• 4. Exercises some of the complex combinations of
  all the variables.
• 5. Is simpler to generate and less error prone
  than test sets created by hand.

21
Mistakes to Avoid

• Applying OATS manually.
• Focusing the testing effort on the wrong area of
  the application.
• Using OATS for minimal testing efforts.
• Using OATS for high-risk applications.
• Picking the wrong parameters to combine.

22
Applying OATS manually

• Manual entry of OATS test parameters is time
  consuming and prone to inaccuracies.
• Utilize free orthogonal array freeware, to avoid
  manually applying OATS.
• Orthogonal freeware can generate various array
  sizes.

23
Focusing the testing effort on the wrong area of
the application

• Focus OATS on the critical areas of the
  application.
• Evaluate where applying OATS is most feasible.
• It is important to evaluate where the application
  testing effort will be most effective and
  efficient to apply.

24
Using OATS for minimal testing efforts

• There are times when OATS might ask for more test
  cases than is actually necessary.
• Determine if budget and time allows for a full
  testing effort or a good enough testing
  effort.
• If utilize OATS-generated test case combination,
  omit unnecessary tests.

25
Using OATS for high-risk applications

• Testing medical device systems or other
  life-critical applications require more than just
  OATS.
• Other testing techniques must be used to
  complement OATS.
• OATS is used to minimize the test combination
  inputs.
• Not a technique for exhaustive testing.

26
Picking the Wrong Parameters to Combine

• Understand how the application functions.
• Enlist the help of
• Subject Matter Experts (SMEs)
• Developers
• Business Analysts
• Pick the right parameter combinations based on
  feedback from SME, Developers, and BAs.

27
How to Utilize and Apply OATS?

• Steps to use OATS are outlined below
• Identify independent variables
• Determine the maximum number of values
• Find a suitable array with the smallest number of
  runs
• Map the factors and values onto the array
• Choose values for any remaining level
• Transcribe the runs into test cases

28
What Can OATS Provide?

• OATS provides a means to select a test set that
• Guarantees testing the pair-wise combinations of
  all selected variables.
• Creates an efficient and concise test set with
  fewer test cases.
• Creates a test set that has an even distribution
  of all pair-wise combinations

29
What Can OATS Provide? (contd)

• OATS provides a means to select a test set
  (contd)
• 4. Exercises some of the complex combinations of
  all variables.
• 5. Is simpler to generate and less error prone
  than test sets created by hand.

30
Definition of Pairwise Testing

• Recall Orthogonal Array Testing Strategy is a
  systematical, statistical way of testing pairwise
  interactions by deriving a suitable small set of
  test cases from a large number of scenarios.
• Pairwise testing is a type of combination testing
  method, where each pair of input parameters to a
  system (usually software algorithms), tests all
  possible combinations of the parameters.

31
Pairwise Facts

• Pairwise testing is also known as Orthogonal
  Array Testing.
• The goal of using orthogonal arrays in testing
  is
• to get the biggest bang for the testing buck
• performing the smallest number of tests likely to
  expose defects
• To keep the test scenario numbers low, the
  desired combinations are pairs of variables in
  each of their possible states, rather than
  combinations of three or more variables. This
  makes pair-wise testing a kind of subset of
  orthogonal array testing.

32
Where to get All Pairs?

• All Pairs software will test for all possible
  discrete combinations of the parameters
  identified for testing.
• All Pairs can be downloaded from
  http//www.satisfice.com/tools.shtml

33
Summary

• Orthogonal Array Testing Strategy is a
  systematical, statistical way of testing
  pair-wise interactions by deriving suitable small
  set of test cases from a large number of
  scenarios.
• OAT should be used to reduce the number of
  combinations and provide maximum coverage with a
  minimum number of test cases.
• OAT guarantees testing the pair-wise combinations
  of all selected variables.

34
Summary

• Mistakes to Avoid.
• Focusing the testing effort on the wrong area of
  the application.
• Using OATS for minimal testing efforts.
• Using OATS for high-risk applications.
• Picking the wrong parameters to combine.
• Pairwise testing is a type of combination testing
  method, where each pair of input parameters to a
  system (usually software algorithms), tests all
  possible combinations of the parameters.

35
References

• Bolton, M. (2004). Pairwise testing. Retrieved on
  December 1, 2009, from http//www.developsense.com
  /testing/PairwiseTesting.html
• Dustin, E. (2001). Orthogonally Speaking.
  Retrieved on November 17, 2009, from
  http//www.stickyminds.com/sitewide.asp?Functione
  detailObjectTypeMAGAZINEObjectId5031tthDYNt
  tsiteemailiDyn2table3gotable3go

36
References (contd)

• Harrell, J. (2001). Orthogonal Array Testing
  Strategy (OATS) Technique. Retrieved on November
  29, from http//upiia.uab.es/teach/a21291/apunts/p
  rovaOO/OATS.pdf
• Learn Software Development. (2007). Orthogonal
  Array Testing Strategy (OATS). Retrieved on
  November 21, 2009, from http//learnsoftwareproces
  ses.com/2007/08/21/orthogonal-array-testing-strate
  gy-oats/

37
References (contd)

• StickyMinds. (2001). Orthogonally Speaking.
  Retrieved on November 15, 2009, from
  http//www.stickyminds.com/sitewide.asp?Functione
  detailObjectTypeMAGAZINEObjectId5031tthDYNt
  tsiteemailiDyn2table3gotable3go
• VTB. (2008). Orthogonal Array Testing Strategy
  (OATS). Retrieved on November 10, 2009, from
  http//www.vietnamesetestingboard.org/zbxe/?middo
  wnloadcategory12674listStylelistdocument_srl
  55352

38
Resourceful Websites

• http//www.combinatorialtesting.com/clear-introduc
  tions
• http//www.isixsigma.com/library/content/c030106a.
  asp
• http//upiia.uab.es/teach/a21291/apunts/provaOO/OA
  TS.pdf
• http//www.vietnamesetestingboard.org/zbxe/?middo
  wnloadcategory12674listStylelistdocument_srl
  55352
• http//www.developsense.com/testing/PairwiseTestin
  g.html