Short Vs. Long

1
Short Vs. Long - Size Does Make a Difference
Alan Hartman, Shmuel Ur, Avi Ziv ur_at_il.ibm.com Ver
ification Technologies IBM Research Lab in Haifa
2
Motivation

• The verification process consists of
• Creating test requirements
• Generating tests that cover them
• Possible ways to create tests that cover
  requirements
• Short and simple tests, each for a single
  requirement
• Long tests that combine several requirements

3
Advantages of Short Tests

• Short tests have many advantages
• Easier to write
• Easier to execute
• Easier to debug
• Easier to maintain, automate and document
• Easier to understand
• But do they find bugs?

4
Advantages of Long Tests

• Long tests also have advantages
• Less set up time
• Smaller total execution time
• Create unintended intermediate scenarios
• Do they find more bugs?!

5
The Prevailing Wisdom - Folklore

• Longer tests are better at finding bugs because
  in a long test many unintended side effects occur
• However short test are easier to run, create,
  automate, maintain and debug
• Finding the middle ground - use tests that are
  not short but also are not very long

6
Comparing Long and Short Tests

• Experiment steps
• Create test requirements
• Generate two suites that cover these requirements
• Short and simple tests for a single requirement
• Long tests that cover several requirements
• Compare the quality of the suites

7
Test Plan Short and Long
Test 1
Test 2
Test 3
Test 4
Test 5
Req 1
x
x
Req 2
x
x
Req 3
x
x
Req 4
Req 5
x
Req 6
x
Req 7
Req 8
x
8
How to Compare Quality of Tests Suites?

• How they achieve their objective
• Number and importance of bugs found
• Coverage of "unintended" requirements
• Divide the test requirements into two sets A and
  B
• Create a set of tests SA for A
• Check how SA covers B

9
Experiment Details

• Create an abstract FSM model for the applications
  
• Create several sets of coverage targets
• Use traversal tools to generate sets of tests for
  each target
• Check coverage of the sets on the other targets,
  simulating "unintended" requirements

10
GOTCHA-TCBeans Software Testing Methodology
Test Scripts
Model Test Driver Interface
TCBeans A2C
Scripting Bugs
Tester
Abstract Test Suite
GOTCHA Behavioral Model
Specs
Tester
GOTCHA
Code Bugs
Spec Bugs
Tester
Test Results
Model Application Interface
Interface Bugs
TCBeans A2E
Code Bugs
11
Model Driven Testing

• Start from specification
• That's an improvement already
• Build behavioral model and test interface
• Team testers with developers
• Find early bugs - before code exists
• The model is the test plan
• Model generates the test suite
• Coverage is guaranteed
• Execute the test suite
• Compare model with product
• Find bugs

12
Experiment - Processor Pipeline
Dispatch logic
Data Fetch
Data Fetch
Data Fetch
Data Fetch
Execute
Execute
Execute
Execute
Write Back
Write Back
Write Back
Write Back
Simple Arithmetic
Complex Arithmetic
L/S
Branch
13
Coverage of Short and Long Tests
14
Suite Length of Short and Long Tests
15
Intuition
16
Conclusions

• Longer tests seem better than short ones
• Better coverage of "unintended" requirements in
  less cycles
• More experiments are needed
• On other types of coverage models
• Code coverage - (statement, branch, etc.)
• Functional coverage
• Ability to find bugs
• Effects of random test generators
• Open question
• What is the right test length

17
Next Experiment

• Experiment
• Create short/long tests that cover part of an
  application
• Repeat until coverage reaches set goal
• Measure coverage on the rest of the application
• Is this important? Why not just get good tests
  for the entire application?

18
Contributions

• Suggesting a quantitative method for comparing
  the quality of suites of tests
• Validating folklore about importance of test
  length
• Research into the optimal length of tests, mainly
  in domains in which automated test generators
  exist

19
(No Transcript)