White (Clear) Box Testing

1
White (Clear) Box Testing

• Verilog code is available to help generate tests
• Easier to detect additive design errors
• Possible to automate test generation

a in1 in2 b 0 c 0 while (c lt a) c
c in1 if (c lt in2) out a b else out
a c
Control-Dataflow Graph (CDFG)
2
Code Coverage Metrics

• Coverage metric is a numerical measurement of the
  adequacy of an arbitrary test set for validation.
• High coverage is a goal for test generation
• The measure is based on code execution
• Statement coverage - Fraction of statements
  executed
• Branch coverage - Fraction of branch directions
  executed
• Path coverage - Fraction of control-flow paths
  executed

3
Code Coverage, Statement Coverage

• Ensure that the code is completely covered
  during testing
• Statement Coverage Requires that all statements
  are executed

a in1 in2 b 0 c 0 while (c lt a) c
c in1 if (c lt in2) out a b else out
a c
0,0
T
F
-1,1
T
F
4
Branch Coverage

• Branch Coverage Requires that all branch
  directions are executed
• 2 branches, 4 possibilities, 3 are executed, 75
  branch coverage

a in1 in2 b 0 c 0 while (c lt a) c
c in1 if (c lt in2) out a b else out
a c
0,0
T
F
-1,1
T
F
5
Test Sequence for Higher Coverage

• 100 statement and branch coverage
• High coverage goal directs test generation

a in1 in2 b 0 c 0 while (c lt a) c
c in1 if (c lt in2) out a b else out
a c
1,0
-1,1
T
F
T
F
6
Path Coverage

• Each pattern covers a different control path
• Infinite number of paths with a loop

7
Domain (Boundary) Coverage

• Each conditional predicate defines partitions the
  input space
• A small error may move the partition

foo (in1, in2) if (in1 lt in2) out 1
else out 0

• What if in1 lt in2 should be in1 lt in2?
• What if in1 or in2 has a slightly incorrect value?

8
Domain Test Goal

• Test with input data on the boundary and just on
  either side of the boundary

in2
out1
foo (in1, in2) if (in1 lt in2) out 1
else out 0
out0
2
1
1
in1

• Test 1 in1, in2 1,1 - False, on the boundary
• Test 2 in1, in2 1,2 - True, just off the
  boundary

9
Domain Test Example

• Need to satisfy 2 conditions for each predicate
• True and near-boundary
• False and near-boundary

• Test 1 in1, in2 0,0
• Sets c 0, and a 0
• False and on-boundary for c lt a
• False and on-boundary for c lt in2

a in1 in2 b 0 c 0 while (c lt a) c
c in1 if (c lt in2) out a b else out
a c

• Test 2 in1, in2 1,0
• Sets c 0, and a 0
• True and near-boundary for c lt a
• c set to 1 in loop
• False and on-boundary for c lt in2

• 3 of 4 conditions satisfied, 75 coverage

10
Domain Test Complications

• Non-standard Domain Boundaries

Piecewise Linear Boundaries Test each linear
piece separately
(in2 gt in1) and (in1 gt 2)
Higher Degree Boundaries No standard solution
(in2 gt in12)
11
Coverage Metrics as Test Constraints

• Coverage metrics provide constraints which can be
  used to evaluate the completeness of testing
• Testing is complete when all constraints are
  satisfied
• Uses of Test Constraints
• Evaluation of Test Sequence - Does this test
  sequence satisfy all constraints?
• Directing Test Generation - Given these
  constraints, create a test sequence which
  satisfies them.

12
Statement Coverage Constraints
No Constraints
Local Input
c lt in2
in2gt0 and in1lt0
Local Input
No loop
(in2 lt 0) and (in1 in2 lt 0)
or (in1 in2 gt 0)
c gt in2
Loop
13
Tests for Statement Coverage

• 0,0 satisfies test constraints for outac
• -1,1 satisfies test constraints for outab

0,0
T
F
-1,1
T
F
(in2 lt 0) and (in1 in2 lt 0)
(in2 gt 0) and (in1 lt 0)
14
Optimism in a Coverage Metric

• 100 statement coverage does not guarantee error
  detection
• Test constraints may not be the same as detection
  constraints

• Test 0,0 executes erroneous statement but does
  not detect the error
• Detection constraint includes a - c ! a c

15
Pessimism in a Coverage Metric

• 100 coverage may not be possible
• Fault detection conditions may not be satifiable
• Undetectable faults are redundant, can be ignored
• Identifying redundant faults in NP-complete

• This path cannot be executed
• Sub-100 coverage is misleading