T'Y' Chen

1
Semi-Proving an Integrated Method Based on
Global Symbolic Evaluation and Metamorphic
Testing
T.Y. Chen Swinburne University of Technology,
Australia
T.H. Tse and Zhiquan Zhou The University of
Hong Kong
(speaker)
2
Presentation Outline

• Conventional Program Testing and Proving
• Metamorphic Testing
• Our method Semi-Proving
• Summary.

3
Presentation Outline

• Conventional Program Testing and Proving
• Metamorphic Testing
• Our method Semi-Proving
• Summary.

4
Conventional Program Testing and Proving
Given a bijective function f A Program F_Sort
(a1, a2, ..., an), n ? 2 Output (a1, a2, ...,
an), such that 1. (a1, a2, ..., an) is a
permutation of (a1, a2, ..., an) 2. f (a1) ?
f (a2) ? ... ? f (an).
5
Conventional Program Testing and Proving

• Testing

1. Design test cases e.g. (2, 6, 3) for n3 2.
Run F_Sort (2, 6, 3) (6, 3, 2) 3. Check f
(6) lt f (3) lt f (2) ?
6
Conventional Program Testing and Proving

• Proving correctness

1. F_Sort terminates for any valid input 2. The
output is correct.
7
Conventional Program Testing and Proving

• Proving properties

F_Sort (a1, a2, ..., an) (a1, a2, ..., an)

Permutation.
8
Metamorphic Testing

• Metamorphic Testing
• Employing relationships between different
  executions

Fact different permutations will produce same
output F_Sort (a1, a2, a3)

F_Sort (a3, a1, a2)
Metamorphic Relation
9
Metamorphic Testing
Metamorphic Test Cases (2, 6, 3), (3, 2, 6)
Metamorphic Testing 1. F_Sort (2, 6, 3) (6,
3, 2)
No matter whether an oracle is available or
not Very useful when the oracle cannot be found.

2. F_Sort (3, 2, 6) (6, 3, 2)
PASS
10
Metamorphic Testing
Metamorphic Test Cases (2, 6, 3), (3, 2, 6)
Metamorphic Testing 1. F_Sort (2, 6, 3) (6,
3, 2)
2. F_Sort (3, 2, 6) (3, 6, 2)
Failure.
11
Presentation Outline

• Conventional Program Testing and Proving
• Metamorphic Testing
• Semi-Proving Verifying Metamorphic Relations
• Summary.

12
Semi-Proving Verifying Metamorphic Relations

• Objective
• If the program does not satisfy a metamorphic
  relation on some inputs, locate these inputs
• Otherwise prove the satisfaction of the
  metamorphic relation over all inputs.

13
Semi-Proving Verifying Metamorphic Relations

• Why called Semi?
• Proving necessary properties, which may not be
  sufficient for program correctness
• Characteristics of Semi-Proving
• Multiple symbolic executions
• Testing and proving.

14
Semi-Proving Verifying Metamorphic Relations
double GetMid (double x1, double x2, double
x3) double mid mid x3 if (x2 lt x3) if
(x1 lt x2) mid x2 else if (x1 lt
x3) mid x1 else if (x1 gt x2) mid
x2 else if (x1 gt x3) mid x1
return mid
15
Semi-Proving Verifying Metamorphic Relations

• Specification
• GetMid (X, Y, Z) returns the median of (X, Y,
  Z)
• E.g. GetMid (3, 4, 1) 3.

16
Semi-Proving Verifying Metamorphic Relations

• Verifying GetMid by Semi-Proving
• Identify a Metamorphic Relation
• GetMid ( X, Y, Z ) GetMid ( permute(X, Y, Z) )

any numbers
any permutation
17
Semi-Proving Verifying Metamorphic Relations

• Basic concepts
• Transposition
• simple permutation that exchanges two elements

(1, 2, 3)
? (2, 1, 3)
......... ?1
(1, 2, 3) ? (1, 3, 2)
......... ?2
18
Semi-Proving Verifying Metamorphic Relations

• Basic concepts
• Composition of Transpositions

A tuple (1, 2, 3) A permutation (2, 3, 1) (1, 2,
3)
(2, 1, 3)
(2, 3, 1)
19
Semi-Proving Verifying Metamorphic Relations

• Result from Group Theory
• Any permutation of (X, Y, Z) can be achieved by
  compositions of transpositions (X, Z, Y) and (Y,
  X, Z).

20
Semi-Proving Verifying Metamorphic Relations

• Purpose
• GetMid ( X, Y, Z ) GetMid ( permute(X, Y, Z) )
• Only need to verify

Any permutation.
21
Semi-Proving Verifying Metamorphic Relations

• Purpose
• GetMid ( X, Y, Z ) GetMid ( permute(X, Y, Z) )
• Only need to verify

22
Semi-Proving Verifying Metamorphic Relations

• Global Symbolic Evaluation on GetMid (X, Y, Z)
• Execute all the possible paths.

23
Semi-Proving Verifying Metamorphic Relations
double GetMid (double x1, double x2, double
x3) double mid mid x3 if (x2 lt x3) if
(x1 lt x2) mid x2 else if (x1 lt
x3) mid x1 else if (x1 gt x2) mid
x2 else if (x1 gt x3) mid x1
return mid
24
Semi-Proving Verifying Metamorphic Relations
25
Semi-Proving Verifying Metamorphic Relations
X when C1 is true GetMid (X, Y, Z) Y
when C2 is true Z when C3 is true
?
26
Semi-Proving Verifying Metamorphic Relations
?
?
PASS
C4 (Z ? X lt Y) OR (Y lt X ? Z) C5 (X lt Z lt
Y) OR (Y ? Z lt X) C6 (Z lt Y ? X) OR (X ?
Y ? Z)
27
Semi-Proving Verifying Metamorphic Relations
X when C1 is true GetMid (X, Y, Z) Y
when C2 is true Z when C3 is true
?
?
?
PASS
C4 (Z ? X lt Y) OR (Y lt X ? Z) C5 (X lt Z lt
Y) OR (Y ? Z lt X) C6 (Z lt Y ? X) OR (X ?
Y ? Z)
C1 (Y ? X lt Z) OR (Z lt X ? Y)

? Contradiction
28
Semi-Proving Verifying Metamorphic Relations
X when C1 is true GetMid (X, Y, Z) Y
when C2 is true Z when C3 is true
?
?
?
C4 (Z ? X lt Y) OR (Y lt X ? Z) C5 (X lt Z lt
Y) OR (Y ? Z lt X) C6 (Z lt Y ? X) OR (X ?
Y ? Z)
XYltZ OR ZltYX

C1 (Y lt X lt Z) OR (Z lt X lt Y)
29
Semi-Proving Verifying Metamorphic Relations
X when C1 is true GetMid (X, Y, Z) Y
when C2 is true Z when C3 is true
?
?
?
Yes. XY
PASS
C4 (Z ? X lt Y) OR (Y lt X ? Z) C5 (X lt Z lt
Y) OR (Y ? Z lt X) C6 (Z lt Y ? X) OR (X ?
Y ? Z)
XYltZ OR ZltYX

C1 (Y lt X lt Z) OR (Z lt X lt Y)
30
Semi-Proving Verifying Metamorphic Relations
X when C1 is true GetMid (X, Y, Z) Y
when C2 is true Z when C3 is true
?
GetMid (X, Z, Y)
31
Semi-Proving Verifying Metamorphic Relations

• Conclusion

X when C1 is true GetMid (X, Y, Z) Y
when C2 is true Z when C3 is true
?
GetMid (X, Z, Y)
32
Semi-Proving Verifying Metamorphic Relations

• Conclusion

X when C1 is true GetMid (X, Y, Z) Y
when C2 is true Z when C3 is true
?
GetMid (X, Z, Y)
33
Semi-Proving Verifying Metamorphic Relations

• Conclusion

X when C1 is true GetMid (X, Y, Z) Y
when C2 is true Z when C3 is true
GetMid (X, Z, Y)
34
Semi-Proving Verifying Metamorphic Relations

• Conclusion

X when C1 is true GetMid (X, Y, Z) Y
when C2 is true Z when C3 is true
GetMid (X, Z, Y)
Composition of transpositions
GetMid (X, Y, Z) GetMid ( Permute(X, Y, Z) )
35
Semi-Proving Detecting Program Faults

• Detecting Program Faults

36
double GetMid (double x1, double x2, double
x3) double mid mid x3 if (x2 lt x3) if
(x1 lt x2) mid x2 else if (x1 lt
x3) mid x1 else if (x1 gt x2) mid
x2 else if (x1 gt x3) mid x1
return mid
37
Semi-Proving Detecting Program Faults
Verify GetMid (X, Y, Z) GetMid (X, Z, Y)
X when Y ? X lt Z
Y when (Z lt Y ? X ) OR (Y ? Z AND X ? Z)
38
Semi-Proving Detecting Program Faults
Verify GetMid (X, Y, Z) GetMid (X, Z, Y)
X when Y ? X lt Z
Y when (Z lt Y ? X ) OR (Y ? Z AND X ? Z)
?(YXltZ) OR (YltXltZ)
39
Semi-Proving Detecting Program Faults
Verify GetMid (X, Y, Z) GetMid (X, Z, Y)
X when Y ? X lt Z
Y when (Z lt Y ? X ) OR (Y ? Z AND X ? Z)
failure
?
?
?(YXltZ) OR (YltXltZ)
Can identify all the failure-causing inputs.
Failure-causing input
40
Summary

• A proving technique all the paths
• A testing technique
• failure-causing inputs
• selected path(s)
• Characteristics
• Metamorphic relations
• Multiple symbolic executions
• Employing global symbolic evaluation and
  constraint solving.

41
Questions are welcome