QuickCheck: Specificationbased Random Testing

1
QuickCheckSpecification-based Random Testing

• Koen Lindström Claessen
• Chalmers University of Technology
• Gothenburg, Sweden
• Trends in Testing, August 2004, Skamania Lodge

2
Formal Methods are too expensive
3
Pragmatic Verification
Formal Methods are too expensive
Properties, Formal Models, ...
Formal Specification
Formal Verification
Informal Verification
Model Checking, Theorem Proving, ...
Test, Symbolic Simulation, Sanity Checks, ...
4
Pragmatic Verification

• Formal specification of properties
• Forces people to think in new ways
• Increases understanding of system under test
• Minimize difficulty in writing
• Cheap, informal checks
• Formal spec usually wrong
• One MUST provide feedback
• Finding bugs

5
Haskell

• Purely Functional Programming Language
• No side effects
• Higher-order
• Functions can be created and passed as arguments
• Lazy
• Evaluation order is unspecified and unpredictable
• Advanced type system
• Polymorphism, overloading, ...

6
Typical FP Applications

• Symbol manipulation
• Compilers
• Theorem provers
• Analyzers
• Structured-data processing
• XML, ...
• Document scripting
• Domain-specific computing
• Complex algorithms

7
QuickCheck

• Specification of properties of programs
• Simple spec. language
• Formal, logical meaning
• Operational meaning
• Property testing
• Random test
• Test-data generators
• Cheap! So its done often

Tool automates what is easy, tool makes easy what
is hard
Hughes Claessen
8
On Random Testing

• Works very well at a fine grain
• Purely functional all dependencies are there
• Test-data generators
• Built-in default generators
• Default generators for user-types
• User-controllable quantification

9
QuickCheck (II)

• Embedded language
• Everything is a library
• Light-weight
• Natural to specify properties about X in X
• No need to learn new language
• No need for new tools

10
Example Words
words String -gt String unwords String
-gt String prop_Words s unwords (words s) s
GHCigt quickCheck prop_Words
Falsifiable, after 12 successful
tests C\t\138ÚØu\128a½Á
Generating random values small ones first
11
Example Words (II)
built-in QuickCheck operator
prop_Words s all (not . isSpace) s gt
unwords (words s) s
GHCigt quickCheck prop_Words OK, passed 1000 tests.
1000 tests that satisfied the condition
12
Example Insert
insert Ord a gt a -gt a -gt a insert
... prop_Insert x xs ordered xs gt
ordered (insert x xs)
GHCigt quickCheck prop_Insert OK, passed 1000
tests.
13
Example Insert (II)
prop_Insert Int -gt Int -gt Property prop_Inser
t x xs ordered xs gt collect (length xs)
ordered (insert x xs)
Monitoring data coverage
14
Example Insert (III)
GHCigt quickCheck prop_Insert OK, passed 1000
tests. 39 1 23 0 18 2 7 3 5 4 4 5 2
7 2 11
15
Example Insert (IV)
custom generator orderedLists Gen Int
prop_Insert Int -gt Int -gt Property prop_Inser
t x forAll orderedLists \xs -gt ordered
(insert x xs)
built-in operator forAll Gen a -gt (a -gt
Property) -gt Property
16
Example Model Finder
models Formula -gt Model models ...
17
Soundness Completeness
built-in generator elements a -gt Gen a
prop_Sound f forAll (elements (models f))
\m -gt m f prop_Complete f forAll
(valuations (names f)) \v -gt v f gt
v elem models f
custom generator valuations Name -gt Gen
Model
18
Default Generators
in QuickCheck
class Arbitrary a where arbitrary Gen
a instance Arbitrary Int where arbitrary
... instance Arbitrary a gt Arbitrary a where
arbitrary ... instance Arbitrary Formula
where arbitrary ...
manually by the programmer
19
Generating Structured Data

• Frequences for constructors
• Not enough, will not always terminate!
• Limited by a size
• Be careful not te create size dependencies!
• Can start with small tests
• Random seeds
• Reproducibility
• Splitting seeds ? No theory!

20
Compiler/Interpreter
data Imp Skip Var Expr
If Expr Imp Imp Imp gt Imp
... obey Imp -gt Trace compile Imp
-gt Instr exec Instr -gt Trace
21
Congruence
For observability
approx Int -gt Trace -gt Trace -gt Bool approx 0
_ _ True approx n t1 t2 ... prop_Congruenc
e Int -gt Imp -gt Property prop_Congruence n p
n gt 0 gt approx n (obey p) (exec
(compile p))
22
A Bug in the Interpreter
... run (If e p q) s case eval e s of Log
True -gt run q s Log False -gt run p s _
-gt (Crash, s) ...
23
QuickChecking
GHCigt quickCheck prop_Congruence
Falsifiable, after 35 successful tests 24 If
(Val (Log False)) (While (Uno Minus (Duo Div (Val
(Num (-17))) (Uno Minus (Duo Sub (Uno Minus (Duo
Or (Uno Not (Duo Less (Uno Not (Var "t")) (Duo
And (Var "o") (Val (Log False))))) (Duo Or (Duo
Div (Duo Mod (Val (Log False)) (Var "p")) (Duo Eq
(Var "e") (Val (Num (-24))))) (Var "a")))) (Duo
Sub (Duo Less (Uno Minus (Var "i")) (Uno Not (Duo
Less (Var "m") (Val (Num 31))))) (Duo Div (Var
"f")
24

(Var "b")) (Val (Log True)))) (Duo Div (Uno Minus
(Val (Num 34))) (Val Wrong))) (Uno Not (Duo Mod
(Duo And (Uno Not (Duo Less (Var "o") (Var "f")))
(Uno Not (Uno Minus (Val (Log True))))) (Duo Div
(Uno Not (Uno Not (Var "l"))) (Var "i"))))) (Uno
Minus (Var "t")))) (Print (Duo Eq (Uno Not (Duo
Eq (Duo And (Duo Div (Uno Minus (Val (Log
False))) (Uno Minus (Duo Less (Val Wrong) (Val
(Num (-33)))))) (Duo Mod (Var "k") (Duo And (Var
"j") (Duo And (Val (Num 28)) (Var "g"))))) (Duo
Add (Uno Minus (Var "s")) (Var "n")))) (Var
"b"))))) (If (Duo Eq (Duo LessEq (Duo Sub (Duo
Div (Duo Mul (Duo Mod (Val (Num 19)) (Var "k"))
(Uno Not (Val (Log False)))) (Uno Minus (Uno Not
(Var "f"))))
25

(Duo Sub (Duo Sub (Var "h") (Duo Or (Var "q")
(Var "f"))) (Uno Not (Duo Less (Var "b") (Var
"f"))))) (Duo Mod (Duo Eq (Val (Num 0)) (Uno
Minus (Var "t"))) (Uno Not (Var "b")))) (Duo Mod
(Uno Minus (Duo Mul (Uno Not (Duo And (Uno Not
(Var "p")) (Uno Not (Var "o")))) (Val (Log
True)))) (Duo Mul (Var "e") (Duo Div (Var "r")
(Uno Minus (Uno Minus (Var "o"))))))) (While (Duo
Div (Val Wrong) (Uno Not (Var "b"))) (Print (Duo
Less (Uno Minus (Duo Eq (Var "k") (Duo Or (Duo
Div (Duo Eq (Duo Eq (Val Wrong) (Val (Log
False))) (Var "p")) (Var "t")) (Duo Eq (Duo Less
(Duo Div (Var "e") (Var "h")) (Duo Mod (Val (Log
True)) (Var "k"))) (Duo Less (Duo And (Var "c")
(Val (Num (-27)))) (Var "q")))))) (Var "q"))))
(If (Uno
26

Minus (Val (Num 20))) ("n" (Duo Mod (Duo
LessEq (Uno Not (Duo Eq (Duo And (Duo Add (Uno
Not (Val Wrong)) (Duo Eq (Var "a") (Var "k")))
(Val (Num 27))) (Uno Not (Duo Eq (Duo Add (Var
"p") (Var "f")) (Duo Mod (Val (Log False)) (Var
"j")))))) (Duo Less (Val (Log True)) (Duo Sub
(Duo And (Var "p") (Duo LessEq (Var "e") (Var
"h"))) (Duo Less (Duo Mul (Var "j") (Var "i"))
(Duo Eq (Var "p") (Var "h")))))) (Val (Num 34))))
("n" (Duo Mul (Duo Or (Duo Mul (Duo Add (Duo
Mul (Duo Add (Var "h") (Var "e")) (Val (Num
(-1)))) (Duo Or (Duo And (Var "t") (Var "b"))
(Duo Or (Var "k") (Var "p")))) (Uno Minus (Duo
Less (Duo Sub (Var "m") (Var "s")) (Duo Sub (Val
(Log False)) (Val (Log True))))))
27
Summing up

• Random testing can be effective
• Compare with enumerating cases
• Manual test-data generators
• Found counter examples can be hard to understand
• Starting small, then increasing helps
• Retesting helps
• Need a better solution though

28
Shrinking
After a counter example is found, shrink it
syntactically in order to find smaller counter
examples
Gill, Runciman Claessen
29
Extending Class Arbitrary
class Arbitrary a where arbitrary Gen a
shrink a -gt a shrink _
30
Shrink Instances (products)
instance (Arbitrary a, Arbitrary b) gt
Abritrary (a,b) where arbitrary ...
shrink (x,y) (x,y) x lt-
shrink x (x,y) y lt- shrink y
one-step shrinking
31
Shrink Instances (recursive)
instance Arbitrary a gt
Abritrary a where arbitrary ...
shrink shrink (xxs) xs
xxs x lt- shrink x
xxs xs lt- shrink xs
32
Shrinking Iteratively
locally smallest
(nested quantification)
33
Using Shrink
GHCigt quickCheck prop_Words Falsifiable, after 9
tests (shrunk 12x) "\v"
GHCigt quickCheck prop_Congruence Falsifiable,
after 39 tests (shrunk 11x) 6 If (Val (Log
False)) Skip (Print (Val (Num 3)))
34
Summary QuickCheck Logic

• Forall-quantification
• forAll gen (\x -gt p)
• Implication
• b gt p
• Non-logical test data monitors
• collect x p

35
Experience

• Successful
• Used in big projects
• One QC module for every Haskell module
• Have to be careful with random testing
• Known pitfalls
• Typically, bugs occur
• 50 in program
• 50 in specification

36
Kind of Properties

• Powerful language
• Simple unit tests
• Equivalence checks
• Sanity checks
• Abstract datatypes
• Algebraic specification
• Refinement
• ...

37
QuickCheck Today

• Ships with all major Haskell compilers
• GHC, NHC, Hugs
• Is very popular
• Reports weekly
• Industry
• Adapted for other languages
• ML, Mercury, OCaml, Erlang, ...
• C, Java

38
Current Work

• Writing test-data generators
• Polytypic, generic programming?
• Reflection helps
• Integration with debugging tool Hat
• Trace browser
• Test, then retest using Hat
• Point coverage

39
Current Work (II)

• Mix with symbolic evaluation
• Already done for hardware
• New datastructure BTD
• Symbolic bundling of structural test cases
• Combine with theorem proving
• Cover project at Chalmers
• Prove top-level theorems
• Test boring and obvious lemmas
• Test as sanity check
• Prove soundness/completeness of generators

40
Current Work (III)

• Testing Logic
• Test A. Test A ? B. We now know as much as
  testing B?
• Linear logic flavour
• A (x) B
• A B