a generic testsystem

1
a generic test-system

• Pieter Koopman, Artem Alimarine, Jan Tretmans,
  Rinus Plasmeijer
• Nijmegen, NL

2
overview

• testing
• why
• what will be tested
• how do we test
• by an automatic system
• generic testing by a generic system
• automatic and systematic generation of test data
• research question can we make such a system
• conclusions / related future work

3
why testing

• even in functional programming we make errors
• even errors that the type system does not spot
• to improve quality and confidence the software
• proving is often too much work, or not feasible
• manual testing is
• much work (up to 50 of project cost)
• dull
• to be repeated after changes
• ? make an automatic system
• encourages writing specifications
• makes testing easier, faster and more reliable

4
what will be tested

• there is a rich palette of quality aspects
• suitability validating
• obeying the specification functional testing
• efficiency
• ...
• we restrict ourselves to obeying the
  specification, functional testing
• specification can be
• executable specification inefficient but
  obviously correct
• reference implementation older version
• relation between input and output
• ...

5
specification

• specification by functions in Clean
• prop_DeMorgan Bool Bool -gt Boolprop_DeMorgan
  x y x y not (not x not y)
• semantics ? x, ? y x y not (not x not
  y)
• prop_DropDrop Int Int Int -gt
  Boolprop_DropDrop m n xs drop n (drop m xs)
  drop (nm) xs
• prop_Sqrt Real -gt Propertyprop_Sqrt x
  xgt0.0 gt let r sqrt x in rr x
• monomorph in order to generate test data
• for polymorphic functions this tests all types
• for overloaded functions the instance does matter

6
how basic idea

• generate test data, finite and ? ?
• check property for all these values
• test prop and evaluate prop x \\ x lt-
  generate "Passed"
  "Counter-example found"
• requirements
• one generate for all types
• information about the counter-example
• information about test data used
• what test data should be generated ?

7
what test data?

• random ? (as in QuickCheck by Claessen
  Hughes) easy to implement easy to use works
  pretty well for small programs- are all relevant
  cases covered?- duplicates (which are useless)
• systematic ! coverage of standard cases
  (guaranteed) Int 0, 1, ... List Int , 0
  , 1 , ... no duplicates more confidence,
  better results- harder to implement

8
systematic generation of test data

• what do we want?
• finite types Bool True False Colour
  Red Yellow Blue
• all values can and should be tested
• very large types (always basic types)Int, Real,
  Char, String, ...
• common border values (like 0, 1) and random
  values
• recursive typesList a Nil Cons a (List
  a)Tree a Tip Node a (Tree a) (Tree a)
• always handled by recursive functions
• systematic generation from small to large

9
systematic generation 2

• basic types (Int, Real, ...) are handled
  separately
• for all other types systematic generation from
  small to large is better than random
• covers interesting cases
• avoids duplicates
• how to define one general generate function ?
• one function for all types
• no new instance for each new type
• systematic generation
• no duplicates
• small to large
• use generics !

10
what are generics

• uniform tree representation of types
• conversion between actual type and tree
  representation by system
• tree representation can be manipulated in generic
  functions
• we use standard generics
• See Hinze, Alimarine, ...
• enables us to define one generate function
• systematic generation of generic trees
• automatic conversion of trees to needed types

11
generic trees

• UNIT UNIT // tip of tree EITHER a b
  LEFT a RIGHT b // choice PAIR a b PAIR
  a b // product type
• example Colour Red Yellow Blue
• generic representation of this type Colour
  EITHER UNIT (EITHER UNIT UNIT)
• generic representation of constructorsRed
  LEFT UNITYellow RIGHT (LEFT UNIT)Blue
  RIGHT (RIGHT UNIT)

12
generic trees 2

• representation of (which is Nil)
• LEFT UNIT note same representation as Red!
• representation of Red ( Cons Red Nil)
• RIGHT (PAIR (LEFT UNIT) (LEFT UNIT))

13
generic functions

• Basic idea
• instead of a function F A -gt B for each A and
  B
• we define a generic function F
• we implement F as GenToB o F o AToGen
• transformations are generated by the compiler

14
Example a generic equality

• generic eq a a a -gt Bool
• instances for generic type eqUNIT UNIT UNIT
  True
• eqPAIR eqx eqy (PAIR x1 y1) (PAIR x2 y2)
  eqx x1 x2 eqy y1 y2
• eqEITHER eql eqr (LEFT x) (LEFT y) eql
  x yeqEITHER eql eqr (RIGHT x) (RIGHT y)
  eqr x yeqEITHER eql eqr e1 e2
  False
• eqInt n m n m // similar for other basic
  types
• for each type T we either use the generic
  versionderive eq T
• or define an own instanceeqT t1 t2 ...

15
function generate

• using generics we can define one generate
  function
• algorithm
• systematic generation of generic trees
• automatic conversion to desired type

16
systematic generation of generic trees

• preorder traversal
• remember current tree
• first all subtrees starting with LEFT
• then all subtrees starting with RIGHT
• works fine for Colour, Colour, ...
• problems if left generic subtree is infinite
• Tree a Tip Node a (Tree a) (Tree a)using
  T CTree T generates Tip, Node C Tip
  Tip, Node C (Node C Tip Tip) Tip, Node C (Node
  C (Node C Tip Tip) Tip) Tip, ...never generates
  Node C Tip (Node C Tip Tip),...
• T generates , , C, C, C,
  C, C, C,...never generates , , ,
  C, C, ,...

17
generation of generic trees 2

• breadth first traversal
• order trees with respect to number of LEFT/RIGHT
  nodes, or constructors
• yield trees with increasing depth
• within same depth left to right
• problems
• not easy to generate trees with increasing depth
  efficiently
• not every tree represents a valid data value
• adding a constructor requires adding its
  arguments
• large administration needed

18
generation of generic trees 3

• preorder does not work with left recursion
• breath first is nasty
• use mixed approach
• remember generated branches of tree
• extend tree for next test-data item
• at each EITHER node choose randomly between
  extending LEFT or RIGHT branch
• Properties
• systematic
• no duplicates
• interesting cases occur soon (with high
  probability)
• also bigger values earlier in list of test-data

19
testing

• administrate arguments used in a record
• Result ok Maybe Bool, args String
  
• define class Testable
• class Testable awhere evaluate a RandomStream
  Result -gt Result
• instance Testable Boolwhere evaluate b rs result
  result ok Just b
• instance Testable (a-gtb) Testable b TestArg a
  where evaluate f rs result let (rs, rs2)
  split rs in forAll rs2 f result (generate
  rs)forAll rs f r list diagonal apply
  (genRand seed) f a r \\ alt-list seed lt- rs
  apply rs f a r evaluate (f a) rs r args
  show ar.argsuse generic functions generate
  and show

20
conditional tests

• prop_Sqrt Real -gt Propertyprop_Sqrt x
  xgt0.0 gt let r sqrt x in rr x
• implementation
• Property Prop (RandomStream Result -gt
  Result)
• (gt) infixr 0 Bool p -gt Property Testable
  p(gt) b p
• b Prop (evaluate p) // continue
  testing Prop (\rs r r) // stop testing,
  dummy result
• instance Testable Property
• where evaluate (Prop p) rs result p rs result

21
example results

• prop_Sqrt Real -gt Propertyprop_Sqrt x
  xgt0.0 gt let r sqrt x in rr x
• counter-example found after 2 tests .1191576
• prop_DropDrop Int Int Int -gt
  Boolprop_DropDrop m n xs drop n (drop m xs)
  drop (nm) xs
• counter-example found after 13 tests -1 1 0,0
• prop_RevRev Int -gt Propertyprop_RevRev xs
  classify (isEmpty xs) xs ( reverse (reverse
  xs) xs )
• Passed 1000 tests 1 (0.1)
• prop_DeMorgan Bool Bool -gt Boolprop_DeMorgan
  x y x y not (not x not y)
• counter-example found after 3 tests False True
• with correction specification Passed 4 tests

22
conclusions

• a test system is useful
• writing specifications is encouraged
• testing improves quality and confidence
• a generic test system is useful
• property is arbitrary Clean function
• system applies predicate to test-data
• system generates test-data automatically
• use generics to generate, show and compare values
• systematic generation of test data
• no duplicated tests
• covers interesting cases
• stops if all cases are tested

23
related work

• for FPL QuickCheck
• advantages of our generic approach
• one generate function for all types
• no user defined instances needed
• generate
• show
• equal
• systematic generation of data
• no duplicates
• covers interesting cases
• stops when all cases are tested

24
future work

• handling ?
• generating better functions
• prop_Map (Int-gtInt) (Int-gtInt) Int -gt
  Boolprop_Map f g xs map f (map g xs) map (f
  o g) xs
• easier user control over test-data
• better info about test-data used
• case studies
• GUI
• testing application specified in Clean but
  written in some other programming language
• integration with proof-system
• ...