Integrated Automated Test Case Generation and Static Analysis

1
Integrated Automated Test Case Generation and
Static Analysis

• Prof. Dr. Jan Peleska
• Centre for Computing Technologies, University of
  Bremen, Germany
• Dr. Ing. Cornelia Zahlten
• Verified Systems International GmbH, Bremen,
  Germany

2
Motivation ...

• ... for an integrated approach to automated
  module testing and static analysis
• Verification specialists perspective
• static analysis gives insight with respect to
  useful test cases and expected module behaviour
• Module testing can help to verify or falsify
  fault hypotheses found in static analysis
• Tool builders perspective
• Algorithms needed for automated test case
  generation are also useful for automated static
  analysis and vice versa
• Tool capabilities required are illustrated by
  means of RT-Tester, developed by Verified
  Systems

3
Overview

• The objectives of module testing static
  analysis
• The tool capabilities required
• Fundamental techniques
• Structural test case generation enables
  functional test case generation
• Tool architecture
• Static analysis by abstract interpretation
• Test data generation
• Optional integration of techniques into
  model-driven development cycle

4
Objectives of Module Testing

• Given a functional specification of the required
  module behaviour, find test cases such that
• every aspect of functional requirements is
  checked at least once (functional testing)
• the modules code is fully covered according to
  the applicable coverage requirements (structural
  testing), while again checking compliance with
  functional requirements
• the module is robust against illegal inputs,
  fulfils performance requirements, ...
  (non-functional tests)

5
Objectives of Static Analysis

• Verify the modules conformance with
  non-functional requirements
• Absence of run-time errors, e.g.
• Correctness of memory access (array boundaries,
  pointer utilisation, string handling, memory
  copies)
• Well-definedness of arithmetic operations
• Absence of unintended endless loops
• Absence of unreachable code
• Module complexity is acceptable (SW metrics ...)
• Module code conforms to coding standards
• ...

6
The Tool Capabilities Required

• Capability 1 Module specification support
• Capability 2 Stub specification support
• Capability 3 Specification support for
  Module-internal assertions
• Capability 4 Automated test data generation for
  structural coverage
• Capability 5 Automated test data generation for
  functional coverage
• Capability 6 Run-time error detection
• Capability 7 Advanced debugging support

7
Side Remark The Objectives of Automation

• The tool capabilities listed above allow
  verification specialists to focus on their most
  important tasks
• Specify expected module behaviour
• Specify meaningful conditions about the
  environment behaviour for module testing, this
  is expressed by preconditions and stub
  specifications
• Specify module-internal assertions
• Verify / falsify potential failures
• Identify error locations in module code

8
Capability 1 Module specification support (1)

• The RT-Tester tool allows module specification by
  means of
• Preconditions Logical conditions about the legal
  input parameters, global variable / object
  attribute values (pre-states) to be met when
  calling the function / method
• Postconditions Logical conditions about the
  return values, output parameters and resulting
  global variable / object attribute values
  (post-states) on function / method return

9
Capability 1 Module specification support (2)

• Example of Unit Under Test (UUT) specification
• with RT-Tester
• double globx
• _at_uut double f(double x, double y, int i)
• _at_pre 0 lt x and x lt 100 and
• -10 lt y and exp(y) lt x and
• 0 lt i and i lt 10
• _at_post _at_rttAssert( globx globx_at_pre )
• if (-10 lt y and exp(y) lt x )
• _at_rttAssert( f 1/(x-exp(y)) )
• else _at_rttAssert( f lt 0 )

10
Capability 1 Module specification support (3)
Keywords for module specification, pre- / post
conditions

• double globx
• _at_uut double f(double x, double y, int i)
• _at_pre and exp(y) lt x
• _at_post _at_rttAssert( globx globx_at_pre )
• if (-10 lt y and exp(y) lt x )
• _at_rttAssert( f 1/(x-exp(y)) )
• else _at_rttAssert( f lt 0 )

Conditions may contain arbitrary
parameter/variable relations
References to variable pre-state
Checked only if condition -10 lt y and holds
Checked only if not(-10 lt y and exp(y) lt x)
11
Capability 2 Stub specification support (1)

• Suppose UUT f() calls sub-function double
  g(double w)
• _at_stub double g(_at_inout double w)
• _at_assert w and w gt 0
• _at_constraint g gt (w)_at_pre and
• (w)_at_pre lt w

Keyword for input/output reference parameter
Keyword for stub declaration
Condition to be checked whenever UUT calls g()
signal UUT failure if violated
Test data generator only generates data
satisfying constraint
12
Capability 2 Stub specification support (2)

• Further stub specification capabilities
• Number and sequence of stub calls performed by
  UUT can be referenced in postconditions
• Pre- / poststates of stub calls can be stored and
  referenced in postconditions
• Stub body may also be explicitly programmed
  test data generator takes explicit stub code into
  account

13
Fundamental Techniques Tool Architecture
Memory model allows evaluation of array / pointer
handling
Selection of paths through the program /
specification model
(1) C/C interpretation semantics (2) Abstract
interpretation semantics
Internal encoding of programs and specifications
Hierarchic Hybrid Transition Systems
Solution of path constraints results in concrete
test data
14
Fundamental Techniques Structural Test Case
Generation Enables Functional Test Case
Generation

• Consider UUT f()
• Assume f() is to be tested against precondition
  P(v)
• Assume expected results are specified by
  postcondition
• Q(v,v_at_pre)
• (C_1(v,v_at_pre) ? Q_1(v,v_at_pre))
• and and (C_k(v,v_at_pre) ? Q_k(v,v_at_pre))

15
Structural coverage of augmented function f_aug()
results in functional coverage of f()

1. void f_aug()
2. if ( P(v) ) // Test data meets precondition
3. f() // UUT is invoked
4. if ( C_1(v,v_at_pre) )
5. // First functional feature has been tested
6. assert(Q_1(v,v_at_pre))
7. if ( C_2(v,v_at_pre) )
9. else
10. f() // Robustness test

16
Static Analysis by Abstract Interpretation (1)

• Static analysis derives program properties from
  program abstractions
• Suppose you wish to analyse C/C-function/method
  f() and prove property P about f().
• Instead of proving P directly, using C/C
  operational semantics and analysing all possible
  execution states of f(), we analyse an abstracted
  function A(f) of f() and an abstracted property
  A(P) such that
• (A(f) satisfies A(P)) ? (f() satisfies P)

17
Static Analysis by Abstract Interpretation (2)

• Static analysis derives program properties from
  program abstractions
• If property
• (A(f) satisfies A(P)) ? (f() satisfies P)
• holds then A(f) is called an abstract
  interpretation of f().
• For static analysis of functions/methods, A(f)
  often
• keeps the same control structures as f()
• but operates on abstracted program variables

18
Static Analysis by Abstract Interpretation (3)

• Using abstract interpretations speeds up the
  analysis process, but false alarms may occur,
  that is
• (A(f) satisfies A(P)) does NOT hold, but
• (f() satisfies P) holds
• Therefore, if abstract input data A(x) can be
  found such that
• A(f)(A(x)) runs into an abstract state
  satisfying not(A(P))
• we look for concrete input data x such that f(x)
  runs into a state satisfying not(P)

19
Abstract Interpretation Example Interval Analysis

• Interval analysis uses interval ranges for
  variable valuations instead of concrete values
• Instead of investigating concrete input values x
  and calculating concrete results, say, y f(x),
  we use
• input interval ranges A(x) x0 ,x1 and
  calculate
• output interval ranges A(y) y0, y1 such that
  
• For all x in x0 ,x1 f(x) in y0, y1

20
Abstract interpretation example Interval
Analysis (continued)

• All operations x ? y on program variables x,y can
  be abstracted to corresponding interval analysis
  operations I ? J on intervals I, J by setting
• I ? J a0 , a1 with
• a0 Infimum( a ? b a in I and b in J )
• a1 Supremum( a ? b a in I and b in J )

21
Abstract interpretation example Interval
Analysis (continued)

• For many basic operations this leads to simple
  and easy-to-implement interval abstractions, e.g.
• x1 , x2 y1 , y2 x1 y1 , x2 y2
  
• Abstract interpretation by interval analysis
  gives rise to 3-valued logic For example,
• x1 , x2 lt y1 , y2 true for
  x2 lt y1 false for y2 x1
  undecided otherwise x1 lt y2 and y1 x2

22
C-Sample-Function f()

1. double globx, globy
2. double f(double x, double y, int i)
3. double z
4. int j, k, error0 0
5. if ( i lt 0 )
6. k 0
7. else
8. k i
9. if ( x lt 5 and y lt exp(x) )
10. x x - y - globy
11. else
12. x y - y - globx

23
C-Sample-Function (continued)

1. for ( k k 1
2. k lt x and error0 0
3. k k2)
4. if ( k lt 0 )
5. error0 1 // reachable ?
6. if ( error0 0 )
7. z log((double)k-x)
8. else
9. z 0
10. return z

24
Example 1 Unreachable code in line 17 ?

• Assume precondition x,y,globx,globy,i in
  -10,10
• Perform abstract interpretation by interval
  analysis with input value abstraction A(x) A(y)
  ... A(i) -10,10
• double f(double x, double y, int i)
• // A(x) A(y) ... A(i) -10,10
• double z
• // A(z) -8,8, since stack values are
  undefined
• int j, k, error0 0
• // A(j) A(k) -8,8, A(error0) 0,0
• if ( i lt 0 ) k 0 else k i
• // A(k) -10,10, because A(i)lt0
  undecided
• // Result can be improved to A(k) 0,10,
• // because k i is only assigned for 0 lt i

25
Example 1 Unreachable code in line 17 ?

• For loops, calculate interval interpretation
  fixpoint
• // A(k) 0,10, A(error0) 0,0, A(x)
  -30,30
• for ( k k 1
• k lt x and error0 0
• k k2)
• if ( k lt 0 )
• error0 1 // line 17
• // Loop interpretation fixpoint
• // A(k) 1,58, A(error0) 0,0, A(x)
  -30,30
• As a consequence, line 17 is unreachable ?
  Static analysis provides information for
  structural coverage test data generation process

26
Example 2 Structural coverage test data
generation

• Suppose we wish to cover if-branches line 6 and
  line 10, and skip the for-loop in lines 13 17
  in the C-sample function f() introduced above
  with precondition as given above. This results in
  the following constraints to be solved
• x,y,globx,globy,i in -10,10 // Precondition
• and (i lt 0) //
  if-condition line 5 true
• and (x lt 5.0) //
  if-condition line 9 true, 1st conjunct
• and (y lt exp(x)) //
  if-condition line 9 true, 2nd conjunct
• and (1 gt x y globy) // loop condition
  line 14, k lt x, false
• which evaluates to true if
• A(i) -10,-1 and (A(x) subset -10,5) and
• (A(y), A(globx), A(globy) subset -10,10) and
• (A(y) lt exp(A(x))) and (1,1 A(x) -
  A(y) - A(globy)) is true

27
Example 2 Structural coverage test data
generation

• Solution technique for interval constraints
  above
• Start with initial interval valuations A(i)
  -10,-1 and (A(x) -10,5 ) and
  (A(y), A(globx), A(globy) -10,10)
• If one conjunct evaluates to false, no solution
  exists, and the path is infeasible, i.e. cannot
  be covered
• If all conjuncts evaluate to true, interval
  solution has been found, select any x in A(x), y
  in A(y), ..., i in A(i) These inputs will lead
  to coverage of the desired path
• If one conjunct evaluates to undecided, perform
  bi-partitioning on interval with largest diameter

28
Example 2 Structural coverage test data
generation

• Bi-partitioning example Suppose (A(y) lt
  exp(A(x))) evaluates to undecided, and that
• diameter(A(y)) lt diameter(A(x))
• Define A1(x) Inf(A(x)),(Inf(A(x))Sup(A(x)))/2
  A2(x) (Inf(A(x))Sup(A(x)))/2,
  Sup(A(x))
• Re-perform steps 2 4 above with two possible
  solution candidates
• A1(x),A(y),A(globx),A(globy),A(i)
• A2(x),A(y),A(globx),A(globy),A(i)
• Use forward-backward constraint propagation to
  avoid too many bi-partitioning steps

29
Example 2 Structural coverage test data
generation

• Test data generator uses the same abstract
  interpretation by interval analysis technique as
  static analysis
• Static analysis applies over approximation
  Possible concrete executions are a subset of
  possible executions identified by static analysis
  ? fast, but may lead to false alarms
• Test data generation applies under approximation
  Every tuple (x,y,...) selected from solution
  intervals A(x), A(y),... covers the desired path
  ? slower, but guaranteed reachability
• As a consequence, the test data generator can be
  used to verify potential errors indentified by
  static analysis, by means of constructing
  concrete inputs (x,y,...) leading to the error
  situation

30
Example 3 Test Case Generator Supports Static
Analysis

• Consider modified for-loop in example above
• // A(k) 0,10, A(error0) 0,0, A(x)
  -30,30
• int kOld k // A(kOld) 0,10
• for ( k k 1
• k lt x and error0 0
• k k2)
• if ( kOld gt k ) // (A(kOld) gt A(k))
  0,1
• error0 1 //
  undecided
• else
• kOld k
• // A(k) 1,58, A(kOld) 1,29
• // A(error0) 0,1, A(x) -30,30

31
Example 3 Test Case Generator Supports Static
Analysis

• Conventional static analysis tool might stop with
  false alarm error0 1 (line 7) potentially
  reachable
• Path selector, constraint generator and solver
  can falsify this result as follows
• Path selector unwinds loop for n 0,1,2, ...
  cycles
• Constraint generator collects conditions to be
  fulfilled in order to stimulate n loop cycles
• Solver establishes that at most n 5 cycles are
  possible for initial valuations A(k)
  0,10, A(x) -30,30 at loop entry
• Solver establishes that if-condition (kOld gt k)
  cannot be fulfilled for n 5.
• Test case generation can confirm that error0 1
  (line 7) is reachable if A(i) 230, 230,
  A(x) 231,8 at program start

32
Integration of Techniques Into Model-Based
Development Cycle (1)

• In model-driven development code is generated
  from structural and functional models, for
  example in UML
• Composite structure diagrams, class diagrams for
  SW architecture
• Statecharts for reactive behaviour
• Method specifications for transformational
  behaviour
• Transformational behaviour is typically coded
  directly in the target programming language

33
Model-Based Development ... (2)

• The techniques described in this talk are applied
  to the testanalysis of these hand-coded methods
• RT-Tester also performs automated test case
  generation from UML-Statecharts
• Test cases are paths through the Statechart
• RT-Tester calculates the necessary input data to
  be passed to the system under test whenever an
  input method is invoked
• Intermediate model representation allows to use
  the same techniques as sketched for module
  testing

34
Conclusion (1)

1. We have described the main objectives of modules
   testing and static analysis
2. The utilisation of combined testinganalysis has
   been illustrated, using the RT-Tester tool as an
   example
3. The fundamental techniques for implementing the
   capabilities described have been sketched

35
Conclusion (2)

• The integration of the methods and techniques
  described in this presentation into the
  model-driven development process have been
  sketched
• We expect that a variety of tools supporting
  testinganalysis will become available in the
  near future
• All features are currently applied and evaluated
  in verification and test projects for railway
  control and avionic systems

36
Conclusion (3)

• Road map for RT-Tester
• Available NOW
• Module testing support
• Test case generation from UML Statecharts only
  in combination with case tool Borland Together
• January 2008 Static analysis and light-weight
  functional verification available
• March 2008 Full model-based testing support
  available support for various case tools
• Acknowledgements This work has been supported by
  BIG Bremer Investitions-Gesellschaft under
  research grant 2INNO1015B