Testing and Verification

1
Testing and Verification

• Testing Computer Code, Debugging, Reporting
  Results, and Documenting Computational Experiments

2
Sources of Error

• Typically, a programmer works from a
  specification of the program
• Programmers may interpret the specs differently
  and may introduce bugs
• Level of detail vs. loose guidelines
• Several sources of error
• The specification itself
• Mis-implementation of the spec.
• Run-time errors in the computer systems
• Unexpected behavior from users of the program

3
Software Development Process

• Requirements
• What do the customers want?
• Architecture
• Writing a specification
• Deciding on representations
• Implementation
• Testing
• Documentation
• Deployment

4
The Software Lifecycle

• The 80-20 rule
• Software developers spend 80 of their time on
  debugging already written code and 20 on writing
  new code
• Correctness is our first principle
• "Program testing can be used to show the presence
  of bugs, but never to show their absence!" --
  Edsger Dijkstra

5
Programming by Contracts

• Each routine should have pre- and post-conditions
• The preconditions state the conditions that need
  to be satisfied before a piece of code or a
  routine
• The postconditions give the state of all
  conditions after the execution
• Defines a contract between the caller and the
  code

6
Example
void write_sqrt(double x) // Precondition x
0 // Postcondition print the square // root
of x to stdout
7
Babysitters

• Each programmer should be teamed up with a tester
• They work using the same specification
• The programmer implements the specifications
• The tester writes tests to find bugs
• Remember that bugs are due to human error
  computers hardly ever create bugs

8
Formal Verification

• Some programs can be transformed into a
  mathematical/logical system
• Model checking
• Pi-calculus
• We can use math to prove that a specification is
  correct
• We can also identify errors in implementation
• A discipline called Formal Methods in Computer
  Science
• Hard to apply to realistic and large programs
• Active field of research

9
Testing Code

• Incremental testing
• Make sure each set of lines, routine, set of
  routines, code fragment, etc. is working before
  the complete system is put together
• Regression testing
• Problem that triggers a bug is included in the
  test suite
• Re-do all tests when changes are made

10
Testing Code

• Extreme Programming
• Construct tools for testing and set up test data
  before you start writing a new code
• Automate the testing and add new tests as you
  refine and find bugs
• When you think everything is working, rerun the
  complete set of tests
• A version of this Nightly builds

11
Localizing Bugs

• Print statements (Not stone age!)
• printf(Entering routine xxx-yy\n)
• printf(Before main loop in xxx algorithm\n)
• Non-buffered printing
• fprintf(stderr,This output is not buffered)

12
Localizing Bugs, contd

• In C Define a macro
• ifdef DEBUG
• define checkpoint() \
• fprintf(stderr,At line d in file
  s\n,__LINE__,\
• __FILE__)
• else
• define checkpoint()
• endif
• If DEBUG is defined, the code
• checkpoint()
• will produce output like
• At line 16 in file smoother.c

13
Localizing Bugs, contd

• Print contents of variables and data structures
• Write print routines for displaying the contents
  of your data structures in readable format
• Print data in condensed form, e.g. the norm of a
  vector instead of all the vector entries
• Use the more or less paging commands for
  examining output, or use a text editor
• Use grep/sed/awk commands to filter data

14
Localizing Bugs, contd

• Compare and contrast
• A working program in another language, on another
  system, etc. is a very valuable resource
• BUT Can the working program be trusted?
• Often subproblems, or simplified problems, can be
  easily implemented in, e.g., Matlab
• Use the same test data for both programs!

15
Localizing Bugs, contd

• Use a debugger
• gdb, or a more advanced vendor-specific debugger
• Compile using -g flag
• Some debuggers can give at least some debug
  information also for optimized code (-O, -fast)
• Set breakpoints and view contents of variables
  and call trees

16
Memory Bugs

• Often cause problems far from where the bug
  really is
• Often only occurs for special data sets
• May occur in a seemingly non-deterministic way
• Influence from the environment (system load,
  etc.)
• May be found in programs that were supposed to be
  tested and verified, e.g. when porting to
  another system

17
Memory Bugs, contd

• Unallocated memory
• double A
• .
• for (i0 i
• Ai 0.0
• Can cause execution to stop when the variable is
  used, or an attempt to deallocate the memory is
  made
• Most compilers can check for unallocated memory

18
Memory Checkers

• In gcc, use mudflap,
• http//gcc.gnu.org/wiki/Mudflap_Pointer_Debugging
• http//gcc.fyxm.net/summit/2003/mudflap.pdf
• gcc -Wuninitialized -Wall -fmudflap
• Another tool is the Memcheck tool in Valgrind
• http//valgrind.org/info/tools.htmlmemcheck
• valgrind --leak-checkyes myprog

19
More Memory Bugs

• Overwriting or overreading memory
• Common error in C and Fortran Using arrays
  outside the array bounds
• allocate(A(N))
• .
• do i1 to N
• Diff(i)(A(i1)-A(i))/h
• end do
• May result in erroneous results or program crash
• Program crash may occur later, e.g. when calling
  memory allocation routine
• Many compilers can generate code that checks
  array bounds during execution (gcc mudflap) .
  NOTE This reduces performance!
• Handled in Java and, e.g., Pascal (Runtime error)

20
More Memory Bugs, contd

• Dangling pointers
• A pointer to a piece of memory that once was
  allocated to a variable, but then was deallocated
• Common error when copying and deallocating
  derived types with dynamically allocated fields
• Other common error Returning a pointer to a
  local variable
• Hard to debug! Some debugging tools may help you

21
More Memory Bugs, contd

• Memory leaks
• Allocated memory is lost without deallocation
• The program runs out of memory for large
  problems, many iterations
• Hard to detect. Use e.g. top to monitor memory
  usage
• Hard to debug. Instrumented system call libraries
  may give some information
• valgrind
• Handled in e.g. Java, and other languages with
  built-in garbage collection

22
Computational Bugs

• Floating-point problems
• Catastrophic Cancellation (subtraction)
• Precision (using number of very different size)
• Accuracy of intrinsic functions and constants
• Algorithms used near their limit of applicability
• Near-indefinite matrix
• Near-singular matrix
• Near-degenerate eigenvectors
• Near-degenerate triangles in computational grid
• Stress testing Include extreme inputs in test
  data set

23
Subtle Computational Bugs

• Order reduction caused by indexing error
• Example Using a one-off index in, e.g., a
  boundary condition, can reduce order of accuracy
  from two to one
• Verify the order of accuracy by grid refinement
  studies
• Or by comparing to a known solution to a simple
  problem
• Accuracy or order of accuracy may be reduced only
  for certain problem classes
• Example A minor bug may cause order reduction
  for stiff problems in an ODE solver

24
Performance Bugs

• Traps to software computations for NaN, Inf,
  etc.
• Change of algorithms in libraries. Ex FFT for
  prime number length vectors is performed as
  matrix-vector multiplication (8n2 operations
  instead of 5nlog(n))

25
Reporting Computational Experiments

• Indicators should if possible be independent of
  the experiment. Traditional indicators are e.g.
• CPU time
• Numerical accuracy
• Number of iterations
• Scope, applicability
• Portability
• Storage requirement
• Operation count (flops)

26
Reporting Computational Experiments

• Presentation of algorithms
• Complete description of the algorithm
• Specification of the domain of applicability of
  the algorithm
• If possible Computational complexity
• If possible Convergence results
• If possible Rate of convergence, order of
  accuracy

27
Reporting Computational Experiments

• Presentation of implementation
• Programming language used
• Compiler name and options used
• Computer environment, System and OS
• Input data
• Tolerance parameter settings

28
Reporting Computational Experiments

• Presentation of experiments
• Objective of the experiment
• Description of problem generator/input data set

29
Reporting Computational Experiments

• Presentation of results, e.g. CPU time
• Description of how the timings were produced
• What parts of the code are included in the
  measurement?
• What is the variability of the timings?
• How is the variability handled?