Structural Testing

1
Structural Testing

• Motivation
• The gcov Tool
• An example using gcov
• How does gcov do it
• gcov subtleties
• Further structural testing measures
• Flowcharts
• Structural testing and flowcharts
• Minimal test suites
• Path coverage
• Testing loops

Time to cover 25-35 minutes
2
Structural Testing Motivation

• Consider a typical programming project
• We have designed code based on requirements
• We have also built in some additional features
• Additional features interact with required
  features in various ways
• We need to test code of additional features
• We dont remember what all of them are
• Because new features are not mentioned in
  requirements, testing based on requirements will
  not help us
• Consider a function to sort an array of integers
• Requirements state only that resulting array must
  preserve original elements and be sorted
• Many different algorithms exist for doing this
• A test suite which tests one algorithm thoroughly
  may not test another

3
Structural Testing Motivation continued

• In both these cases, the essential problem is
• How do we know we have tested all of our code?
• For all we know, some code may never have been
  executed by any test that we have run
• User may do something that executes this untested
  code
• It will therefore be run for the first time by
  the user
• Code run for the first time often has bugs!
• Have we tested all of our code?
• Tools exist to help us answer this question
• These tools generally called code coverage
  tools
• Help us to see what percentage of code has been
  covered by some test case
• These tools usually targeted to individual
  programming languages
• Good free code coverage tools exist for C
• Seems only commercial tools exist for Java
• We will look at one for C

4
gcov A Structural Testing Tool

• gcov GNU Project coverage utility
• Based on earlier utility tcov
• GNU Project
• Supplies free software
• Created gcc compiler for C
• Most Linux utility programs come from GNU Project
  
• To use gcov (overview)
• Compile your C program using gcc with special
  switches
• Run your program normally
• While running, your program will write
  information to special log files
• After running test cases, run gcov
• gcov will generate a coverage report listing
  which lines have been executed
• We will look at each of these steps in more
  detail

5
Normal Compilation of a Program

• In the above diagram,
• Solid lines inputs
• Dashed lines outputs
• Normally, we just
• Compile a program from source code
• Execute it on some selected test cases
• Look at the output for correctness

6
Compilation Using gcov

• Compilation / execution with gcov is basically
  the same
• However, extra switches to gcc cause extra things
  to happen
• Switches fprofilearcs ftestcoverage

7
Compilation Using gcov continued

• When we compile with the extra switches, gcc does
  the following
• extra things
• Generates map file of the blocks of code in our
  source files
• Generates object code which counts the number of
  times each block of code has been executed
• Block of code
• Any sequence of statements such that executing
  one of them guarantees we must execute the next
  one
• Example (program wordcount.c)
• Instead of just gcc wordcount.c, we say gcc
  fprofilearcs ftestcoverage wordcount.c
• gcc generates map file wordcount.gcno
• Exact format of the map file is not important
  to us

8
Running a gcov-Compiled Program

• Run program as normal on test cases
• First time program is run, a coverage data file
  appears (with extension .gcda)
• Coverage data file contains information about how
  many times each line of code has been executed
• Example
• After running wordcount, file wordcount.gcda is
  created
• Coverage data is updated every subsequent time
  program is run
• Coverage data is cumulative e.g.
• Line 42 executed 5 times on first run
• Line 42 executed 10 times on second run
• Therefore, coverage data stored in .gcda file
  shows 15 executions of line 42

9
Getting Information from gcov

• Run gcov with source file name as argument (e.g.
  gcov wordcount.c)
• gcov writes some statistics to screen, creates a
  file with extension .gcov (e.g.
  wordcount.c.gcov)
• .gcov file contains
• On right source code
• On left number of times each line has been
  executed
• If lines have never been executed, .gcov file
  shows to highlight it

10
Example wordcount

• Program count the number of lines, words, chars
  in input file
• Original Source file

include ltstdio.hgt define TRUE 1 define FALSE
0 main() int nl0, nw0, nc0 int inword
FALSE char c c getchar() while (c
! EOF) nc if (c \n)
nl if (c c\n c \t)
inword FALSE else if (!inword)
inword TRUE nw c getchar()
printf(d lines, d words, d chars\n, nl, nw,
nc)
11
wordcount Example Compiling and Running

• We compile with
• gcc fprofilearcs ftestcoverage o wordcount
  wordcount.c
• We get
• Executable in wordcount
• System files wordcount.gcno
• We run wordcount
• We give it as input two empty lines (two carriage
  returns)
• File wordcount.da is created
• We now run gcov wordcount.c
• gcov tells us
• 86.67 of 15 source lines executed in file
  wordcount.c Creating wordcount.c.gcov.

12
- 0Sourcewordcount.c -
0Graphwordcount.gcno - 0Datawordcount.gcd
a - 0Runs1 - 0Programs1 -
1include ltstdio.hgt - 2define TRUE 1 -
3define FALSE 0 1 4main() 1 5
int n10, nw0, nc0 1 6 int
inwordFALSE - 7 char c - 8 1
9 cgetchar() 4 10 while (c ! EOF)
2 11 nc 2 12 if (c'\n') 2
13 n1 4 14 if (c' '
c'\n' c'\t') 2 15 inwordFALSE
16 else if (!inword)
17 inwordTRUE nw 2 18
cgetchar() - 19 1 20 printf("d
lines, d words, d chars\n", - 21 n1,
nw, nc) - 22
13
wordcount Example Explanation of the Output

• The two lines marked with have never been
  executed
• The other lines have been executed
• For instance
• The initial c getchar() has been executed once
  (1 in left margin)
• The nc has been executed twice (2 in left
  margin)
• This is because each carriage return counts as 1
  character
• The declaration lines
• int nl0, nw0, nc0
• int inword FALSE
• are marked as executed because they contain
  initialization code
• The declaration line
• char c
• is not marked at all because it is just a
  declaration

14
wordcount Example Carrying On

• We now run wordcount again, this time giving just
  the line Zippy as input
• We now run gcov wordcount.c
• gcov tells us 100.00 of 15 source lines
  executed in file wordcount.c Creating
  wordcount.c.gcov.

15
Wordcount Example New .gcov File

• - 0Sourcewordcount.c
• - 0Graphwordcount.gcno
• - 0Datawordcount.gcda
• - 0Runs2
• - 0Programs1
• - 1include ltstdio.hgt
• - 2define TRUE 1
• - 3define FALSE 0
• 2 4main()
• 2 5 int n10, nw0, nc0
• 2 6 int inwordFALSE
• - 7 char c
• - 8
• 2 9 cgetchar()
• 12 10 while (c ! EOF)
• 8 11 nc
• 8 12 if (c'\n')
• 3 13 n1
• 11 14 if (c' ' c'\n'
  c'\t')

16
wordcount Example New Output

• We have given the system an input file of 6
  characters on this run (5 letters carriage
  return)
• Along with the 2 characters we gave on the
  previous run, the runs of wordcount have read 8
  characters
• Hence, nc line has been executed 8 times in
  total
• Counts on other lines have similarly been updated
  
• We have now executed the case where we have a
  nonwhitespace
• character
• Therefore, the lines else if (!inword)
  inword TRUE nw
• have been executed
• Therefore, 100 of the lines of the program have
  now been executed

17
Measuring the Usefulness of a Test Suite

• A good structural test suite will execute most
  lines of code
• It may not be possible to execute all lines
• Executing a large percentage may be acceptable
• Hence, measure of code covered ( of lines
  executed) is a good measure of the usefulness of
  such a test suite
• However, with gcov
• As we keep executing test cases, the .gcda file
  keeps getting updated
• Therefore, to measure goodness of a test suite,
  what we would like to do is
• Reset all counts in .gcda file to 0
• Run test suite
• Run gcov and look at of lines covered
• To reset all counts in the .gcda file
• Just remove the file!
• Your compiled code will
• Assume this is the first time it has been run
  again
• Create the .gcda file anew

18
How Does gcov Do It?

• Each block in code numbered
• gcc inserts some code into your program at
  beginning of main, beginning of each block, end
  of main
• At beginning of main
• Program zeros out a big array
• During execution
• Every time block n executed, entry n of array
  incremented
• At end of main
• If .gcda file does not exist, program creates it
• Otherwise, program reads .gcda file, increments
  counts in array, writes out .gcda file again
• Does similar processing for calls to exit() as at
  end of main
• Map files map source line number to block
• gcov program matches block execution counts in
  .da file with source lines

19
gcov Subtleties

• If we compile and link many source files with the
  gcov options
• Each source file gets a .gcno file (the map
  file)
• When we run the program, each source file gets a
  .gcda file (the coverage data file)
• Thus, we can/must get a separate report from gcov
  on each separate source file
• Useful if we are interested in coverage of each
  module of our program
• Example
• We have a main program main.c and two modules,
  StackImplementation.c and IO.c
• On compilation, we get main.gcno,
  StackImplementation.gcno and IO.gcno
• Every time we run the program, files main.gcda,
  StackImplementation.gcda, and IO.gcda are updated
  
• We can say gcov StackImplementation.c
• Get a report on StackImplementation.c separately
• Useful if that is the module we are interested in

20

• include ltstdio.hgt
• int main()
• int x
• int y
• int z0
• printf("\nEnter x")
• scanf ("d", x)
• printf("\nEnter y")
• scanf("d", y)
• z xy
• if (zgtx)
• printf("\nHi")
• else
• printf("\nBye")
• if (zgty) printf ("\nYes")
• else
• printf("\nNo")

Lets try it out on the gaul network lsgcc
-fprofile-arcs -ftest-coverage junk2.c
21
gcov, Lines and Statements

• Consider the following code
• scanf(d, x) / Read x from terminal /
• if (x gt 0)
• pos 1
• printf(d d \n, x, pos)
• We can execute 100 of lines in this code only if
  we input a positive number for x sometime
• Now consider the following code
• scanf(d, x) / Read x from terminal /
• if (x gt 0) pos 1
• printf(d d\n, x, pos)
• We can execute 100 of lines in this code even if
  we input only negative numbers for x

22
gcov, Lines and Statements continued

• Reason
• In second code fragment, the substatement pos
  1 of the if is on the same line as the if
• We have executed the if in the sense that we have
  evaluated its condition (xgt0)
• Line is considered executed even when only some
  of the code on it is executed
• Moral to ensure all statements executed
• Must put each statement on a separate line

23
Is gcov Enough?

• gcov is OK for a simple analysis of the
  thoroughness of a test suite
• It tells us what percentage of the lines of code
  we have executed
• However, a test suite executing 100 of lines may
  still not catch some problems
• Example consider again the code
• scanf(d, x) / Read x from terminal /
• if (xgt0)
• pos 1
• printf(d d\n, x, pos)
• If all tests in our test suite set x to a
  positive number, we can achieve 100 line
  coverage
• However, we have still not tested the case in
  which x is negative or zero
• That case might be handled incorrectly
• For instance, maybe pos is supposed to be 1 if x
  is positive, and 0 otherwise
• To talk about stronger measures of thoroughness,
  it is useful to have a graph of the program
  control flow

24
int main() int n1, n2, n3, ans
printf("\n\nEnter a value for n1")
scanf("d",n1) printf("\n\nEnter a value for
n2") scanf("d",n2) printf("\n\nEnter a
value for n3") scanf("d",n3) ans
getnum(n1, n2, n3) printf("\nThe number was
d\n,ans) return 0 int getnum(int
num1,int num2,int num3) if (num1gtnum2) return
num1 else if (num2gtnum3) return
num2 else if (num1num2)
return num3 else
return num1num2
Review Questions

• Consider the code to the right and answer the
  following questions
• gcov would count 16 lines, which lines do you
  think get counted?
• What coverage would the following test case
  given1?5, n2?4, n3?3
• Can we get 100 coverage? If so, list each of the
  test cases you would use to get that coverage,
  give the minimal number of test cases in your
  answer needed to get 100 coverage.
• Junit info