The Testing Process CS320 Fundamentals of Software Engineering

1
The Testing Process CS320 Fundamentals
of Software Engineering
2
What is Testing?

• IEEE
• The process of operating a system or component
  under specified condition, observing or recording
  the results, and making an evaluation of some
  aspect of the system or component.
• Specified Condition Test Cases

3
What is testing?

• Rick Craig and Stefan Jaskiel
• Testing is a concurrent lifecycle process of
  engineering, using and maintaining testware in
  order to measure and improve the quality of the
  software being tested
• Includes planning, analysis, and design that lead
  to test cases

4
What is testing?

• Testing is the process of comparing what is
  with what ought to be

5
Boris Beizers Five Levels

• Level 0 No difference between testing and
  debugging
• Level 1 Purpose is to show software works
• Level 2 Purpose is to show that software
  doesnt work (tough cases, boundary values)
• Level 3 Purpose is not to prove, but to reduce
  the perceived risk of not working at an
  acceptable value
• Level 4 Mental discipline that result in low
  risk software (Make software more testable from
  inception)

6
Challenges

• Not enough time
• Too many combos
• Difficult to determine the expected result of
  each test
• Non-existent or rapidly changing reqs
• No training
• No tool support
• Management doesn't understand testing

7
Test Cases

• Need to be designed
• Systematic approach
• Documentation Planning
• Goal and purpose

8
Test Case

• Inputs
• Expected Outputs
• Order of Execution
• (and Testing Criteria) how do I know when Im
  done testing???

9
Inputs

• Data
• Files
• Interfacing Systems (keyboards, mice )
• databases

10
Outputs

• Excepted Results (oracle)
• Program, process or data
• Regression test suites
• Validation data
• Purchase test suites
• Existing program

11
Order of Execution

• Cascading test cases (build on each other)
• Ex. Create record in database
• Use record
• Delete record etc.
• Indecent test case (order does not matter)

12
Testing Criteria

• When do I stop testing
• Ex1 When I cover all def use pairs
• Ex2 All paths

13
Types of Testing

• Black Box
• Reqs and spec, no knowledge of internal structure
• White Box
• Based on internal paths, structure
• Ex1 - definition use pairs

14
Testing Levels

• Unit Testing smallest piece
• Integration Testing assemble pieces into larger
  units and test
• System Testing all of the software hardware,
  user manuals etc.
• Acceptance Testing testing, that when completed
  will result in delivering the software to the
  customer

15
Impossible to Test Everything

• int scale(int j)
• j j-1 // should be j j1
• j j/30000
• return j
• Input range -32768 to 32767 65,536 test cases
  (6 test cases reveal problem)
• -30,001, -30000, -1, 0, 29999, 30000

16
Unit testing with JUnit
17
One minute summary

• Code that isnt tested doesnt work
• Why test?
• Code that isnt regression tested suffers from
  code rot (breaks eventually)
• Why regression test?
• A unit testing framework help make unit and
  regression testing more efficient

18
What is unit testing?

• Testing is often divided into categories such as
• Unit testing
• Testing a unit of code, usually a class
• Integration testing
• Testing a module of code (e.g. a package)
• Application testing
• Testing the code as the user would see it (black
  box)

19
What is a testing framework?

• A test framework provides reusable test
  functionality which
• Is easier to use (e.g. dont have to write the
  same code for each class)
• Is standardized and reusable
• Provides a base for regression tests

20
Why formalize unit testing?

• Unit testing has often been done, but in an ad
  hoc manner
• E.g. adding a main method to a class, which runs
  tests on the class
• Axiom
• Code that isnt tested doesnt work
• If code has no automated test case written for
  it to prove that it works, it must be assumed not
  to work. (Hightower and Lesiecki)

21
Why use a testing framework?

• Each class must be tested when it is developed
• Each class needs a regression test
• Regression tests need to have standard interfaces
• Thus, we can build the regression test when
  building the class and have a better, more stable
  product for less work

22
Regression testing

• New code and changes to old code can affect the
  rest of the code base
• Affect sometimes means break
• I need to run tests on the old code, to verify it
  works these are regression tests
• Regression testing is required for a stable,
  maintainable code base

23
Regression Testing

• Regression testing - retesting code to check for
  errors due to changes
• Rothermel and Harrold (1994) provide a simple
  definition for regression testing (T0 original
  tests)
• Identify changes to the software
• Evaluate which tests from set T0 are no longer
  valid and create a subset T1
• Use T1 to test software
• Identify if any parts of the system have not been
  tested adequately (based upon test adequacy
  criteria) and generate a new set of tests T2
• Use T2 to test software

24
Regression Testing

• Rothermel and Harrold built framework of several
  retest selection techniques (1996)
• Minimization
• Linear equations
• Coverage
• ID changed definition-use pairs in control flow
  graphs
• Safe techniques
• Control dependency graphs.

25
Background Regression Testing

• Leung and White classify test cases according to
  reuse (1989).
• Reusable test cases cover unmodified code (kept
  but not used)
• Retestable test cases cover modified code (used)
• Obsolete test-cases cover removed code (deleted)

26
Regression Testing

• Briand et al. (TR 2004) uses Use Cases,
  Sequence Diagrams, and Class Diagrams to
  regression test code.
• Creates mapping between design changes and code
  changes to classify code test cases.
• Uses Leung and White approach to classify test
  cases
• Makes assumptions regarding how the UML is used
• (1) assumes all possible use-case scenarios can
  be realized by Sequence Diagrams
• (2) assumes that each test case maps to a
  Sequence Diagram Scenario

27
Regression testing and refactoring

• Refactoring is a technique to restructure code
  in a disciplined way. (Martin Fowler)
• Refactoring is an excellent way to break code.
• Regression testing allows developers to refactor
  safely if the refactored code passes the test
  suite, it works

28
Running automated tests

• The real power of regression tests happens when
  they are automated
• This requires they report pass/fail results in a
  standardized way
• Can set up jobs to
• Clean check out latest build tree
• Run tests
• Put results on a web page send mail (if tests
  fail)
• JUnit ant have code to do all of this

29
Some background

• eXtreme programming (XP) is a programming
  methodology that stresses (among other things)
  testing and refactoring
• The Apache community provides open-source
  software, including the Jakarta project (server
  side java tools, e.g. tomcat (servlet engine))
• Ant is a build tool (like make)
• Part of the Jakarta project
• Is becoming the de facto standard for java
  projects
• Is written in java, uses xml to specify build
  targets

30
What is Junit?

• JUnit is a regression testing framework written
  by Erich Gamma and Kent Beck
• It is found at www.junit.org
• It consists of classes that the developer can
  extend to write a test
• Notably junit.framework.TestCase and related
  structure to run and report the tests

31
JUnit..

• JUnit helps the programmer
• Define and execute tests and test suites
• Formalize requirements and clarify architecture
• Write and debug code
• Integrate code and always be ready to release a
  working version

32
Installing Junit and CppUnit

• For JUnit
• Download (and unpack) the distribution
• Put the jar file on your classpath
• For CppUnit
• Download (and unpack) the distribution
• On linux, run make (configure make make
  install)
• Bob Goddard figured out the required switches on
  Solaris, contact Bob, Pete Brodsky or I for
  details

33
Tool integration

• Java IDEs
• Netbeans - http//www.netbeans.org/index.html
• Claims to include Junit integrationIts free
• Idea - http//www.intellij.com/idea/
• Includes the Junit jar is Junit savvy
• Also supports ant
• Not free, but not expensive (400-600)

34
What JUnit does

• JUnit runs a suite of tests and reports results
• For each test in the test suite
• JUnit calls setUp()
• This method should create any objects you may
  need for testing

35
What JUnit does

• JUnit calls one test method
• The test method may comprise multiple test cases
  that is, it may make multiple calls to the method
  you are testing
• In fact, since its your code, the test method
  can do anything you want
• The setUp() method ensures you entered the test
  method with a virgin set of objects what you do
  with them is up to you
• JUnit calls tearDown()
• This method should remove any objects you created

36
Creating a test class in JUnit

• Define a subclass of TestCase
• Override the setUp() method to initialize
  object(s) under test.
• Override the tearDown() method to release
  object(s) under test.
• Define one or more public testXXX() methods that
  exercise the object(s) under test and assert
  expected results.
• Define a static suite() factory method that
  creates a TestSuite containing all the testXXX()
  methods of the TestCase.
• Optionally define a main() method that runs the
  TestCase in batch mode.

37
Fixtures

• A fixture is just a some code you want run before
  every test
• You get a fixture by overriding the method
• protected void setUp()
• The general rule for running a test is
• protected void runTest() setUp() ltrun the
  testgt tearDown()
• so we can override setUp and/or tearDown, and
  that code will be run prior to or after every
  test case

38
Implementing setUp() method

• Override setUp() to initialize the variables, and
  objects
• Since setUp() is your code, you can modify it any
  way you like (such as creating new objects in it)
• Reduces the duplication of code

39
Implementing the tearDown() method

• In most cases, the tearDown() method doesnt need
  to do anything
• The next time you run setUp(), your objects will
  be replaced, and the old objects will be
  available for garbage collection
• Like the finally clause in a try-catch-finally
  statement, tearDown() is where you would release
  system resources (such as streams)

40
The structure of a test method

• A test method doesnt return a result
• If the tests run correctly, a test method does
  nothing
• If a test fails, it throws an AssertionFailedError
  
• The JUnit framework catches the error and deals
  with it you dont have to do anything

41
Test suites

• In practice, you want to run a group of related
  tests (e.g. all the tests for a class)
• To do so, group your test methods in a class
  which extends TestCase
• Running suites we will see in examples

42
assertX methods

• static void assertTrue(boolean test)
• static void assertFalse(boolean test)
• assertEquals(expected, actual)
• assertSame(Object expected, Object actual)
• assertNotSame(Object expected, Object actual)
• assertNull(Object object)

43
assertX methods

• assertNotNull(Object object)
• fail()
• All the above may take an optional String message
  as the first argument, for example,static void
  assertTrue(String message, boolean test)

44
Organize The Tests

• Create test cases in the same package as the code
  under test
• For each Java package in your application, define
  a TestSuite class that contains all the tests for
  validating the code in the package
• Define similar TestSuite classes that create
  higher-level and lower-level test suites in the
  other packages (and sub-packages) of the
  application
• Make sure your build process includes the
  compilation of all tests

45
JUnit framework
46
Writing a test

• Directions can be found in the Junit cookbook -
  http//junit.sourceforge.net/doc/cookbook/cookbook
  .htm
• Summary
• Create an instance of TestCase
• Write methods which run your tests, calling them
  testltFoogt
• Call a test runner with your test
• When you want to check a value, call an assert
  method (e.g. assertTrue()) and pass a condition
  that is true if the test succeeds

47
Example Counter class

• For the sake of example, we will create and test
  a trivial counter class
• The constructor will create a counter and set it
  to zero
• The increment method will add one to the counter
  and return the new value
• The decrement method will subtract one from the
  counter and return the new value

48
Example Counter class

• We write the test methods before we write the
  code
• This has the advantages described earlier
• Depending on the JUnit tool we use, we may have
  to create the class first, and we may have to
  populate it with stubs (methods with empty
  bodies)
• Dont be alarmed if, in this simple example, the
  JUnit tests are more code than the class itself

49
JUnit tests for Counter

• public class CounterTest extends
  junit.framework.TestCase Counter counter1
• public CounterTest() // default
  constructor
• protected void setUp() // creates a
  (simple) test fixture counter1 new
  Counter()
• protected void tearDown() // no
  resources to release

50
JUnit tests for Counter

• public void testIncrement()
  assertTrue(counter1.increment() 1)
  assertTrue(counter1.increment() 2)
• public void testDecrement()
  assertTrue(counter1.decrement() -1)
  // End from last slide

51
The Counter class itself

• public class Counter int count 0 public
  int increment() return count public
  int decrement() return --count
• public int getCount() return
  count

52
Comparing JUnit 3 to JUnit 4

• All the old assertXXX methods are the same
• Most things are about equally easy
• JUnit 4 makes it easier to test that exceptions
  are thrown when they should be
• JUnit 4 can still run JUnit 3 tests
• JUnit 4 provides protection against infinite
  loops
• JUnit 4 has some additional features

53
Migrating from JUnit 3

• JUnit 4 requires Java 5 or newer
• Dont extend junit.framework.TestCase just use
  an ordinary class
• Import org.junit. and org.junit.Assert.
• Use a static import for org.junit.Assert.
• Static imports replace inheritance from
  junit.framework.TestCase
• Use annotations instead of special method names
• Instead of a setUp method, put _at_Before before
  some method
• Instead of a tearDown method, put _at_After before
  some method
• Instead of beginning test method names with
  test, put _at_Test before each test method

54
Writing a JUnit test class, I

• Start by importing the JUnit 4 classes you need
• import org.junit.import static
  org.junit.Assert.
• Declare your class in the usual way
• public class MyProgramTest
• Declare any variables you are going to use
  frequently, typically including an instance of
  the class being tested
• MyProgram programint arrayint solution

55
Writing a JUnit test class, II

• If you wish, you can declare one method to be
  executed just once, when the class is first
  loaded
• This is for expensive setup, such as connecting
  to a database
• _at_BeforeClasspublic static void setUpClass()
  throws Exception // one-time
  initialization code
• If you wish, you can declare one method to be
  executed just once, to do cleanup after all the
  tests have been completed
• _at_AfterClasspublic static void tearDownClass()
  throws Exception // one-time cleanup
  code

56
Writing a JUnit test class, III

• You can define one or more methods to be executed
  before each test typically such methods
  initialize values, so that each test starts with
  a fresh set
• _at_Beforepublic void setUp() program new
  MyProgram() array new int 1, 2, 3, 4,
  5
• You can define one or more methods to be executed
  after each test typically such methods release
  resources, such as files
• _at_Afterpublic void tearDown()

57
_at_Before and _at_After methods

• You can have as many _at_Before and _at_After methods
  as you want
• Be warned You dont know in what order they will
  execute
• You can inherit _at_Before and _at_After methods from a
  superclass execution is as follows
• Execute the _at_Before methods in the superclass
• Execute the _at_Before methods in this class
• Execute a _at_Test method in this class
• Execute the _at_After methods in this class
• Execute the _at_After methods in the superclass

58
Writing a JUnit test class, IV

• A test method is annotated with _at_Test, takes no
  parameters, and returns no result
• All the usual assertXXX methods can be used
• _at_Testpublic void sum() assertEquals(15,
  program.sum(array)) assertTrue(program.min(ar
  ray) gt 0)

59
Special features of _at_Test

• You can limit how long a method is allowed to
  take
• This is good protection against infinite loops
• The time limit is specified in milliseconds
• The test fails if the method takes too long
• _at_Test (timeout10) public void greatBig()
  assertTrue(program.ackerman(5, 5) gt 10e12)
• Some method calls should throw an exception
• You can specify that a particular exception is
  expected
• The test will pass if the expected exception is
  thrown, and fail otherwise
• _at_Test (expectedIllegalArgumentException.class)pu
  blic void factorial() program.factorial(-5)
  

60
Parameterized tests

• Using _at_RunWith(valueParameterized.class) and a
  _at_Parameters method, you can run the same tests
  with multiple datasets
• _at_RunWith(valueParameterized.class)public class
  FactorialTest private long expected
  private int value _at_Parameters public
  static Collection data() return
  Arrays.asList( new Object 1, 0 , 1,
  1 , 2, 2 , 120, 5 ) public
  FactorialTest(long expected, int value) //
  constructor this.expected expected
  this.value value _at_Test
  public void factorial()
  assertEquals(expected, new Calculator().factorial(
  value))
• Source http//today.java.net/pub/a/today/2006/12/
  07/junit-reloaded.html

61
Ignoring a test

• The _at_Ignore annotation says to not run a test
• _at_Ignore("I dont want Dave to know this doesnt
  work")_at_Testpublic void add()
  assertEquals(4, program.sum(2, 2))
• You shouldnt use _at_Ignore without a very good
  reason!

62
Test suites

• As before, you can define a suite of tests
• _at_RunWith(valueSuite.class)_at_SuiteClasses(valueM
  yProgramTest.class,
  AnotherTest.class)public class AllTests

63
Other stuff

• Failed tests now throw an AssertionError, rather
  than JUnit 3s AssertionFailedError
• There is now a version of assertEquals for arrays
  of objectsassertEquals(Object expected,
  Object actual)
• Unfortunately, there is still no assertEquals for
  arrays of primitives
• JUnit 3 had an assertEquals(p, p) method for each
  kind of primitive p, but JUnit 4 only has an
  assertEquals(object, object) and depends on
  autoboxing

64
A gotcha

• The following method
• long sum(long x, long y) return x y
• with the following test
• _at_Testpublic void sum() assertEquals(4,
  s.sum(2, 2))
• gives
• expected lt4gt but was lt4gt
• This is due to your friend, autoboxing
• assertEquals no longer exists for primitives,
  only for objects
• Hence, the 4 is autoboxed to an Integer, while
  sum returns a long
• The error message means expected int 4, but got
  long 4
• To make this work, change the 4 to a 4L

65
JUnit 4 in Eclipse and NetBeans

• As usual, the easiest way to create a test class
  is just to let your IDE do it for you
• Here is the recommended test-driven approach
• Create a class containing all the stub methods
  you initially think you will need
• Have the IDE create the test class, with all the
  test methods
• Repeat
• Write a test
• Make sure the test fails
• Write the method being tested
• Make sure the test now succeeds
• Note When you create the test class, NetBeans in
  particular puts a lot of garbage lines into each
  test method you can just delete these and put in
  your own code

66
Result

• We will see with the help of tool
• When ??

!! Now !!
67
Why JUnit

• Allow you to write code faster while increasing
  quality
• Elegantly simple
• Check their own results and provide immediate
  feedback
• Tests is inexpensive
• Increase the stability of software
• Developer tests
• Written in Java
• Free
• Gives proper uniderstanding of unit testing

68
Problems with unit testing

• JUnit is designed to call methods and compare the
  results they return against expected results
• This ignores
• Programs that do work in response to GUI commands
• Methods that are used primary to produce output

69
Problems with unit testing

• I think heavy use of JUnit encourages a
  functional style, where most methods are called
  to compute a value, rather than to have side
  effects
• This can actually be a good thing
• Methods that just return results, without side
  effects (such as printing), are simpler, more
  general, and easier to reuse

70
Junit addons

• Junit has a bunch of add on stuff that has been
  written for it (your mileage may vary)
• Examples
• Code generators make guesses at what tests are
  needed
• Load testers
• Code coverage
• Testers for thread safety
• Helpful classes (e.g. RecursiveTestSuite is a
  TestSuite that will recursively walk through the
  directory structure looking for subclasses of
  TestCase and will add them.)

71
Cactus, HttpUnit, Mock Objects

• Cactus (from jakarta) is a simple test framework
  for unit testing server-side java code (Servlets,
  EJBs, Tag Libs, Filters, ...).
• HttpUnit emulates the relevant portions of
  browser behavior to allow automated testing (on
  sourceforge)
• Mock objects emulate objects from other parts of
  the application, but have controlled behaviour

72
Resources

• Web sites
• Junit - http//www.junit.org
• CppUnit - http//cppunit.sourceforge.net/
• Ant - http//jakarta.apache.org/ant/index.html