CS 141a Distributed Computation LaboratorySoftware Testing

1
Recap

• Design by Contract

2
Today

• Software Testing

3
Software Testing

• Testing is an important part of the software
  engineering process, because there are almost
  always differences between what is specified and
  what is initially implemented.
• Misunderstanding of specifications.
• Incorrect choices of algorithms or data
  structures.
• Pathological cases that arent considered.
• Typographical errors that happen to compile.

4
Types of Software Testing

• There are various types of software testing
• Unit testing attempts to locate errors in
  isolated pieces of a software system.
• Integration testing attempts to locate errors
  that result from interactions between pieces of a
  software system.
• Interactive testing attempts to locate problems
  that arise when humans must interact with a
  software system - such as flaws in its human
  interface, or in the assumptions made about
  users knowledge or experience.
• Well mainly concern ourselves with the first
  two, because our user interfaces consist
  primarily of command line parameters.

5
Unit Testing

• A unit test is a test that attempts to validate
  one particular piece of a software system.
• The size of the piece being tested can vary
• Method
• Class
• Package
• A set of unit tests is called a test suite.
• There are two main forms of unit test, whitebox
  and blackbox.

6
Unit TestingWhitebox

• Whitebox tests access the internal structure of
  the component being tested
• Assertion checking at runtime is a common form of
  whitebox testing.
• A test method on a class that accesses the
  classs private fields or methods is also a
  whitebox test.
• Whitebox tests are almost always incorporated
  into the component being tested.
• This is necessary in most object-oriented systems
  because of information hiding.

7
Unit TestingBlackbox

• Blackbox tests use only the public interfaces of
  the components being tested.
• A separate class that exercises one or more other
  classes would be an example.
• Blackbox tests are typically outside the
  component (class, package) being tested.
• Ensures that they dont accidentally use inside
  information that they shouldnt use.
• Allows the component to be built without the test
  being included, if necessary.

8
Unit Testing

• The world of blackbox and whitebox testing isnt
  entirely black and white - some situations
  require hybrid, or greybox, tests that observe
  the internal workings of a component.
• Most systems require both blackbox and whitebox
  testing - neither one alone is enough.
• Testing isnt a substitute for good design and
  programming methods.

9
Unit TestingResponsibility

• The people who design and build software systems
  are not necessarily the people who write the test
  suites for those systems.
• Whitebox tests are almost always written by the
  same people as the component being tested,
  because theyre internal to the component or rely
  on internal information.
• Blackbox tests are sometimes written by the same
  people, but are often written by special test
  designers whose job it is to try to find things
  that the software designers and builders didnt
  account for.

10
Unit Test Coverage

• What should be covered in unit tests?
• Whitebox tests should primarily cover consistency
  conditions (preconditions, postconditions,
  invariants).
• Blackbox tests should exercise every feature of
  every class in a system.
• This may involve invoking multiple features as
  part of a single test, to set up an environment
  or establish some particular condition.

11
Unit TestingExample

• Assume we have a Point class, with the following
  methods
• x, y, rho, theta - accessors for the coordinates
  of the point.
• distance - takes a point as a parameter and
  calculates the distance between the two points.
• translate, scale, rotate - change the position of
  the point.
• Constructors for polar and Cartesian coordinates.
• Other standard methods (equals, hashCode, clone).
• Assume further that a reasonable set of whitebox
  tests are included in the class as assertions.
• What blackbox tests can we do to satisfy
  ourselves that all these methods work properly?

12
Unit Testing Example

• Constructors, equals, hashCode, x, y, rho, theta
• Create a few points and verify that they get
  created correctly, by examining x, y, rho, theta.
• Ensure that points created with equivalent polar
  and Cartesian coordinates end up equal to each
  other, and have the same hashCode.
• Degenerate conditions create points with theta
  gt 2p, or theta lt 0, and ensure that theyre
  created correctly (this only applies if we dont
  have a precondition restricting the values of
  theta).

13
Unit TestingExample

• distance
• Check that the distance between a point and
  itself is 0.
• Check that the distance between a point and a
  clone of that point is 0.
• Create two points a known distance away from each
  other and check to see that distance gives the
  known distance.
• Call distance on both objects and ensure the
  result is identical.
• Do this with enough pairs of points to convince
  yourself that distances work between points in
  different quadrants, and when the origin is one
  of the points.

14
Unit TestingExample

• translate
• Boundary condition attempt to translate a point
  by (0, 0).
• Attempt to translate a point by (, ), (, -),
  (-, ), (-, -) amounts.
• Attempt to translate a point between quadrants,
  through the origin, etc.
• Perhaps attempt to induce a numeric overflow and
  see how its handled

15
Unit TestingExample

• scale
• Boundary conditions scale by a factor of 0
  (should make the point (0, 0), regardless of the
  original point) and a factor of 1 (should leave
  the point unchanged).
• Scale by positive and negative factors.
• Perhaps attempt to induce a numeric overflow and
  see how its handled

16
Unit TestingExample

• rotate
• Boundary condition attempt to rotate the point
  (0, 0) by various amounts.
• Boundary condition attempt to rotate a point by
  an angle of 0.
• Attempt to rotate a point by various amounts and
  check the results (probably cover all 4
  quadrants).
• Degenerate conditions attempt to rotate points
  with angle gt 2p, or angle lt 0, and ensure that
  theyre rotated correctly (this only applies if
  we dont have a precondition restricting the
  values of angle).

17
Unit Testing

• Thats a whole lot of tests for one little class,
  and in many cases you wouldnt need to be quite
  so exhaustive.
• Every class for which it makes sense to test in
  isolation should have its own individual suite of
  unit tests (in addition to its assertions).
• Sometimes this requires a test harness, a special
  environment in which to conduct a test of a
  particular component.
• For instance, if youre testing a class that
  implements a dealer in a card game, you need to
  fool it into thinking that it has players to
  deal to

18
Unit TestingImplementation

• Tests can get pretty complex.
• Our tests for Point were all very simple, most of
  them involving only one or two method calls, but
  sometimes it takes a lot more effort to set up a
  particular situation to be tested.
• For example, to test the behavior of a Stack
  class when it reaches its maximum capacity, youd
  have to put in a whole lot of distinct objects,
  then remove them again and ensure that the order
  you got them back in is exactly what it should be.

19
Unit TestingAutomation

• Automation can help a lot, both with creating and
  with running unit tests.
• One thing you absolutely never want to do is to
  sit at a keyboard and interact with a unit test.
• Waste of time - you should be working on your
  design and code, not interacting with some test
  program.
• Error-prone - youre not necessarily guaranteed
  to provide the same input over and over again,
  which makes tests less useful.
• Unit tests should be automated to the maximum
  extent possible.

20
Unit TestingAutomation

• Jtest and JUnit are unit testing automation tools
  for Java, and well be using them this term.
• JUnit is free, and lets you create unit tests
  just by implementing some classes with specially
  formatted methods.
• Jtest is commercial, and can automatically
  generate a unit test suite based on your
  Jcontract assertions. It has the added benefit of
  automatically generating JUnit skeletons for you,
  which you can fill in with your own more complex
  blackbox tests.

21
Integration Testing

• Integration testing can also be automated, in
  much the same ways as unit testing.
• For example, if you write a class that sends and
  receives a bunch of messages through your router,
  that is effectively an integration test.
• Integration testing should only be done once
  youre convinced that every component works on
  its own.
• Integration can cause interesting things to
  happen, especially in concurrent systems, and if
  the individual components dont work correctly in
  isolation its a lot harder to figure out why the
  interesting things happen

22
Interactive Testing

• Interactive testing should usually be the last
  phase of testing.
• Once the system is working according to its
  specification, then you can see how it works when
  subjected to the various abuses that users can
  put on it.
• Interactive testing can often reveal the need for
  more unit or integration tests, when a user does
  something unpredictable.
• User interfaces are typically the components most
  affected by interactive testing.

23
Test Failures

• So we have all these different kinds of tests -
  what happens when one fails?
• Blackbox tests and integration tests are only
  going to fail during explicit testing runs, so
  those failures are handled by tracking down the
  problem and re-running the tests.
• Whitebox tests can fail at runtime even after the
  software has been delivered to clients (assertion
  checks).

24
Test Failures

• When a whitebox test fails, its sometimes
  appropriate to stop the entire system, but there
  are often better ways to handle it.
• Attempt to fix the problem automatically, through
  exception handling, retries, etc.
• Keep the system running but limit its
  functionality (the requested function is
  currently unavailable, please try again later).

25
Homework 3 Overview

• Homework 3 is the first design assignment.
• Goal To carry out a design using Design by
  Contract, and work with more threads.
• Design and implement a thread-safe active router.
• Functionality just like the passive router,
  except that it has its own thread(s) that handle
  message delivery, and messages may be sent to
  multiple destinations rather than a single
  destination.
• Must explicitly copy message objects, so they
  cant be modified after being sent. This will
  likely affect your design.

26
Homework 3 Overview

• What you have to turn in
• A set of class skeletons, containing full Javadoc
  comments and full contracts for the entire public
  API of your router.
• Textual documentation (text, HTML, its up to you
  as long as its not a Word file) describing
  aspects of your design that dont necessarily
  belong in Javadoc comments.
• Homework 4 will be the implementation of this,
  along with a test suite.
• Its extremely unlikely that you will get a grade
  for Homework 3 before having to work on Homework
  4, but well take that into account when grading
  both of them.

27
Next Class

• Formal Reasoning about Concurrent Systems