Testing

1
Testing
CEN 4010 Class 25 11/22

• Review Class 22
• Testing Activities
• Test Documentation
• Project - Testing

2
Testing Activities

• Unit testing,
• Integration testing,
• System testing.

3
Unit Testing

• Focuses on the building blocks of the software
  system i.e., objects and subsystems.
• In principle, all the objects developed during
  the development process should be tested.
  Usually not feasible due to budget reasons.
• Many unit testing techniques have been devised
  including equivalence testing, boundary testing,
  path testing, state-based testing, data flow
  testing.

4
Unit Testing Equivalence Testing

• Equivalence testing is a blackbox testing
  technique that minimizes the number of test
  cases.
• Possible inputs are partitioned into equivalence
  testing classes, and a test case is selected from
  each class.
• Assumption - system behaves in a similar way for
  all members of an equiv. class.
• Criteria used to determine equivalence classes
  coverage, disjointedness, representation.

5
Unit Testing Equivalence Testing

• Example Valid inputs to test the
  getNumberDaysInMonth() method

Equivalence class Value for month input Value for year input
Months with 31 days, non-leap yrs. 7 (July) 1901
Months with 31 days, leap yrs. 7 (July) 1904
Months with 30 days, non-leap yrs. 6 (June) 1901
Months with 30 days, leap yrs. 6 (June) 1904
Months with 28 or 29 days, non-leap yrs. 2 (February) 1901
Months with 28 or 29 days, leap yrs. 2 (February) 1904
6
Unit Testing Path Testing

• Path testing is a whitebox testing technique that
  identifies faults in implementation of the
  component.
• Selected test cases exercises all possible paths
  through the code at least once.
• Technique requires knowledge of the source code,
  unlike blackbox testing techniques.
• Starting point is the flow graph.
• A flow graph consists of nodes representing the
  flow of control in the source code.

7
Unit Testing Path Testing
Flow graph

• Example Source code
• public int count(int x)
• k 0
• while (x lt 10)
• if (x2 ! 0)
• k k 1
• x x 1
• return k

1
K0
while(xlt10)
2
if(x2 ! 0)
3
kk1
4
Select test cases for x to cover the following
paths 1, 2, 6 x ? 1, 2, 3, 5, 2, 6 x ? 1,
2, 3, 4, 5, 2, 6 x ?
xx1
5
return k
6
Every statement and every branch (both outcomes)
has been executed once.
8
Testing Activities cont

• Integration testing
• Detects faults by focusing on small groups of
  components e.g. methods, classes, packages.
• Two or more components are integrated and tested,
  if no faults are revealed then additional
  components are added and tested.
• Most recently added component is usually the one
  that triggers the most recently discovered fault.
• Note the order in which components are integrated
  can influence the total testing effort i.e. the
  creation of stubs and drivers.

9
Integration Testing cont

• Several approaches
• Big bang testing
• Bottom-up testing
• Top-down testing
• Ordering based on dependencies of components
• Big bang assumes that all components are first
  tested individually and then tested together as a
  single system.
• Debugging is a problem, i.e., where is the fault?
• Reduces the of stubs and drivers required.

Traditional
10
Integration Testing cont

• Bottom-up individually test each component of
  the bottom layer and then integrate them with
  components of the next layer (assumes a
  hierarchical structure).
• Drivers required to simulate the components of
  higher layers.
• No test stubs are required.
• Top-down unit test the components of the top
  layer first and then integrate the components of
  the next layer down.
• Stubs used to simulate the components of lower
  layers
• Drivers not required.

11
Integration Testing cont

• Ordering based on dependencies a dependency
  graph is created (similar to a class diagram),
  then an order is generated. The leaf
  components are tested first then the inner
  components.
• Problem if graph contains cycles.
• Uses stubs in the event of cycles.
• (A Parameterized Cost Model to Order Classes for
  Class-based Testing of C Applications. Malloy,
  et al. ISSRE 2003.)

12
Testing Activities cont

• System testing
• Once components have been integrated, system
  testing ensures that the complete system complies
  with the functional and nonfunctional
  requirements of the system.
• At this point most of the faults in the
  individual components and the interfaces have
  been identified and corrected.
• Testing activities during system testing
• Functional testing
• Performance testing

13
System Testing cont

• Testing activities during system testing cont
• Pilot testing
• Acceptance testing
• Installation testing
• Functional testing (or requirements testing) -
  finds differences between the functional
  requirements and the system.
• Blackbox technique
• Test cases derived from the use case model
• Tester selects those tests that are relevant to
  the user and have a high probability of
  uncovering failures.

14
System Testing cont

• Performance testing finds differences between
  goals (nonfunctional requirements) selected
  during system design and the system. Following
  tests are performed
• Stress testing checks if the system can respond
  to many simultaneous request.
• Volume testing attempts to find faults
  associated with large amounts of data.
• Security testing few systematic methods for
  finding security faults. Use experience hackers
  to break into the system to expose faults.

15
System Testing cont

• Timing tests attempts to find behaviors that
  violate timing constraints.
• Recovery tests evaluate the ability of the
  system to recover from errors e.g.,
  unavailability of resources, a hardware failure,
  or a network failure.
• Pilot testing (or field test) - system is
  installed and used by a set of users. Two types
  of test
• Alpha test is a pilot test with users
  exercising the system in the development
  environment. (Bespoke)
• Beta test acceptance test performed by a
  limited number of end users in the target
  environment. (Generic)

16
System Testing cont

• Acceptance testing final stage in the testing
  process before the system is accepted for
  operational use. Three flavors
• Benchmark test client prepares a set of test
  cases that represent typical conditions under
  which the system should operate.
• Competitor test new system is tested against an
  existing system or competitor product.
• Shadow testing new and legacy systems are run
  in parallel and their outputs are compared.

17
System Testing cont

• Installation testing system is installed in the
  target environment and evaluated.
• Repeats the test cases executed during function
  and performance testing in the target
  environment.
• Some requirements cannot be tested in the
  development environment so new test cases are
  developed.

18
Other Testing Activities

• Regression Testing
• Recall OO development is an iterative process.
• Developers modify, integrate, and retest
  components often, as new features are implemented
  and/or existing features are improved.
• Note, a modification can introduce side effects
  or reveal previously hidden faults in other
  components.

19
Other Testing Activities cont

• A system that fails after the modification of a
  component is said to regress.
• Regressing testing is a testing activity whereby
  test are rerun to ensure that the system does not
  regress after a change i.e., the system passes
  all the tests it did before the change.
• There are several approaches used in regression
  testing
• Retest all
• Selective retest

20
Other Testing Activities cont

• Automated Testing
• Manual testing involves a tester
• feeding predefined inputs into the system using
  the user interface, a command line console, or a
  debugger,
• comparing the outputs generated by the systems to
  the expected oracle.
• Manual testing can be costly and error prone.
• The repeatability of test execution can be
  achieved with automation.

21
Other Testing Activities cont

• The focus of test automation has been the
  execution of test cases.
• Automation of systems test include specification
  of tests in terms of sequence and timing of
  inputs and an expected output trace.
• The test harness executes a number of test cases
  and compares the system output with the expected
  output trace.

22
Documenting Testing

• Test Plan focuses on the managerial aspects of
  testing. Documents the scope, approach,
  resources, and schedule of testing activities.
• Test Case Specification each test is documented
  by a test case specification . Document includes
  inputs, drivers, stubs, expected outputs of the
  tests, and task to be performed.
• Test Incident Report each execution of each
  test is documented i.e., the actual results of
  each test and differences from the expected
  output are recorded.

23
Documenting Testing

• Test Summary Report list all the failures that
  were discovered during the tests and those needed
  to be investigated.
• Developers analyze and prioritize each failure
  and plan for changes in the system and in the
  models.
• Changes can trigger the creation of new test
  cases and new test executions.

24
Project- Testing

• Testing Process
• Introduce the testing chapter.
• 7.1 System Tests use format given in class
  (see the example on the class web page).
• 7.2 Subsystem Tests test at least one
  subsystem or class using a specification-based
  testing technique. This will involve the
  creation of a test driver i.e. a main in the
  package containing the subsystem or class. Use a
  similar format to that in section 7.1 to document
  the tests performed. Include the code for the
  test driver in appendix G.

25
Project- Testing

• 7.2 Evaluation of Tests evaluate how
  successful the tests were in terms of adequacy
  and coverage (see example on the class web page).
• Appendix G Test Driver.
• Go through test case format on web page.