Chapter 10: Testing

1
Chapter 10 Testing

• Lecturer Dr. Mai Fadel

2
Basic Definitions

• A failure is an unacceptable behavior exhibited
  by a system.
• E.g. the production of incorrect outputs, the
  system running too slowly, or the user having
  trouble finding online help.
• A defect is a flaw in any aspect of the system
  including the requirements, the design, and the
  code, that contributes, or may potentially
  contribute, to the occurrence of one or more
  failures. A defect is also known as a fault.
• e.g. forwarding a message to their selves in a
  chatting program causes the system to hang or
  crash (infinite loop). No documentation or source
  code comments are provided for a complex
  algorithm.

3
Effective and efficient testing

• Testing is the process of deliberately trying to
  cause failures in a system in order to detect any
  defects that might be present.
• To test effectively, you must use a strategy that
  uncovers as many defects as possible.
• To test efficiently, you must find the largest
  possible number of defects using the fewest
  possible tests.
• Black-box testing
• Testers treat a system as a black box, i.e. they
  provide the system with inputs and observe the
  outputs, but the cannot see what is going on
  inside.
• Cannot see neither code, internal data or
  documentation describing the internal of the
  system.
• Glass-box testing
• An alternative approach is to treat the system as
  a glass-box.
• The tester can examine the design documents, the
  code, and observe at run time the steps taken by
  algorithms and their internal data.
• Glass-box is more time consuming, but it removes
  much of the guess work. It is usually applied by
  individuals to test their own code.
• Most of the technique in this chapter are
  oriented toward black-box testing.

4
Equivalence classes a strategy for choosing what
to test

• In order to test efficiently, you should divide
  the possible inputs into groups that you believe
  will be treated similarly by reasonable
  algorithms. Such groups are called equivalence
  classes.
• A tester needs only to run one test per
  equivalence class, using a representative member
  of that class as input.
• The set of equivalence classes for a system as a
  whole in the set of all possible combinations of
  inputs.
• The total number of eq. classes for the system is
  the product of classes of the individual inputs.
• e.g. 6 inputs gt 2x4x11x3x3x4 3168
• Combinatorial explosion of the space required
  the total number becomes very large.
• You cannot realistically test every possible
  system-wide eq. class.
• A reasonable approach
• Test the eq. classes for each individual input.
• The, where possible test all combinations where
  one input is likely to affect the interpretation
  of another.
• In addition to testing a representative value of
  each equivalence class, you should also test
  values at the boundaries of eq. classes (checking
  the extreme values).

5
Detecting specific categories of defects

• The next few sections give a non-exhaustive list
  of some of the most common categories of defects.
• This will help in designing the appropriate test
  cases.
• It will help when designing software (avoiding
  such defects)

6
Defects in ordinary algorithms

• Defects that are most common in all types of
  algorithms.
• More often introduced by the designer or the
  programmer.
• Incorrect logical conditions
• Defect the logical conditions that govern
  looping, and if-then-else statements are wrongly
  formulated.
• Conditions needs completely restructuring
• More subtle defect, such as nesting paranthess
  incorrectly, reversing comparison operators (lt
  becomes gt), or mishandling the equality case (lt
  becomes lt)
• Testing strategy use equivalence class and
  boundary testing. To compute the equivalence
  classes consider each variable used in the
  logical condition as an input.

7
Defects in algorithms performing a calculation
in the wrong part of the control construct

• Defect the program performs an action when it
  should not, or does not perform an action when it
  should do.
• Typically caused by inappropriately excluding the
  action from, or including an action in, a loop or
  if-then-else construct.
• Testing strategy design tests that execute each
  loop zero times, exactly once, and more than
  once.
• Ensure that anything unusual that could happen
  while looping is made to occur on the first and
  last iteration.

8
Defects in algorithms

• Not terminating a loop or recursion
• Ensure they reach a terminating case
• Analyze what causes a repetitive action to be
  stopped, and run test cases that might not be
  handled correctly.
• Not setting up the correct preconditions for an
  algorithm
• Run test cases in which each precondition is not
  satisfied
• Not handling null conditions
• Determine all possible null conditions and run
  the test cases.
• Not handling singleton or non-singleton
  conditions
• Brainstorm to determine unusual conditions
• Off-by-one errors
• Develop boundary tests in which you verify that
  the program computes the correct numerical
  answer, or performs the correct number of
  iterations.
• In graphical applications, study the display to
  see if objects slightly overlap or have slight
  gaps

9
Defects in algorithms

• Operator precedence errors
• Apply code inspection
• Use of inappropriate standard algorithms
• Requires knowledge of the properties of
  algorithms and design test
• e.g. of bad choices
• Inefficient sort algorithm bubble sort.
  Increase the number of items to be sort and
  observe the execution time. It it double four
  time the as long gt inefficient.
• Inefficient search algorithm ensure that the
  search time does not increase unexpectedly. Check
  if the search time is affected by the position of
  the item in the list.
• A search or sort that is case sensitive when it
  should not be, or vice versa check with mixed
  case data.

10
Defects in numerical algorithms

• Numerical computations defects are special class
  of algorithmic defects. They can occur in any
  software that performs mathematical calculations
  (e.g. involving floating point values)
• Not using enough bits or digits to store maximum
  values
• Using insufficient places after the decimal point
  or too few significant figuresgt may cause the
  system to round excessivelygtdata is stored
  inaccurately or lead to a build-up of errors

11
Defects in numerical algorithms

• Ordering operations poorly so that errors build
  up
• Occurs when you do small operations on large
  floating point numbers. e.g. add or subtract 1
  from 3.54 x 1028 the answer will be equal to the
  large number. The defect occurs when the
  programmer intends that the large number should
  be modified. (not enough significant figures)
• Assuming a floating-point value will be exactly
  equal to some other value
• e.g. incorrect -gt d ! 10.0 correct -gt d lt
  10.0

12
Defects in timing and co-ordination deadlocks,
livelocks, and critical races

• Timing and co-ordination defects arise in
  situations involving some form of concurrency.
• Several threads or processes interact in
  inappropriate ways.
• A deadlock is a situation in where two or more
  threads or processes are stopped, waiting for
  each other to do something before they either can
  proceed. Since neither can do anything, they
  permanently stop each other from proceeding.
• A livelock is similar to deadlock, in the sense
  that the system is stuck in a particular behavior
  that it cannot get out of. The difference is
  whereas in deadlock the system is normally hung,
  with nothing going on, in livelocks, the system
  can do some computation, but it can never get out
  of a limited set of states.
• A critical race is a defect in which one thread
  or process can sometimes experience a failure
  because another thread or process interferes with
  the normal sequence of events. The defect is
  not the interference of the other thread, but
  that the system allows interference to occur.
• e.g. two threads are working to achieve some
  outcome, if one sped up or slowed down the result
  will be different (set and get to the same data).

13
Defects in handling stress and unusual situations

• These are encountered only when a system is being
  heavily used, or forced to its limits in some
  other way.
• They represent lack of robustness.
• To test for such defects you must run the system
  intensively, supply it with a very large amount
  of input, run many copies of it, run it on a
  computer or network that is busy running other
  systems.
• Insufficient throughput or response time on
  minimal configurations
• Incompatibility with specific configurations of
  hardware or software
• Defects in handling peak loads or missing
  resources
• Inappropriate management of resources
• Defects in the process of recovering from a crash

14
Writing formal test cases and test plans

• A test case is an explicit set of instructions
  designed to detect a particular class of defect
  in a software system, by bringing about a
  failure.
• A test plan is a document that contains a
  complete set of test cases for a system, along
  with other information about the testing process.
  (a standard type of documentation)
• No test plan gt ad-hoc testinggt poor quality
  software.
• Testing can start long before the testing phase,
  for example writing test cases for use cases
  during requirement analysis.

15
Information to include in a formal test case

• Each test case should have the following
  information
• Identification and classification. test id, a
  descriptive title that indicates its purpose,
  part of the system being tested, the importance
  of the test case.
• Instructions. Tell the tester exactly what to do.
  How to put the system in the required initial
  state and what input to provide.
• Expected results.
• Cleanup (when needed). Tells the tester how to
  make the system go back to normal or shut down
  after the test. e.g. erroneous data be added to
  the database.
• Test cases can be organized into groups or
  tables. (common instructions can be reported to
  the group not each test case)
• It is common to completely automate the testing
  process. Each test case may become a method that
  throws an exception if the test fails. Test case
  would need to report an identification of what
  failed.

16
Levels of importance of test cases

• Classify test cases according to their
  importance, the most important should be executed
  first.
• Level 1 first pass critical test cases. These
  designed to verify that the system runs and is
  safe.
• Level 2 general test cases. These verify that
  the system performs the day-to-day functions
  correctly gt success.
• Important to fix, may still permit testing other
  aspects of the system.
• Level 3 test cases of lesser importance. e.g.
  testing cosmetic aspects (button become
  greyed-out when it is not needed).

17
Strategies for testing large systems

• There are several strategies that can be use to
  test the entire system that has many subsystems
  and thousands of test cases.

18
Integration testing big bang versus incremental
approaches

• Integration testing is testing how the parts of a
  system work together.
• Unit testing is testing an individual module or
  component in isolation.
• Big bang testing (integration) you take the
  entire integrated system and test it all at once.
• Satisfactory, but when failure occurs it may be
  hard to tell in which subsystem a defect lies.
• Incremental testing (integration) you first test
  each individual subsystem in isolation, and then
  continue testing as you integrate more and more
  subsystems.
• Incremental testing can be performed horizontally
  or vertically depending on the architecture of
  the system.
• Horizontal testing can be used when the system is
  divided into separate sub-applications, you
  simply test each sub-application in isolation.

19
Vertical incremental testing

• Can be top-down, bottom-up, or sandwich.
• Top-down testing you start by testing only the
  user interface, with the underlying functionality
  simulated by stubs.
• Then you work downwards, integrating lower and
  lower layers, each time creating stubs for the
  layers that remain un-integrated.
• A stub is a piece of code that have the same
  interface (API) as the lower-level layers, but
  which do not perform any real computations or
  manipulate any real data.
• Any call to a stub will typically return a
  default value.
• Any defect can be reasonably confident that the
  defect is in the layer that calls the stubs.
• Drawback writing the code for stubs. There are
  automated tools for doing so.

20
Vertical incremental testing

• Bottom-up testing you start by testing the very
  lowest levels of the software.
• This might include a database layer, a network
  layer, a layer that performs some algorithmic
  computations.
• You need drivers to test lower layers. A driver
  is a simple program designed specifically for
  testing they make calls to the lower layers.
• A test harness is a driver that fully automates
  the testing of the lower layers.
• Sandwich testing/ mixed testing is a hybrid
  bottom-up and top-down testing testing the user
  interface using stubs, and testing the very
  lowest level functions using drivers.
• The middle layer will remain on which you perform
  the final set of tests.
• The most effective.

21
Deciding when to stop testing

• Not a practical approach go on re-testing
  software until all the test cases has passed.
• Poor strategy to stop testing because merely you
  have run out of time or money gt poor-quality
  system.
• Establish a set of criteria to decide when to
  testing should be complete
• All the level 1 test cases must have been
  successfully executed.
• Certain predefined percentages of level 2 and 3
  test cases must have been executed successfully.
• The targets must have been achieved and then
  maintained for at least two cycle of builds. A
  build involves compiling and integrating all the
  components of the software, incorporating any
  changes since the last build.

22
Testing performed by users and clients alpha,
beta and acceptance

• Testing is first performed by software engineers
  in the development organization.
• The testing process normally involves users.
• It includes the involvements of users in testing
  versions of the system that are almost ready to
  be put into production.
• It starts once the developers believe the
  software has reached a sufficient level of
  quality.
• Alpha testing is testing performed by users and
  clients, under the supervision of the software
  development team (invited users).
• Beta testing is testing performed by the user or
  client in their normal work environment.
• Users know that the software will contain more
  defects but they have access to its functionality
  before the others.
• Beta testers are responsible for reporting
  problems when they discover them.
• Acceptance testing is performed by users and
  clients to decide whether software is of
  sufficient quality to purchase.