CIS224 Software Projects: Software Engineering and Research Methods

1
CIS224Software Projects Software Engineering
and Research Methods

• Lecture 7
• Product Quality Verification, Validation and
  Testing
• (Based on Stevens and Pooley, 2006, Chapter 19)

David Meredith d.meredith_at_gold.ac.uk www.titanmus
ic.com/teaching/cis224-2006-7.html
2
Introduction

• To build high-quality systems, we need techniques
  that focus on
• The quality of the product
• The quality of the process
• Product-focused techniques for ensuring software
  is of high quality
• Verification
• Making sure weve built the product right
• Product satisfies design and specified
  requirements
• The purpose of Verification (VER) is to ensure
  that selected work products meet their specified
  requirements. (Capability Maturity Model
  Integration for Development, version 1.2)
• Validation
• Making sure weve built the right product
• Design satisfies real-world requirements and fits
  intended usage
• The purpose of Validation (VAL) is to
  demonstrate that a product or product component
  fulfills its intended use when placed in its
  intended environment. (Capability Maturity Model
  Integration for Development, version 1.2)
• Testing is used for both verification and
  validation

3
Quality review

• Software is high quality if it meets the users
  requirements
• Useful and usable
• Makes peoples lives easier or better
• Reliable
• Few bugs
• Must be easy to thoroughly test software
• Flexible
• Easy to change software in response to
  requirements change
• Easy to maintain and debug software
• Affordable
• Not too expensive to buy and maintain
• Implies easy and quick to develop and maintain
• Available
• Available for different platforms
• Development must complete

4
How can high quality software be achieved?

• Need to combine two orthogonal approaches
• Focus on product
• Testing for verification and validation
• Focus on process
• Faults in product may be due to faults in process
• Need to evaluate process frequently and improve
  it when necessary

5
Verification

• Involves testing product against specification
• If using UML, then, in testing phase of each
  iteration, we might
• Verify that use cases satisfy requirements
• Verify that classes can provide use cases
• Verify that code correctly implements classes
• Includes sanity checks that ensure project
  satisfies basic requirements, e.g.,
• code compiles cleanly
• program runs without errors (e.g., segmentation
  faults, unhandled exceptions)
• UML diagrams syntactically correct

6
Verification

• Verification can be
• Informal developer manually compares code with
  design and ensures they correspond
• Best if tester not the same person as the
  programmer
• Devise a checklist of most common problems to
  look for
• Reviews
• Formal formal proof of equivalence of code and
  design
• Rarely done in practice
• Only possible if two things being compared have
  formal, well-defined semantics
• UML does not have formal semantics, therefore
  formal proof of equivalence of UML with code is
  impossible in principle
• UML can be guaranteed syntactically correct if
  produced using a UML modelling tool that enforces
  correct syntax
• Equivalence of UML with code presumably ensured
  if using MDA or executable UML

7
Validation

• Validation harder than verification because dont
  know exactly what youre looking for
• Have to identify ways in which system is less
  useful to the customer than it should be
• Implies requires customer involvement
• Worst errors arise from communication failure
  between customer and developer on what is
  required
• Need to have frequent customer evaluations as in
  iterative process
• May need to build rapidly a throw-away prototype
  to ensure that customer and developer have same
  idea of what is required

8
Usability

• System must not just provide required
  functionality
• Must also allow users to exploit functionality
  effectively they must be able to
• Carry out their tasks quickly and easily with
  minimum stress
• Recover from errors
• Developers bad at evaluating usability of their
  software
• Real users make mistakes that developers could
  never imagine
• If understand how something can be done, hard to
  imagine not understanding it
• Developers usually not typical users
• Typically understand system better than users and
  domain less well than users
• Employ expert in usability and consult users
  frequently
• Thomas Landauer proposes user-centred design
• One days focus on usability can increase work
  efficiency of system by 25
• Landauer, Thomas K. (1996). The Trouble with
  Computers Usefulness, Usability and
  Productivity. MIT Press, Cambridge, MA.

9
Testing

• Testing contributes more to verification than
  validation
• Can test designs and system components as well as
  complete running systems
• Testing has three aims
• To find bugs (most important aim)
• To convince customer that there are no bugs
• To provide information for system evolution, e.g.
• Information on future requirements
• Information on present performance

10
Testing

• A successful test is one that finds a bug
• Different types of testing
• Usability testing checks if system is easy to
  use effectively
• Module (unit) testing tests individual modules
  of a system
• in OO, modules will usually be classes
• Integration testing tests that parts of system
  work together properly
• System testing checks that system meets
  functional and non-functional requirements
• Acceptance testing validates that system is fit
  for purpose
• Performance testing checks satisfactory
  performance of module, collaborating modules or
  system
• Stress testing puts extreme loads on system to
  ensure that it degrades gracefully and does not
  fail catastrophically
• Regression testing tests that a change to a
  system does not introduce new bugs. Includes
• Module tests for changed modules
• Integration tests for subsystems including
  changed modules
• Some whole-system tests (usability, system,
  acceptance, performance, stress)

11
Choosing and carrying out tests

• Tests can be
• Black box
• Chosen by looking at specification of thing to be
  tested
• White box
• Chosen by looking at structure of thing to be
  tested
• Petschenik, N. H. (1985). Practical priorities in
  system testing. IEEE Software, 2(5), pp.18-23.
• It is more important to
• Test whole system than its components
• Check that the system can still do what it could
  do before than to check that new features work
• To test typical cases than boundary value cases

12
Choosing and carrying out tests

• When functionality changes, must carry out
  regression tests which must be
• Repeatable
• Documented (both tests and results)
• Precise
• In iterative development, same test carried out
  many times
• Need automated testing
• There exist specialist testing tools (e.g.,
  junit)
• But can also automate tests using scripts (e.g.,
  Perl, shell scripts, DOS batch files, etc.)
• Write test specifications as soon as requirements
  are understood
• Some bugs only show up after complex and unusual
  sequence of events
• Have someone test software with the aim of
  breaking it, however sneakily
• e.g., IBMs Black team (DeMarco, T. and Lister,
  T. (1987). Peopleware Productive Projects and
  Teams. Dorset House, New York.)
• Especially hard to test GUIs systematically
• Many problems with GUIs are to do with usability
• Usability testing cannot be automated

13
Unit testing in an OO system

• Special problems arise when testing an OO system
• What counts as a unit?
• Checking hidden data in an encapsulated unit
• Problems arising from inheritance and polymorphism

14
What counts as a unit?

• Unit of test must be at least the class
• Can be a collection of collaborating classes
  (e.g., a package)
• Class harder to test than a function
• Cannot test each method in isolation because a
  method might change an objects state in such a
  way that reveals a bug in another method
• Can use state machine diagrams to identify
  important states and then test object on every
  state transition
• Avoid classes with complex state diagrams since
  they are hard to understand and test!

15
Testing and encapsulation

• Encapsulation can reduce likelihood of bugs by
  not allowing access to details that client does
  not need to know about
• But checking which state an object is in during a
  test may require access to private attributes
• One solution might be to provide a method that is
  only used during testing but then rely on
  correctness of this testing method!

16
Inheritance and testing

• Class D overrides foo() operation in class G but
  not bar() operation?
• Do we only need to test the foo() operation in
  class D?
• NO! Why?
• Because bar() may call foo()
• because of dynamic binding, class D objects
  foo() method might be called when bar() message
  sent to it
• To be safe, all subclasss methods must be tested
  even inherited ones!

17
Testing and polymorphism

• Two interacting objects a of class C and b of a
  subclass of D
• Could be that class of b was invented after C
• Must make sure that way in which b behaves when
  sent messages in the public interface of class D
  is consistent with how an object of class D would
  behave in response to same messages
• This must be tested when class of b is defined
• Class tightly coupled with its superclasses which
  creates difficulties when testing
• USE INHERITANCE ONLY WHEN ITS ADVANTAGES OUTWEIGH
  ITS DISADVANTAGES!

18
Why is testing often done badly?

• Most people find it boring!
• Need to automate process as much as possible
• Need to minimise bureaucracy
• Its expensive
• Could take 30-50 of project time
• If left until end of project, then squeezed by
  deadline pressure
• Hence should use iterative process in which
  testing done in each iteration and spread over
  whole project
• Customers often dont realise the importance of
  testing and pressure developers to deliver
  product even if it has not been thoroughly tested

19
Reviews and inspections

• Formal technical reviews (FTR) help mostly with
  validation but also with verification
• Meeting to find problems in a deliverable (could
  be code or design)
• FTR participants include author, moderator,
  scribe, possibly one or two users
• Artifact studied by participants before meeting
• Meeting focuses on defects found, not solutions
• After meeting, author addresses defects and then
  meets with moderator to discuss solutions
• Problems with FTRs
• Can feel like an attack on the developers
• Can be very time-consuming
• Can produce long lists of trivial defects and
  miss more fundamental problems

20
Summary

• To achieve high-quality product, need
  product-focused and process-focused techniques
• Concentrated on product-focused techniques in
  this lecture
• Verification - making sure the product satisfies
  the stated requirements
• Validatation - making sure the product if fit for
  its intended purpose in the environment in which
  it will be used
• Testing required to verify and validate a product
• Reviewed criteria for a high quality software
  system
• Useful, usable, reliable, flexible, affordable,
  available
• Verification involves testing product against
  specification
• can be informal (e.g., manual comparison)
• or formal (e.g., proof of equivalence)
• Validation involves identifying ways in which
  system is less useful to customer than it should
  be
• necessitates customer involvement
• need to have frequent customer evaluation
• Usability - consult users frequently and employ
  expert in usability
• Testing more relevant to verification than
  validation
• Different types of testing usability, unit,
  integration, system, acceptance, performance,
  stress, regression
• Black box testing specification
• White box testing structure
• Special problems with unit testing in an OO
  system