Levels of Software Applications

1
Levels of Software Applications

• In any medium to large software systems, there
  are many programs (sometimes thousands) that
  formulate various levels of functions - to
  components - to a complete application. For
  example an typical ERP package such as SAP or
  PeopleSoft may be composed of multiple layers of
  application sub-systems
• Total ERP package to satisfy a wide range of
  requirements includes
• Human Resource
• Benefits mgmt
• Payroll
• Manufacturing
• Production Planning and Scheduling
• Inventory
• Distribution
• Warehouse mgmt
• Logistics
• Finance
• .
• .

2
System Test

• System test is concerned with testing the entire
  system which may be composed of many very large
  sub-systems, which may themselves be large
  enough to be considered a system.
• The problems we encounter in system test may have
  originated from any of the previous activities
• Requirements Processing
• Designing
• Programming
• Fixing Defects
• System integration and build
• Prior to this stage, we have focused on defect
  discovery and fixing now, at system test, we are
  more interested in seeing if the whole system
  works - - - a slight shift in mentality.

3
Ingredients of Systems Test

• Functionality test
• This part is looking at the interactions among
  the sub-systems and delivering joint
  functionality more than just individual, simple
  function.
• e.g. payroll in HR sub-system should affect the
  accounts payable and general ledger functions in
  the Financial subsystem - - - giving us an
  updated ledger.
• Performance test
• This part is looking at several areas of the
  over-all system
• Throughput
• Response time
• Security
• Reliability Recovery
• Acceptance Test
• This part ensures that, while the developers may
  be happy with the system, the actual users are in
  fact happy with the system
• Functionality completeness
• Usability in terms of human interaction with the
  system (a la prototype)
• Reliability (in terms of error tolerance and
  undo or re-do)
• Installation Test
• This is ensuring that the software can be
  installed in the users environment
• System match
• DB match
• Network match

4
1. Functionality Test

• During system test we are not so much testing
  individual simple functions, but focused more on
  complex sets of functions that may be
  inter-related by combination of conditions.
• Black box testing techniques are mostly used here
  to check out high level.
• A popular technique is to use the decision
  table or truth table approach from logic.

5
Components of a Decision Table
rules
R1 R2 R3 R4 R5 R6 R7 R8
T T T T F F F F
C1 C2 C3
values of conditions
T T F F T T F F
conditions
T F T F T F T F
a1 a2 a3 a4 a5
x x x x
x
x
actions taken
x x

actions
x x x x
x x

Read a Decision Table by columns of rules R1
says when all conditions are T, then actions a1,
a2, and a5 occurs
6
Decision Table and Test Cases

• Each rule may be viewed as a test case.
• If we want to exercise all the combinations of
  conditions, then we would have to have 2n test
  cases for n conditions that take on binary
  values.
• If we want to cover all the actions, we may or
  may not need to execute all the combinations of
  conditions. (see previous slide)

The hard part is to be able to construct the
decision table. One has to possess some domain
specific information - - - testers should include
some requirements analysts
7
2. Performance Test

• Performance tests include a long list of
  different tests
• Stress and Volume test to evaluate the systems
  capability to handle full or more than full level
  of processing (e.g. users, devices, transactions,
  storage, etc.)
• Configuration, environment, and compatibility
  test to ensure that the application does run on
  different platform configurations, as specified
  in requirements and sales brochure.
• Timing test in terms of response time and
  transaction time must be met, especially for
  real-time processing
• Security test ensures that the integrity,
  confidentiality, and protection functions perform
  properly
• Recovery test ensures that the system responds
  properly to errors and loss of information, or
  loss of device.
• Documentation test ensures that the specified
  documents exist, are accurate, and matches the
  system.
• Human Factors and Usability test ensures that the
  system meets the users ease of use requirements
• Quality test evaluates the systems reliability,
  availability and maintainability.

8
Quality Tests

• From earlier testing discussions, we know that
  quality may be looked from many different
  perspectives. One is to consider the following 3
  areas
• Reliability of the system proper behavior of the
  system over a period of time
• Availability of the system the accessibility of
  the system at any instant of time
• Maintainability of the system the fixability or
  supportability of the system

Note that these quality tests all deal with some
aspect of defect. - System may have defect and
still run for a long period of time --- is it
reliable? - System may have defects and still be
available for processing in other functional
areas - - - is it available? - System may have
defects that can be easily fixed and be back
running again - - - is it maintainable?
9
Some Measurements related to Quality

• Meantime to Failure measure the average time
  between failures and sometimes used to describe
  the reliability of a system
• Meantime to Fix measure the average time to
  repair a bug in the system and sometimes used to
  describe the maintainability of a system.
• Meantime between fixes measures the meantime
  between failure including the time when the
  system is being fixed. It is sometimes used to
  measure availability of the system. (This may
  seem strange in that fix time is considered
  available and only instance of failure is
  considered as unavailable points)

10
Reliability theory

• Reliability theory works with the quality
  measurements and probability of failure.
• It builds models (probability distribution of)
  for failures based on historical data
• Use that model to predict the future failures
• Some models take into account of the problems
  fixed and the impact before next failure some
  take into account of the severity of the problems
  fixed and the impact before the next failure
  some take into account of the operational
  profile.
• Some names in the software reliability theory
  field include
• Musa
• Littlewood
• Jelinski
• Moranda
• Ianino
• Okumoto

11
Does the Severity of Defect Matter?

• In looking at quality, we need to realize that
  not all defects are the same in terms of their
  severity of impact.
• We may categorize problems found in terms of
  severity of impact to the users or severity of
  impact to the fixers. (Which one would you
  choose?)

US Military STF 1629 has defined four different
severity levels of problems 1. Catastrophic
a failure that may be life threatening or bring
the total system
down. 2. Critical a failure that may cause
severe injury or major system damage which
may cause some permanent loss of data
or function 3. Marginal a failure that may cause
minor injury or minor system damage that
may result in some system delay or
temporary loss of data 4. Minor a failure not
serious enough to cause injury or system damage
Clearly high severity problems matter more when
considering quality - - - and fixing
12
3. Acceptance Test

• This is the testing of the system mainly by users
  and customers (sometimes with the help of testers
  and developers) to ensure that the system is what
  the users need - - - may not be exactly what the
  users said in the requirements document.
• It is NOT obvious that users will automatically
  perform this - - - it needs to be planned
  because users and customers all have fulltime
  jobs.

13
Types of Acceptance Tests

• Benchmarking
• Users have some very special needs (performance
  or functional) and want to make sure that those
  are met.
• Users will be preparing those special test cases
• Sometimes used for comparing and purchasing
  existing applications.
• Pilot test
• Users want to make sure that the typical,
  everyday usage is covered.
• Test all functions and all major business flows
• May be in alpha (in house users) and beta
  (external users) forms
• Parallel
• Testing the new system next to the existing
  system for a period a time
• Allows comparison and contrast to the existing
  system
• Users gradually transition to the new system

14
4. Installation Test

• If user acceptance test is performed and a broad
  set of user environments (all the combination of
  platforms and interfaces) are completed, then
  this test may be skipped. (Beware that user
  sites may still have something that may conflict
  with the package to be installed!)
• Otherwise ensure that the specified combinations
  in requirements or sales brochures are tested
• All the different versions of operating systems
  are tested
• All the different versions of data bases are
  tested
• The network and different interfaces are tested
• The default data and initialization for different
  combination of applications are tested (e.g. HR
  with Financial)

15
The Effect of Software Process to Testing

• Software process which develops a software by
  increments will naturally need to consider
  integration and system testing techniques
• Incremental development , where a preliminary
  high level design of the whole software is
  developed by parts (possibly in parallel), must
  be coded and tested separately and then
  integrated together and retested.
• Evolutionary development , where a small, but
  well understood, part of the software is first
  designed, coded and tested with additional parts
  developed later, must perform continuous
  integration and testing of the software
• Spiral development , where pieces of software
  evolve through planning, risk assessment, design,
  coding, and testing, must integrate and retest
  as the pieces are accumulated. (much like
  evolutionary)
• Prototype specification based and agile process ,
  where the software or parts of the software are
  prototyped (specified) and developed
  separately, must integrate and test these
  separately specified and developed software.

The result of these processes is that we will
have at least two systems (1) tested and
running and (2) new functionalities and fixes
to the running system
16
System Builds

• Many of the large systems are built in parts and
  integrated into the system.
• The integration and build process is a
  significant component for system or large
  sub-system testing.
• A build (sometimes called a spin) is composed of
• Specifying all the required pieces that make up
  the system
• Mapping out the sequence (pre-requisites and
  co-requisites) for compile and link-edit
• Compiling all the source pieces
• Link-edit all the executables.
• In large system testing, the build (spin) cycle
  may take several hours. The number of system
  tests that can be performed each day depends on
  how many spins can be performed per day.

17
Builds for Multiple Platforms

• A software vendor often provide its application
  for multiple platforms (specified in the sales
  brochure)
• Operating systems (versions of UNIX, MS, Apple)
• Database (versions of Oracle, IBM DB2, Sybase)
• A system test must be performed for several or
  all combinations (e.g. Solaris 10 and Oracle
  11i)
• The application must be compiled and linked with
  each of the combinations of platforms
• Run some of the system test cases that will
  exercise the database and some unique features of
  the operating system

expensive to do
18
Daily System Tests Cycle
2
Merge all the new code fixes
3
Perform the build or the spin
4
Perform System Test
4
1
Record the result report problems
Defects fixing
Prior fixes
19
Regression Testing

• Regression testing is a testing activity to
  ensure that
• the previously running code continues to run
  (does not regress) and
• the new code also works
• This is important whenever we are
• Merging in the last round of problem fixes (as we
  complete system test)
• Merging in the new sub-system (as we integrate
  the incrementally built system and conduct system
  test)

20
Regression Test Steps

• Major steps include
• Merge in the new or changed code
• Testing the areas of the old code that is known
  to be affected by the newly merged code
• Testing the essential functionalities of the
  old code to ensure that all the previously
  working code still works. (possibly using
  equivalence class testing to save some effort)
• System test the new code

The high expense in regression test is step 3
How much of the old, essential functionalities
should one test? - Some just re-run the
complete system bucket that was used before. -
Others choose test samples from each major
functional area (similar to equivalence
partitioning) - Or pick the buggy areas and
focus on those.
21
Regression Test and Re-Builds

• During Regression Test and through out System
  Test, the build cycle may be very long because of
  the large number of parts to merge, compile and
  link.
• There are different reasons for several
  rebuilds
• A new part does not compile or link with existing
  parts (the new part may need to be backed out,
  rebuild without the new part, and proceed to
  test)
• A working old part may not compile or link
  anymore (look for the cause, back out any
  potential cause, rebuild, and test until changes
  arrive)

22
Testers Needed During System Test

• Unit, Functional and Integration (Component)
  tests were mainly the domain of the developers
  and the testers.
• System Testing may include several other
  participants
• Users and customers (especially for acceptance)
• Requirements analysts (performance and
  requirements validation)
• Other Testers who are more familiar with the
  application domain (functionality - - - set up
  intertwined relationships)
• Support and Maintenance personnel (functionality
  test and preparation for post release)