Software Testing

1
Software Testing

• COM 3220

2
Three meanings of bug

• error mistake made by a developer. Mostly
  located in peoples head.
• fault an error may lead to one or more faults.
  Faults are located in text of program.
• failure execution of faulty code may lead to one
  or more failures. A failure occurs when there is
  a difference between the results of the correct
  and incorrect programs.

3
Failure detection

• Compare actual output to expected output.
• Expected output is from specification.
• Specification any external, independent
  description of the program, including user
  documentation.
• Are often incomplete, incorrect, ambiguous or
  contradictory. Specification may be wrong, not
  the program!

4
Motivation

• Derive tests from both the specification and the
  program.
• Derivation is done by predicting likely
  programmer errors or likely program faults.
• Use general rules, e.g., always test boundary
  conditions.

5
Motivation

• Check for faults of omission missed special
  cases.
• Most common type of fault according to a study by
  Glass.
• Experienced testers have a catalog of programming
  cliches and associated errors available. See Test
  Requirement Catalog (low-level omissions).

6
Motivation

• First requirement of test design Be methodical.
  Three stages
• Finding clues
• sources for test requirements
• Expanding them into test requirements
• useful sets of inputs that should be considered
• Writing test specifications
• exact inputs and expected outputs

7
Clues

• What needs testing? Collect from specification,
  program, bug reports, etc.
• Create a checklist.

8
Test requirements

• Create a test requirement catalog

9
Test specifications

• Describes input and exact expected output.

10
Supplementary code inspections

• Some faults that testing is poor at detecting.

11
Test implementation

• Avoid having to write a lot of support code.
• It is better to test larger subsystems because
  less support code needs to be written.
• Individual routines are exercised more.
• Testing the tests test coverage as a crude
  measure.
• During test design do not pay attention to
  coverage criteria.

12
Test implementation

• During test design do not pay attention to
  coverage criteria. Test requirements from other
  sources should do that anyway.
• Complete subsystem testing will usually result in
  high coverage.
• Treat missed branches as clues about weaknesses
  in the test design.

13
Subsystem Specification
Subsystem Code
Catalogued Past Experience
Clues and Test Requirements
Program and Specification Changes
Coverage
Test Specifications
Bug Reports
Implemented Tests
14
Application

• Graph algorithms
• Depth-first traversal
• Finding all paths satisfying some restrictions.
• Happens to be be a subsystem of Demeter/Java.
• You dont have to know anything about Demeter.
  You will learn the minimal things you need.

15
Use Java to write testing code

• You will need to write some Java code for testing.

16
Part of Demeter/Java
Graph traversal
Subsystem Specification
Subsystem Code
Catalogued Past Experience
Clues and Test Requirements
Program and Specification Changes
Coverage
Test Specifications
Use Java/Scope
Bug Reports
Implemented Tests
17
What we want to test

• Given a directed acyclic graph G (no
  multi-edges), traverse all paths from A via B to
  C.
• Given a directed acyclic graph G (no multiedges),
  traverse all paths from A bypassing B to C.

18
Notation for describing graphs

• A B C D. // node A has three successors
• B E. // node B has only one successor
• E . // E has no successor
• This information is put into a file program.cd.
• Two files program.beh are given. Contains the
  traversal specification. Counts visits of C.

19
How to call the program

• demjava test
• The program will print the paths it traversed and
  print how often it visits C.

20
Clue list from A via B to C

• What does program do if there is no path from A
  via B to C?
• What if A or B or C do not appear in the graph.
• Check that paths from A to C not going through B
  are excluded paths of length 1, 2 or 3.

21
Clue list From A bypassing B to C

• What does program do if there is no path from A
  bypassing B to C?
• What if A or B or C do not appear in the graph.
  Is it ok if B does not appear?
• Check that paths from A to C going through B are
  excluded paths of length 1, 2 or 3.

22
Test specifications From A via B to C
AC B X. BC X. C. XC.
AC B. BC. C.
A
A
A
AB BC. C.
B
B
B
C
C
X
C
2 visits
1 visit
1 visit
23
Test specifications From A via B to C
AC B X Y. YB. BC X. C. XC.
A
Y
B
C
X
4 visits
24
Test specifications From A bypassing B to C
AC B X Y. YB. BC X. C. XC.
A
Y
B
C
X
2 visits
25
Fundamental Assumptions of Subsystem Testing

• Most errors are not very creative. Methodological
  checklist-based approaches will have a high
  payoff.
• Faults of omission, those caused by a failure to
  anticipate special cases, are the most important
  and most difficult type.
• Specification faults, especially omissions, are
  more dangerous than code faults.

26
Fundamental Assumptions of Subsystem Testing

• At every stage of testing, mistakes are
  inevitable. Later stages should compensate for
  them.
• Code coverage is a good approximate measure of
  test quality. Must be used with extreme care.

27
A summary of subsystem testing

• Build the test requirement checklist
• Find clues
• Expand clues into test requirements
• Design the tests
• Combine requirements into tests
• Check tests for common testing mistakes
• Supplement testing with code inspections

28
A summary of subsystem testing

• Implement test support code
• Implement tests
• Evaluate and improve tests
• use code coverage tool
• find undertested or missing clues
• find more test requirements
• write more test requirements

29
(No Transcript)
30
Test coverage tool

• For example For each traversal, which fraction
  of traversal methods are used?
• How often is each adaptive method called?
• Define global counters in Main class.
• Use aspect language to instrument code. Generate
  code.
• Testing tool development.

31
Course ideas
Advanced OO systems develops testing tools
for testing class? Test UML graphical editor.
32
Test strategies

• a systematic method used to select and/or
  generate tests to be included in a test suite.
• effective likely to reveal bugs
• Kinds
• behavioral black-box functional
• structural white-box glass-box testing
• hybrid

33
Testing strategies

• behavioral black-box functional
• based on requirements
• structural white-box glass-box testing
• based on program (coverages)
• hybrid
• use combination

34
Classification of bugs

• unit/component bugs
• integration bugs
• system bugs

35
Generic Testing Principles

• Define the graph
• Design node-cover tests (tests that confirm that
  the nodes are there)
• Design edge-cover tests (that confirm all
  required links and no more)
• Design loop tests

36
Generic Testing Principles Example

• Define the graph
• UML class diagram
• Design node-cover tests (tests that confirm that
  the nodes are there)
• Build at least one object of each class
• Design edge-cover tests (that confirm all
  required links)
• use each inheritance edge and association

37
Generic Testing Principles Example

• Define the graph
• Finite state machine
• Design node-cover tests (tests that confirm that
  the nodes are there)
• Use each state at least once
• Design edge-cover tests (that confirm all
  required links)
• use each state transition at least once

38
(No Transcript)
39
Quality factors

• Correctness
• conform to specification
• Maintainability
• ease with which software can be changed
• corrective error fixing
• adaptive requirement changes MAJORITY
• perfective improve system
• Portability

40
Quality factors

• Testability
• how easy to test? Are requirements clear?
• Usability
• effort required to learn and operate system
• Reliability mean-time between failures
• Efficiency use of resources
• Integrity, Security

41
Quality factors

• Reusability
• Interoperability
• Write Quality Manual to address those issues

42
ISO 9000 Series of Standards(5 years old)

• How can customers judge the competence of a
  software developer?
• Adopted by 130 countries.
• ISO 9001 Quality Systems - Model for Quality
  Assurance in Design, Development, Production,
  Installation and Servicing. (general design)

43
ISO 9000 Series of Standards(5 years old)

• ISO 9000-3 Guidelines for the Application of ISO
  9001 to the Development, Supply and Maintenance
  of Software.
• ISO 9004-2 Quality Management and Quality System
  Elements

44
Automatic Verification of Industrial Designs

• Based on two papers in Workshop on
  Industrial-Strength Formal Specification
  Techniques, 1995, Boca Raton, Florida, IEEE
  Computer Society
• Automatic Verification of Industrial Designs,
  pages 88-96
• Timing Analysis of Industrial Real-Time Systems,
  pages 97-107

45
Successful formal methodsin industry

• Formal methods are mathematical techniques that
  have been used in the specification and
  verification of computer systems.
• Want to know Are we building the product
  correctly? (Different from are we building the
  right product).

46
Formal methods

• Many different specification languages and proof
  techniques.
• Some are difficult to apply since computers are
  not good at proving theorems (they need a lot of
  human help)
• Exception Symbolic Model Checking Fast, based
  on OBDD techniques (Ordered Binary Decision
  Diagrams).

47
Symbolic Model Checking

• Determine correctness of finite state systems.
• Developed at CMU by Clarke/Emerson
• Specifications are written as formulas in a
  propositional temporal logic.
• Temporal logic expressing ordering of events
  without introducing time explicitly

48
Temporal Logic

• A kind of modal logic. Origins in Aristotle and
  medieval logicians. Studied many modes of truth.
• Modal logic includes propositional logic.
  Embellished with operators to achieve greater
  expressiveness.
• A particular temporal logic CTL (Computation
  Tree Logic)

49
Computation Tree Logic

• Used to express properties that will be verified
• Computation trees are derived from the state
  transition graphs
• State transition graphs unwound into an infinite
  tree rooted at initial state

50
Computation Tree Logic

• CTL formulas built from
• atomic propositions, where each proposition
  corresponds to a variable in the model
• Boolean connectives
• temporal operators. Two parts
• path quantifier (A, E)
• temporal operator (F,G,X,U)

51
Computation Tree Logic

• Paths in tree represent all possible computations
  in model.
• CTL formulas refer to the computation tree

If the signal req is high then eventually ack
will also be high
52
Computation Tree Logic

• path quantifier (A, E)
• A true for all paths from a given state
• E true for some paths from a given state
• temporal operator (F,G,X,U)
• F? (? holds sometime in the future) is true of a
  path if there exists a state in the path that
  satisfies ?.

53
Computation Tree Logic

• temporal operator (F,G,X,U)
• F? (? holds sometime in the future) is true of a
  path if there exists a state in the path that
  satisfies ?.
• Example EF(started and not ready) It is
  possible to get to a state where started holds
  but ready does not hold.

54
Computation Tree Logic

• temporal operator (F,G,X,U)
• G? (? holds globally) is true of a path if ?
  holds for all states in the path.
• Example AG(req implies AF ack). It is always the
  case that if the signal req is high then
  eventually ack will also be high.

55
Computation Tree Logic

• temporal operator (F,G,X,U)
• X? (? holds in the next state) means that ? is
  true in the next state.
• ? U? (? holds until ? holds) is satisfied by a
  path if ? is true in some state in the path, and
  in all preceding states, ? holds.
• Example AG(send implies Asend U recv). It is
  always the case that if send occurs, then
  eventually recv is true, and until that time,
  send must remain true.

56
Computation Tree Logic

• Example AG EF restart From any state it is
  possible to get to the restart state.

57
Computation Tree Logic

• Examples Dark circle indicates that a
  specification ? is true in corresponding state.
  Light means false.

AG?
AF?
EG?
58
Computation Tree Logic

• Model to be verified Finite state machine.
  (S,I,R) where S is the set of all possible
  states, I the set of initial states, R a binary
  relation on S which defines the possible
  transitions.
• Can verify systems with more than 10120 states
  (1995).

59
Computation Tree Logic Railway Interlocking
Control

• Simple Interlocking Model

C
Avoid derailments and train crashes
4
B
2
A
5
3
Track sections 2,3,4,5 Control Signals A,B,C
60
Computation Tree Logic Railway Interlocking
Control

• Simple Interlocking Model

Inputs 2T 0 no train in 2 1 2 occupied by
train or broken
C
4
B
2
A
5
3
Track sections 2,3,4,5 Control Signals A,B,C
61
Computation Tree Logic Railway Interlocking
Control

• Simple Interlocking Model

SPEC AG!(SignalA1 and
SignalB1) AG!(SignalA1 and
SignalC1) AG(2T0 implies AX SignalA0)
C
4
B
2
A
5
3
Track sections 2,3,4,5 (0 unoccupied) Control
Signals A,B,C(0red, 1green)
62
Output from checker

• Specification AG(SignalA1 and ) is false as
  demonstrated by the following execution sequence
• state 1.1
• state 1.2
• Gives counterexample if there is one.

63
Computation Tree Logic Implementation BDDs

• Binary Decision Diagrams
• A canonical representation for Boolean formulas
  (canonical in simplest or standard form).
• Invented by Randal Bryant, now at CMU.
• Similar to a binary decision tree, but structure
  is a dag rather than a tree. Allows nodes and
  substructures to be shared.

64
Computation Tree Logic Implementation BDDs

• Binary Decision Diagrams

a b c d result 1 1 1 1 1 1 0 1 1 1 1 0
1 1 1
a
1
What is Boolean formula?
0
b
0
c
1
1
d
0
1
0
All paths to 1
0
1
65
Computation Tree Logic Implementation BDDs

• Binary Decision Diagrams

a
1
Given a variable ordering, the BDD for a formula
is unique. There are efficient algorithms to
compute the BDD for not f and f or g given the
BDD of f and g.
0
b
0
c
1
1
d
0
1
0
0
1
66
Computation Tree Logic Implementation BDDs

• Binary Decision Diagrams

a
1
For the purpose of model checking also need to
compute BDD of restricted formulas. Bryant
describes an algorithm for computing the BDD of a
restricted formula such as f, where v0.
0
b
0
c
1
1
d
0
1
0
0
1
67
Computation Tree Logic Implementation BDDs

• Binary Decision Diagrams All Boolean formulas
  are represented by BDDs. BDDs built in a
  bottom-up manner.
• The set of atomic formulas is precisely the set
  of state variables. (BDD for an atomic variable
  one BDD variable)
• Formulas are built from atomic formulas using
  Boolean connectives. Allows CTL formulas.

68
Symbolic Model Checking

• Determine correctness of finite state systems.
• Specifications are written as formulas in a
  propositional temporal logic.
• Models to be checked are represented by state
  transition graphs
• Verification is accomplished by an efficient
  breadth-first search.

69
Symbolic Model Checking

• View transition system as model of logic.
• Verify whether specifications are satisfied for
  model.
• Advantages
• completely automatic
• provides counterexamples (execution trace which
  shows why formula is not true)
• verify partially specified systems

70
Symbolic Model Checking

• Model checkers achieve great efficiency through
  the use of symbolic implementation techniques
• represent states and transitions through Boolean
  formulas in BDD form

71
Symbolic Model Checking

• Representing the Model
• Labeled state-transition graph M.
• Use BDDs to represent graph and check whether
  formula holds.
• Behavior determined by variables V

72
Symbolic Model Checking

• Representing the Model
• Behavior determined by variables V
• current state
• V Second copy of variables
• next state

73
Symbolic Model Checking

• Representing the Model Relationship between
  variables in the current state and the next
  states is written as a formula using V and V.
  Boolean formula N representing transition
  relation. Covert to BDD.

74
Computation Tree Logic
a
a
b
b
s1
s2
a
b
b
a
b
b
b
a
State transition graph and corresponding
computation tree Paths in tree represent all
possible computations
75
Computation Tree Logic

• Used to express properties that will be verified
• Computation trees are derived from the state
  transition graphs
• State transition graphs unwound into an infinite
  tree rooted at initial state

76
Exercise

• Design a finite state machine with start state s
  and final state t and prove that for all
  transitions from s to t any encounter of state y
  is preceded by encountering first state x.
• Run your model and specification with the model
  checker on the CMU model checking home page.