Testing Strategies CFICSE VV08 October 1999 - PowerPoint PPT Presentation

1 / 34
About This Presentation
Title:

Testing Strategies CFICSE VV08 October 1999

Description:

primarily investigates sequences of k,d,u,-: kill, define (note that declared ... Specifying and Generating Functional Tests', Communications of the ACM 31, 6, ... – PowerPoint PPT presentation

Number of Views:26
Avg rating:3.0/5.0
Slides: 35
Provided by: Shep45
Category:

less

Transcript and Presenter's Notes

Title: Testing Strategies CFICSE VV08 October 1999


1
Testing StrategiesCFICSE VV08 October
1999
  • Dr. Terry Shepard
  • Royal Military College of Canada
  • Electrical and Computer Engineering
  • 613-541-6000 ext. 6031

2
Testing Strategies Topics
  • Possible emphases
  • Exhaustive testing is impossible
  • Why does testing work at all?
  • What do testing strategies depend on?
  • Some testing strategies
  • Are all testing strategies driven by coverage
    considerations?
  • Mode transition code is critical
  • Strategy Ideas from Marick 4
  • Beizers Classification
  • Cause-effect graphing
  • Basis path coverage

3
Three possible emphases in selecting test cases
  • Safety
  • Frequency of Use
  • High Error Rates

4
Only a small part of the state space can be
explored
  • exhaustive testing is impossible
  • Example
  • 1 KB of memory is 21024 8 possible states.
  • Visit 1 state per 10-9 seconds
  • takes 1082.5 seconds, about 1075 years
  • effectively, only about 10-75 of the state space
    can be explored.

5
Why does testing work at all? - Slide 1
  • Hoare 1 defects are bimodal
  • reasonableness and predictability in most
    circumstances
  • has not been expressed in quantitative terms
  • not the same as measures of reliability.
  • number of meaningful and feasible paths is much
    less than the size of the state space
  • most of the state space is unreachable

6
Why does testing work at all? - Slide 2
  • partitioning the state space does work in
    practice, in spite of Hamlets arguments
  • Hamlet 2 and others have argued that random
    values drawn from the whole input space are as
    effective in estimating reliability or at finding
    defects remaining as are random values chosen to
    be typical of subdomains of the input space

7
What do testing strategies depend on? - Slide 1
  • size of program being tested unit/module,
    component, subsystem, system
  • development strategy/process e.g. incremental
    vs. big bang
  • costs of developing stubs, drivers and other
    harnesses.
  • testing purpose debugging, reliability
    measurement, usability, load, stress,
    performance, installation, acceptance,

8
What do testing strategies depend on? - Slide 2
  • new system or old (regression testing is always
    present!)
  • type of system window interface, real-time,
    distributed, ...
  • nature of market beta test of shrink-wrapped
    software, not of military software
  • degree of certainty needed
  • availability of specifications
  • top-down or bottom-up integration

9
What do testing strategies depend on? - Slide 3
  • available budget and schedule
  • point in schedule at which test planning starts
  • contractual arrangements and politics
  • market timing considerations
  • customer tolerance for failures
  • extent of automated testing that is possible

10
Kit 3 proposes 4 levels of testing
  • Full testing - starts with requirements or before
    (200 payback)
  • Partial testing - starts with functional design
    (150 payback)
  • Endgame testing - validation oriented
  • Audit-level testing - audit of plans, procedures,
    products checking after the fact
  • 3 also suggests that 40 to 70 of costs are for
    error detection and defect removal.

11
Some testing strategies - Slide 1
  • identify clues and requirements Marick 4
  • debug remove (severe) errors that cause
    (frequent) crashes
  • find faults on most frequently executed paths
  • test scenarios that are most important to the
    customer/user of the software
  • guerilla attacks try to break it, using any
    means available, often in a competition

12
Some testing strategies - Slide 2
  • meet some coverage target (e.g. 80 statement
    coverage)
  • measure a mix of coverage indicators and apply
    judgement as to which to pay attention to
  • observe/inspect by simulated execution of a
    representation of the design (e.g. ObjecTime)
  • Microsoft tester buddies (is this a strategy?)

13
Choice of testing strategy is partly determined
by the style used to write the code
  • roughly three main divisions
  • pre-structured programming
  • structured code
  • object oriented code

14
  • Aside Beizer 5 adds a twist All data
    declarations should be global, and should be kept
    in a data dictionary, on the basis that over the
    life of a piece of software, data tends to
    migrate toward being more global, so why not
    recognize this tendency right up front and make
    all data global? He thinks the jury is still out
    on the effects of object-oriented programming on
    this phenomenon. (p. 46)

15
Are all testing strategies driven by coverage
considerations?
  • Coverage may be applied to any kind of
    documentation
  • A software project contains many different
    structures
  • Many different forms of traversal can be measured
  • It is common wisdom among testers that choosing
    tests to satisfy some code coverage measure is a
    very poor strategy

16
Mode transition code is critical
  • Example 1 dual mission computers, can operate
    with only one, code that makes the transition is
    rarely used, but is critical
  • Example 2 aircraft spends most of its time on
    the ground or in level flight, but code for
    takeoff and landing is critical
  • Example 3 two modes of operation for a printer
    pool jobs printed on arrival, or jobs queued.
    Transition code between modes is critical.

17
Strategy Ideas from Marick 4 - Slide 1
  • clues lead to test requirements, which lead to
    test specifications
  • sources of clues code, specs, earlier testing
    results, experience
  • e.g. uncovered code may lead to new clues
  • most errors are not creative checklists work
    well
  • faults of omission are harder to find, and more
    important

18
Strategy Ideas from Marick 4 - Slide 2
  • a single test specification usually satisfies
    multiple test requirements
  • unit testing is too expensive
  • Marick expresses controllability and
    observability in terms of
  • reachability
  • necessity (fault must produce a different result
    from the correct program)
  • propagation

19
Strategy Ideas from Marick 4 - Slide 3
  • These are ideal fault conditions
  • must be approximated, since fault locations are
    not known
  • Marick has built a Generalized Coverage Tool
    (GCT)
  • from cs.uiuc.edu or from Testing Foundations
  • Test Requirement Catalog 4 Appendix B
  • really a black box testing checklist

20
A Classification of Black and White Box Testing
techniques
  • From 5 Software Testing Techniques, 2nd Edn,
    Van Nostrand Reinhold, 1990, by Boris Beizer)
  • Called strategies by Beizer - to some extent,
    they are.

21
Beizers Classification
  • Flowgraphs and Path testing
  • Transaction-flow testing
  • Data-flow testing
  • Domain testing
  • also called Partition or Equivalence testing
  • includes boundary value analysis
  • Syntax testing
  • Logic-based testing
  • States, state graphs and Transition testing
  • Mutation Testing

22
Flowgraphs and Path testing
  • includes block/statement (C1) and branch coverage
    (C2)
  • hard to achieve 100
  • may be worth testing parts beyond 100

23
Flowgraph and Path testing Strategies
  • keep paths short and simple,
  • make small incremental changes to paths
  • start with loop free paths
  • include paths that test loop limits
  • then decide how extensive to make loop testing
  • may help to artificially lower loop upper limits

24
Transaction-flow testing
  • equivalent to path testing, but at the level of
    interfaces among software components rather than
    interfaces among individual statements.
  • might also be called scenario based or use-case
    based testing.
  • If scenarios or use cases were part of the
    requirements, they can be used to generate tests

25
Data-flow testing
  • primarily investigates sequences of k,d,u,-
    kill, define (note that declared and initialized
    are not distinguished), use, no mention. For
    example, -u is anomalous, in the sense that the
    variable may have been defined globally, but if
    it has not been, this is an error.

26
Domain testing
  • partition the input space, and test using
    representative values from each partition
  • Hamlet 2 questions its effectiveness

27
Syntax testing
  • hidden languages often exist for describing input
    and output. Testing is driven by explicitly
    defining the syntax (e.g. in BNF), and trying to
    validate (and break) instances of the hidden
    language. If the language is not hidden, so that
    a clear syntax exists, syntax testing is less
    productive.

28
Logic-based testing
  • Uses decision tables, path expressions
  • Addresses the issue of how to combine a variety
    of different conditions in an orderly fashion to
    create multiple test cases

29
States, state graphs and Transition testing
  • Some aspects of a design may be represented as
    finite state machine diagrams
  • Testing these parts of a design can be done by
    undertaking a transition tour of the state
    diagram (state graph)
  • This is not useful for state represented in the
    form of integers, floats, arrays, records, etc.

30
Mutation Testing
  • generate mutations of program under test
    (preferably automatically), run tests on all
    mutations (expensive), examine all cases where
    mutants and original program give the same result
    - either add tests so they give different
    results, or conclude that the original program is
    in error.

31
Cause-effect graphing
  • Idea from the 70s
  • not practical
  • By transforming a written specification into a
    set of cause-effect graphs, the tester is
    replacing one complex representation with
    another 6

32
Basis path coverage
  • Select test cases that follow the control flow
    paths on which the McCabe complexity measure
  • McCabes conjecture was that such test cases
    would provide adequate coverage
  • Beizer 5 disagrees

33
References 1
  • 1 C.A.R. Hoare, How did software get so
    reliable without proof?, Formal Methods Europe
    96, Keynote speech
  • 2 D. Hamlet and R. Taylor, Partition Testing
    does not Inspire Confidence, IEEE Transactions
    on Software Engineering, Dec. 1990, pp. 1402-1411
  • 3 Edward Kit, Software Testing in the Real
    World Improving the Process, Addison Wesley, 1995

34
References 2
  • 4 Brian Marick, The Craft of Software Testing,
    Prentice Hall, 1995
  • 5 Boris Beizer, Software Testing Techniques,
    2nd Edn, Van Nostrand Reinhold, 1990
  • 6 T.J. Ostrand and M.J. Balcer, The
    Category-Partition Method for Specifying and
    Generating Functional Tests, Communications of
    the ACM 31, 6, June 1988, pp. 676-686
Write a Comment
User Comments (0)
About PowerShow.com