MIS 515 Systems Design

1
MIS 515 Systems Design
What is Software Testing? And Why Is It So Hard?
James A. Whittaker Florida Institute of
Technology IEEE SOFTWARE, Jan/Feb 2000

• Yan Dow
• David White

2
Introduction

• Software testing is arguably the least understood
  part of the development process.
• Eliminating bugs is tricky and testing is a
  constant trade-off.
• Software testing is considered in the context of
  software development.
• Understanding of roles and interaction of
  software testers and developers.

3
Causes of Software Bugs

• Untested code- time constraints
• Order statements are executed- actual use may
  differ from that during testing order can
  determine whether software works or fails
• Untested input values- combinations simply too
  numerous for testers to apply them all
• Untested operating environment- replication of
  users combination of hardware, peripherals,
  operating system, and applications in testing lab

4
Four phases in testing

• Modeling the softwares environment
• Selecting test scenarios
• Running and evaluating test scenarios
• Measuring testing progress

5
I. Modeling Environment

• Tester tasks
• Simulate interaction between software and its
  environment.
• Identify and simulate the interfaces that a
  software system uses.

6
Common Interfaces to be Tested

• Human interfaces- GUI, command line interface,
  and menu-driven interfaces all still in use
• Software interfaces- application programming
  interfaces (APIs) how software uses an operating
  system, database, or runtime library
• File system interfaces- exist whenever software
  reads or writes data to external files
• Communication interfaces- allow direct access to
  physical devices (such as device drivers,
  controllers, and other embedded systems) and
  require a communication protocol

7
Example Interface Tests

• Using the operating system, one user deletes a
  file that another user has open. Situation the SW
  tries to access that file next time?
• A device gets rebooted in the middle of a stream
  of communication. Will SW react properly or just
  hang?
• Two SW systems compete for duplicate services
  from an API. Will the API correctly service both?

8
Authors Considerations

• Carefully select values for variable input, e.g.
  boundary value partitioning
• Consider entire cross product of value
  combinations
• Sequence inputs, e.g. a graph or state diagram

9
II. Selecting Test Scenarios

• Strive for coverage covering code
  statements(execution paths) and inputs(input
  sequences)
• Test data adequacy criteria adequately and
  economically represent any of the infinite sets
• Typical use scenarios things occur most often

10
Execution Path Test Criteria

• Select a set of tests that cause
• Each source statement to be executed at least
  once
• Each branching structure to be evaluated with
  each of its possible values
• Each data structure to be initialized and then
  used
• Expose each of the seeded faults

11
Input Domain Test Criteria

• Select a set of tests that
• Contain each physical input
• Each interface control to be stimulated
• Have the same statistical properties as the
  entire input domain
• Select a set of paths that are likely to be
  executed by a typical user

12
III. Running Evaluating Test Scenarios

• Automate test scenarios
• Scenario evaluation
• 1.Formalism--with a spec
• 2.Embedded test code
• i)expose certain internal data objects/states
• ii)self-testing programs

13
Regression Testing

• How much retesting of version n is needed using
  the tests that were run against version n-1?
• Tests can temporarily alter the test data
  adequacy criteria

14
Is it really fixed?

• Any specific fix can
• (a) fix only the problem that was reported
• (b) fail to fix the problem
• (c) fix the problem but break something that was
  previously working or
• (d) fail to fix the problem and break something
  else.

15
Authors Suggestions

• To save resources, testers should work closely
  with developers to prioritize and minimize
  regression tests.
• Developers should write code with testing in
  mind.
• If code will be hard to test and verify, then it
  should be rewritten to make it more testable.
• Listen to testers when they tell you about the
  quality of the software ignoring them might be
  the most expensive mistake.

16
IV. Measuring Testing Progress

• Structural completeness
• Functional completeness
• Testability
• Reliability modelspredict future patterns based
  on past data

17
Conclusion

• Lots of unresolved issues need to be solved.
• First thing to be done is to recognize the
  complex nature of testing and take it seriously
• The time to tie academic research to real
  industry products is now

18
Quote Recollect

• Adding quality at the end is an illusion, better
  design is more effective than testing, much
  cheaper

19
Discussion

• Test strategies When to use black-box(functional)
  testing? When to use white-box(structural)
  testing?Why?
• Who should do white-box testing? Development team
  or test team? Why?
• When shall testing start? What are the trade-offs
  of delay testing?

20
Bonus Slide
Source www.extremeprogramming.org