CIS-74 Computer Software Quality Assurance

1
CIS-74 Computer Software Quality Assurance

• Systematic Software Testing
• Chapter 5
• Analysis and Design

2
Creating an Inventory

• 1. Gather reference materials
• 2. Form a brainstorming team
• 3. Determine test objectives
• 4. Prioritize objectives
• ltcontinued on next slidegt

3
Creating an Inventory

• 5. Parse (organize) objectives into lists.
• Create an inventory tracking matrix
• Left-hand column of objectives should be in
  prioritized order from step 4.
• This step known as requirements traceability.
• ltcontinued on next slidegt

4
Creating an Inventory

• 7. Identify tests for unaddressed conditions,
  adding/modifying test cases as needed.
• Evaluate each inventory item for adequacy. (And
  add/modify test cases as needed.)
• Maintain the testing matrix.

5
Black-Box vs. White-Box

• Black-box testing - testing done without
  knowledge of the internal workings of the
  program/system being tested.
• White-box testing - testing done with knowledge
  of the internal workings of the program/system
  being tested.
• White-box testing is also called structural
  testing because its based on structure of the
  code.

6
Black-Box Science Equivalence Partitioning

• An equivalence class is formed by a group of
  tests which
• All test the same thing.
• If one catches a bug, the others do too.
• If one doesnt catch a bug, the others wont
  either.

7
Black-Box Science Boundary Value Analysis

• Test for
• Exact boundaries
• Value immediately above upper boundary
• Value immediately below lower boundary

8
Black-Box Science State-Transition Diagrams

• A state machine is a thing which depends not just
  on current input, but also on past input, for
  determining output.
• A transition occurs when a state is changed by
  its input from one state into another.
• To use in black-box testing
• Create a state-transition diagram.
• Create a test case for every arrow (transition)
  from the diagram.

9
Black-Box Science Orthogonal Arrays

• An Lx(Yz) orthogonal array is a two-dimensional
  array with
• X rows
• Z columns
• 1..y in cells of matrix

10
Black-Box Science Orthogonal Arrays

• Useful with compatibility testing in particular
• What text did not say
• How to create an orthogonal array
• How to use an orthogonal array when the variables
  do not all have the same number of values (as did
  the example in Table 5-9)

11
Black-Box Art Ad Hoc Testing

• Often based on systematic techniques
• Test cases should be created for all bugs found
• Can locate an entire class of tests missed using
  more formal means
• Fun!

12
Black-Box Art Random Testing

• All data fields generated randomly, often with a
  tool
• Can lead to too many negative tests - those with
  an invalid input, and expected results of an
  error message
• Not very useful
• At best, useful for crash-proofing

13
Black-Box Art Semi-Random Testing

• Random combinations of legitimate values for
  variables
• Gives a little added confidence to systematic
  testing

14
Black-Box Art Exploratory Testing

• Test design and execution occur concurrently
• Test results may prompt user to delve deeper into
  that area
• Neither ad hoc nor random in nature
• Productive areas of test are immediately expanded
• Viewed as complementary to structured testing by
  authors

15
White-Box Science White-Box Testing

• White-box should be clear-box or glass-box
  in authors views. (Why?)
• Normally viewed as a technique for unit testing,
  but can also be used with integration system
  testing
• Primarily about path testing (paths through the
  code)

16
White-Box Science Coverage Techniques

• Statement coverage - of lines of code executed
  by the tests
• Decision or branch coverage - of decisions
  executed by the tests
• Path coverage - of paths executed by the tests

17
Test Design DocumentationIEEE Test Design
Specification

• One TDS per feature from the Features to be
  Tested from the corresponding test plan
• Test identification section includes test case
  ids and high-level description for each test
  case but not info on how to execute them.

18
Test Design DocumentationIEEE Test Design
Specification

• Test cases may not be sequential because TDS
  covers all tests necessary to test one
  feature--some tests for other features may be
  reusable.
• Feature pass/fail criteria establish success or
  failure for the TDSs feature--not individual
  test cases.

19
Test Design DocumentationTest Case Specification

• Two approaches
• IEEE Template for Test Case Specification
• Focuses on the what
• Spreadsheet
• Very common method
• Can also be used to record results

20
Test Design DocumentationIEEE Test Procedure
Specification

• Focuses on how tests will be run
• Can be run via scripts--automated test
  procedures. These scripts are normally written
  in a scripting language such as Perl or a
  UNIX/Linux shell language.
• Scripts also refer to testing instructions for
  manually-executed tests.

21
Review Questions
22
Which test design technique was covered in depth
by the first part of Chapter 5?
23
Which test design technique was covered in depth
by the first part of Chapter 5? Inventories
24
What are the column headers in an inventory
tracking matrix?
25
What are the column headers in an inventory
tracking matrix? Test case identifiers.
26
How are the objectives/inventory items in the
column of an inventory tracking matrix
ordered?
27
How are the objectives/inventory items in the
column of an inventory tracking matrix
ordered? Priority order (highest to lowest)
28
With respect to an inventory tracking matrix,
what is the name for the process of adding Xs
to those non-header cells which represent test
case coverage of an objective/inventory-item?

29
With respect to an inventory tracking matrix,
what is the name for the process of adding Xs
to those non-header cells which represent test
case coverage of an objective/inventory-item? Re
quirements traceability
30
What is the name for testing based upon the
requirements, i.e., the internal workings of the
system under test are unknown?
31
What is the name for testing based upon the
requirements, i.e., the internal workings of the
system under test are unknown? Black-box
testing
32
What is another name for black-box testing?
33
What is another name for black-box
testing? Behavioral testing.
34
What is white-box testing?
35
What is white-box testing? Testing in which the
internal workings of the system under test are
known to the tester.
36
What is another name for white-box testing?
37
What is another name for white-box
testing? Structural testing. (Why?) Glass-box
testing Clear-box testing Translucent-box
testing
38
What is coincidental correctness?
39
What is coincidental correctness? Right results
for wrong reason. (A test case passes even
though the input was processed incorrectly.)
40
What are the three defining aspects of a group of
tests making up an equivalence class?
41

• What are the three defining aspects of a group of
  tests making up an equivalence class?
• They all test the same thing
• If one catches a bug, the others will too.
• If one fails to catch a bug, the others will also
  fail.

42
What are the three areas to test when using
boundary analysis?
43

• What are the three areas to test when using
  boundary analysis?
• The exact boundaries
• The value immediately above the upper boundary
• The value immediately below the lower boundary

44
Which black-box science technique involves
creating a table which lists all possible inputs
and outputs?
45
Which black-box science technique involves
creating a table which lists all possible inputs
and outputs? Decision tables
46
In a state-transition diagram, how are the states
and transitions depicted?
47
In a state-transition diagram, how are the states
and transitions depicted? State - circle with
label inside Transition - directed arc between
two states
48
What is an orthogonal array?
49
What is an orthogonal array? A two-dimensional
array of integers Any two columns contain all
combinations of the integers contained in the
array
50
What is meant by L9(34)?
51
What is meant by L9(34)? An orthogonal array
with 9 rows, 4 columns, and cell contents ranging
from 1 through 3, inclusively.
52
What is an example of a pair-wise defect?
53
What is an example of a pair-wise defect? A web
site that does not work properly with a
particular browser and OS even though it works on
many other browser/OS combinations. Example Bug
7019
54
What kind of testing involves generating
miscellaneous data, often using a tool?
55
What kind of testing involves generating
miscellaneous data, often using a tool? Random
testing
56
What is the name for a test in which the tester
supplies an invalid input in the expectation of
receiving an error message?
57
What is the name for a test in which the tester
supplies an invalid input in the expectation of
receiving an error message? Negative test
58
What is the name for testing in which the test
design and test execution are conducted
concurrently?
59
What is the name for testing in which the test
design and test execution are conducted
concurrently? Exploratory testing
60
How do the authors view exploratory testing vs.
structured testing?
61
How do the authors view exploratory testing vs.
structured testing? As a complementary
process
62
What names do the authors claim are better than
white-box testing?
63
What names do the authors claim are better than
white-box testing? Clear-box testing or
glass-box testing.
64
What metric describes the complexity of a
software module?
65
What metric describes the complexity of a
software module? Cyclomatic Complexity
66
End of Chapter 5