CIS-74 Computer Software Quality Assurance

1
CIS-74 Computer Software Quality Assurance

• Systematic Software Testing
• Chapter 3
• Master Test Planning

2
Test Plan Hierarchy

• Master Test Plan (MTP)
• Unit Test Plan
• Acceptance Test Plan
• Integration Test Plan
• System Test Plan
• (These last four are covered in Chapter 4.)

3
MTP

• The process is more important than the actual
  document.
• As the complexity of a testing activity
  increases, the criticality of a good MTP
  increases exponentially (see diagram on p. 57).

4
IEEE Test Plan Template

• 1. Test Plan Identifier
• Unique
• Generated by company
• Identifies
• Version of test plan
• Level of test plan
• Version of software covered

5
IEEE Test Plan Template

• 2. Table of Contents
• Added by authors of text
• Two or more levels preferable

6
IEEE Test Plan Template

• 3. References
• Included in Introduction of IEEE template
• Given own section by texts authors
• Recommended
• Project authorization
• Project plan
• QA plan
• Configuration management plan
• Relevant policies
• Relevant standards

7
IEEE Test Plan Template

• Glossary
• Added by authors of text

8
IEEE Test Plan Template

• Introduction
• Scope of project - key features, history, etc.
• Scope of test plan (levels covered and not
  covered).

9
IEEE Test Plan Template

• 6. Test Items
• Programmatic description of what is to be tested
• High-level test plan application or version
• Low-level test plan program, unit, module, or
  build
• Items to be excluded from testing also

10
IEEE Test Plan Template

• Software Risk Issues
• List of risks from formal risk analysis (covered
  in Chapter 2).

11
IEEE Test Plan Template

• 8. Features to be Tested
• From customer point of view
• Basically, those items above the cut line from
  formal risk analysis

12
IEEE Test Plan Template

• 9. Features Not to Be Tested
• From customer point of view
• Basically, those items below the cut line from
  formal risk analysis
• Amusing point on pp. 68-69 Managers who raise
  their eyebrows about this section of a test plan
  are often the ones who approve schedules that
  dont allow enough time to test everything.

13
IEEE Test Plan Template

• 10. Approach (Strategy)
• Heart of test plan
• Should explain approach for each level
• Should specify entrance and exit criteria for
  each level to another

14
IEEE Test Plan Template

• 10. Approach (continued)
• Test environment is most artificial for unit
  testing, followed by system testing, and then
  integration testing.
• Testing environment is closest to reality for
  acceptance testing.

15
IEEE Test Plan Template

• 10. Approach (continued)
• Methodology decisions
• What testing levels will be used?
• When will testers become involved?
• Will there be a beta?
• Will there be an alpha?
• What testing techniques will be used?
• Etc.

16
IEEE Test Plan Template

• 10. Approach (continued)
• Resource decisions
• Is dedicated test group sufficient?
• If not, where else can resources be obtained?
  Developers? Users? Interns? U. S. contractors?
  Overseas contractors? Customer support?

17
IEEE Test Plan Template

• 10. Approach (continued)
• Test Coverage Decisions
• Code coverage measures the percentage of program
  statements that are executed by a set of test
  cases.
• Requirements coverage measures of business
  requirements covered.
• Design coverage measures the of design covered.
• Interface coverage measures of interfaces
  covered.

18
IEEE Test Plan Template

• 10. Approach (continued)
• Walkthroughs and Inspections
• Reviews are another form of software evaluation
  besides testing.
• Walkthrough - talking/walking through each line
  of code as a group
• Inspection - more formal, may be done by a single
  individual

19
IEEE Test Plan Template

• 10. Approach (continued)
• Configuration Management
• Can be done in a separate document
• In MTP, should cover change management and
  bug-review process

20
IEEE Test Plan Template

• 10. Approach (continued)
• Collection and Validation of Metrics
• Metrics can be a significant overhead, so its
  necessary to plan exactly which are needed.
• More info coming in Chapter 10 - The Test
  Manager.

21
IEEE Test Plan Template

• 10. Approach (continued)
• Tools and Automation
• Caution! It can take more time to automate than
  the time it takes to execute tests manually! Be
  sure automated tests will be re-run often to
  justify the cost of automation.

22
IEEE Test Plan Template

• 10. Approach (continued)
• Changes to the Test Plan
• What type of changes require redoing the approval
  process?
• How should the test plan be published?
• How should the review be conducted?
• Should the test plan go through regular CM?

23
IEEE Test Plan Template

• 10. Approach (continued)
• Meetings and Communications
• Standard meetings
• Status reporting
• Metrics

24
IEEE Test Plan Template

• Item Pass/Fail Criteria
• Pertains to items from Section 6.0 - Test Items -
  NOT to test cases
• Examples
• Test cases passed and failed
• Number, type, severity, and location of bugs
• Usability
• Reliability

25
IEEE Test Plan Template

• Suspension Criteria Resumption Requirements
• Large volumes of bugs
• Critical bugs
• Incomplete tasks
• Code not checked into source-code control system

26
IEEE Test Plan Template

• Test Deliverables
• All testware to be created and maintained for the
  testing effort.

27
IEEE Test Plan Template

• Testing Tasks
• Authors recommend deleting this section, and
  listing all testing tasks in Section 16 -
  Responsibilities.

28
IEEE Test Plan Template

• Environmental Needs
• Hardware
• Software
• Documents
• Security access

29
IEEE Test Plan Template

• Responsibilities
• Authors suggest a matrix
• Rows are labeled with testers names
• Columns are labeled with major testing
  responsibilities

30
IEEE Test Plan Template

• Staffing and Training Needs
• Vary widely, depending on the project
• Examples
• How to use a particular testing tool
• How to use the organizations defect-tracking
  system
• How to use the orgs configuration management
  system

31
IEEE Test Plan Template

• Schedule
• Should be based on schedule in Project Plan PLUS
  testing milestones

32
IEEE Test Plan Template

• Planning Risks and Contingencies
• List of planning risks and contingencies from
  formal risk analysis (covered in Chapter 2).

33
IEEE Test Plan Template

• Approvals
• Approver(s) should be person(s) who will
  eventually have to approve the software being
  ready to move to next level.
• MTP may require several approvers.
• Buy-in and commitment is whats needed, not just
  signature.

34
Review Questions
35
What are the four levels of test plans specified
in the IEEE Std. 829-1998 Standard for Software
Test Documentation?
36
What are the four levels of test plans specified
in the IEEE Std. 829-1998 Standard for Software
Test Documentation? Unit Integration System Acce
ptance
37
When should work start on the MTP?
38
When should work start on the MTP? At the same
time as the requirements specifications and the
project plan are being developed. (p. 58)
39
What does TBD stand for?
40
What does TBD stand for? To Be Determined
41
What is the first section of the IEEE Template
for Test Planning?
42
What is the first section of the IEEE Template
for Test Planning? Test Plan Identifier
43
What three pieces of info should be contained in
the unique company-generated test plan
identifier?
44
What three pieces of info should be contained in
the unique company-generated test plan
identifier? Version of test plan Level of test
plan (acceptance, integration, unit, or
system) Version of software covered by the test
plan
45
What does ISO stand for?
46
What does ISO stand for? International
Standards Organization
47
What does CMM stand for?
48
What does CMM stand for? Capability Maturity
Model
49
What three sections do the texts authors
recommend be added near the beginning of the IEEE
Template for Test Planning?
50
What three sections do the texts authors
recommend be added near the beginning of the IEEE
Template for Test Planning? Table of
Contents References Glossary
51
What are examples of References recommended by
the IEEE standard?
52
What are examples of References recommended by
the IEEE standard? Project Authorization Project
Plan QA Plan Configuration Management
Plan Relevant Policies Relevant Standards
53
What are the two main parts of the IEEE test plan
templates Introduction section?
54
What are the two main parts of the IEEE test plan
templates Introduction section? Scope of the
project (features) Scope of the test plan (levels
of testing)
55
What is covered in the 6.0 Test Items section of
the IEEE template for test plans?
56
What is covered in the 6.0 Test Items section of
the IEEE template for test plans? Test items,
I.e., versions of products, programs, modules,
units, etc. that will be tested, AND those that
will NOT be tested.
57
What is the difference between the contents of
6.0 - Test Items and 8.0 - Features to be
Tested?
58
What is the difference between the contents of
6.0 - Test Items and 8.0 - Features to be
Tested? Test Items covers what to test from the
point of view of the developer or build manager,
whereas Features to be Tested covers what to test
from the point of view of the user/customer.

59
Which section of a MTP should specify entrance
and exit criteria from one level to
another?
60
Which section of a MTP should specify entrance
and exit criteria from one level to another? 10
- Approach (Strategy)
61
Which of the four levels of testing should have a
test environment which mirrors the production
environment as closely as possible?
62
Which of the four levels of testing should have a
test environment which mirrors the production
environment as closely as possible? Acceptance
testing
63
Which of the four levels of testing should be
done first, and who normally does
it?
64
Which of the four levels of testing should be
done first, and who normally does it? Unit
testing, and individual developers.
65
Which of the four levels of testing should be
done second, and who normally does
it?
66
Which of the four levels of testing should be
done second, and who normally does
it? Integration testing, and developers.

67
Which of the four levels of testing should be
done third, and who normally does
it?
68
Which of the four levels of testing should be
done third, and who normally does it? System
testing, and the test team.
69
What approach to creating a testing methodology
was cited by the authors in a case
study?
70
What approach to creating a testing methodology
was cited by the authors in a case
study? Choose a pilot project, create a MTP for
it, and then declare that projects process as
Version 1.0 of the organizations testing
methodology.
71
What two types of events can sabotage test
plans?
72
What two types of events can sabotage test
plans? Development is running late, so wont
provide the testing team the software on
schedule. The ship date has been moved
forward.
73
What is the name for the measurement of the
percentage of program statements that are
executed by a group of test cases?
74
What is the name for the measurement of the
percentage of program statements that are
executed by a group of test cases? Code
coverage
75
What is a secondary benefit of doing code
coverage?
76
What is a secondary benefit of doing code
coverage? Eliminating code that can no longer
be executed because there is no logical path to
reach it.
77
What is the name for such program code that
cannot be executed?
78
What is the name for such program code that
cannot be executed? Dead code.
79
What are three possible explanations for why code
coverage is not done more often?
80
What are three possible explanations for why code
coverage is not done more often? --Purchase of,
and training on, a new tool. --Some functional
level testers arent aware of the concept. --Code
coverage is a moot point for organizations so
resource-constrained that entire programs are not
addressed by tests.
81
What are three other types of coverage
measurements?
82
What are three other types of coverage
measurements? --Requirements coverage --Design
coverage --Interface coverage
83
What is another type of software evaluation
besides testing?
84
What is another type of software evaluation
besides testing? Reviews (of requirements,
design, code)
85
What are the two most common types of
reviews?
86
What are the two most common types of
reviews? Walkthroughs and inspections
87
Which is more rigorous--walkthroughs or
inspections?
88
Which is more rigorous--walkthroughs or
inspections? Inspections
89
What is the name for retesting previously tested
features to ensure that a change or bug fix has
not introduced new problems?
90
What is the name for retesting previously tested
features to ensure that a change or bug fix has
not introduced new problems? Regression
testing
91
What is the name for rerunning tests that
revealed a bug to ensure that the bug was fully
and actually fixed?
92
What is the name for rerunning tests that
revealed a bug to ensure that the bug was fully
and actually fixed? Confirmation
testing
93
What Bugzilla states might a bug get moved to as
a result of confirmation testing?
94
What Bugzilla states might a bug get moved to as
a result of confirmation testing? VERIFIED or
REOPENED
95
What does CCB stand for?
96
What does CCB stand for? Change Control
Board
97
What does the CCB do? Evaluate the severity of
bugs, the cost to fix and test them, and the
priority for fixing them.
98
What is a trade-off to consider with respect to
automation tools?
99
What is a trade-off to consider with respect to
automation tools? Automated tests take more
time to implement than would be required by
manual execution. However, if the same tests are
executed several times (for regression testing
for example), then the automated tests can save
time in the long run.
100
What items are covered in section 11 - Pass/Fail
Criteria?
101
What items are covered in section 11 - Pass/Fail
Criteria? Those test items described in section
6 - Test Items.
102
What are examples of typical pass/fail
criteria?
103
What are examples of typical pass/fail
criteria? Number of test cases passed and
failed Number, type, severity and location of
bugs Usability Reliability and/or
stability
104
What type of chart can be used to show
dependencies between testing activities?

105
What type of chart can be used to show
dependencies between testing activities? Gantt
charts.
106
End of Chapter 3