Test Plan: Introduction

1
Test Plan Introduction

• Primary focus developer testing
• Implementation phase
• Release testing
• Maintenance and enhancement
• Secondary focus formal system verification
• Addressed within current test plan via examples
• Assumed to be primarily independent

2
Tests to be Performed

• Bottom-up integration testing
• Build a module, build a test to simulate how it
  is used
• Black-box
• Based on specification and educated guess
  stress tests
• White-box
• Based on code inspection
• Platform testing
• Establish baseline capability of hardware/OS,
  detect differences
• Performance
• Per module, and peer to peer distributed costs
• Test coverage
• Coverage level dependent on resources, time, and
  cost of failure

3
Allocation of Testing Responsibilities

• Assumption development team performs
• Internal (module-level) performance
• Sample system performance (limited example
  federation)
• Full white-box
• Limited black-box (basic system functionality)
• Assumption external group performs
• Independent, detailed system verification (Spec
  compliant)
• Standardized performance tests
• Test coverage
• Coverage will be measured, analyzed for relative
  effectiveness
• Coverage levels TBD

4
Testing Philosophy

• Testing is not just pre-release! Continual
  process within development phase
• Catch defects as early as possible
• Make sure defects stay fixed
• Track cause of defects repair problem, do not
  keep re-patching the tire
• Need support at both design level and
  implementation level to accomplish these goals

5
Continual Testing Process

• Tests created during development
• Central code repository modules, test suites
  tied together
• Tests are treated as live code to be maintained,
  not once-offs
• Test documentation how to run, what is tested,
  and why
• Revision control on both modules and tests
• Modular development
• System broken down into hierarchies of testable
  components
• Automated, incremental integration testing
• As code is developed, it is tested standalone,
  then incrementally within the confines of the
  system
• Continual feedback into development, maintenance
  cycle
• Weekly postings of performance results, current
  defect status

6
Design Support

• Standard testing and debugging methods on each
  module / class (peek, dump, exercise,
  performance)
• Self-checking code (pre and post parameter
  asserts, valid state calls)
• Debug levels, controlled via runtime flags
• Centralized logging mechanisms to collect
  distributed traces
• Logs used in both developer testing and sample
  user traces from the field

7
Development Tool Support

• Shadow development trees for automated and
  individual testing (ClearCase views)
• Common testing tools, testing approach to
  simplify testing process, and to allow automated
  testing
• Examples
• Standard test harnesses, method of invocation
• Standard testing directories, makefile commands
  per module
• Standard set of test-record-evaluate tools
• Central I/O mechanisms to collect distributed
  test results
• Standard system-level drivers
• Sequential test harnesses and emulated
  distributed harnesses to emulate determinism
  during development

8
Levels of Testing per Module

• Basic used in initial development, later in
  porting
• Minimal level of functionality to establish
  module operation
• Simplicity is critical during development,
  porting
• Detailed used to verify module correctness
  before checking into current baseline
• Does module meet interface contract
• Tests for common coding errors
• Regression replicates previous use which caused
  failure
• Used to verify defect has been corrected
• Used to ensure defect does not re-occur over time
• Performance tracks the CPU usage per module and
  peer to peer performance across distributed
  components

9
Complex Functional Areas Testing Examples

• Memory continual test for memory leaks and
  over-writes
• Automated use of Purify within weekly test suites
  across all modules, all levels of the system
• Threads platform-specific performance and
  implementation variances must be established per
  platform
• Standard set of tests which mimic system use of
  threads
• Causality complex areas of the system (such as
  zero-lookahead ordering) are difficult to
  establish correctness across all use cases
  (dropped packets, simultaneous time stamps with
  variable arrival times, cancelled events, )
• Detailed test scripts, executed in deterministic
  test harnesses with varying error conditions

10
Periodic Testing Activities

• Code walkthroughs encompass both system code and
  associated test suites. Testing focus
• Do the test suites sufficiently stress the module
• Do the test suites still accurately represent the
  expected use of the module
• Has the underlying platform changed, and has
  performance changed accordingly
• Change Control Board formal tracking process and
  tools to establish, record and monitor status of
  functional change requests, defect priority and
  status data