PROCESS

1
PROCESS The GLUE that holds Software Projects
TogetherA Focus on Software Testing
Dennis A. Morrison Senior Principal
Engineer DD(X) System Architecture Dennis_morrison
_at_raytheon.com
2
Presentation Outline

• Software Process Overview
• Software Testing Overview
• Source Code Control
• Trouble Reporting and Bug tracking
• Summary

3
Software Phases(Waterfall Model)
Requirements
4
Software Phases(Process Glue)
Requirements Review Are the Requirements
testable Artifacts Review documentation
Requirements placed under documentation control
Requirements
Design Review Does the design support
requirements? Artifacts Review documentation
Design
Code Review Does the code support requirements
and implement the design?? Artifacts Review
documentation Code under source code control
Code
Test Review Test plans Test Results Did the
tests pass? Artifacts Review documentation
Test plans System Trouble Reports (STR)
Test
Deploy
Test Review Customer Acceptance Test Test
Results Did the tests pass? Artifacts Review
documentation STRs
Support
Customer Survey Is the system meeting customer
expectations Artifacts - Software Updates STRs
5
Software Phases and Loops
Requirements
Design
Code
Test
Phases where most rework occurs
Deploy
Support
New feature requests or feature refinement
6
Software TestingAn Art or a Science ?

• Testing is a phase of software development that
  can become, or seem, endless
• Classically, the most expensive part of the
  project
• Schedule overruns are often attributed to the
  testing cycle
• Testers are usually squeezed into smaller and
  smaller test cycles
• How much testing is enough?
• How do you know when you are finished testing?
• How good is the product when you are finished?

7
Testing Has Many Stages
Test
8

• Phase Unit Test
• Testing is focused on internal operations of the
  software component
• Inputs/outputs
• Logical operations
• Limit testing
• Limited functional testing with stubs
• Testing methodologies
• Formal testing with test plan and review
• Ad-hoc random
• Typically, tests are manually executed

9

• Phase Integration Test
• Testing is focused on functional aspects of an
  integrated set of software components
• Inputs/outputs
• Logical operations from outside of the software
  components
• Limit testing (induced error paths from external
  stimuli)
• Stubs are replaced by real software as they
  become available
• Testing methodologies
• Formal testing with test plan and review
• Semi-automated and manually executed
• Automation yields regression test capability

10

• Phase System Test
• Testing is focused on system level requirement
  verification
• User Interface (Human to Computer Interface)
• Requirement verification
• Limit and normal operational testing
• Stress and longevity testing
• Tests are executed on target hardware and
  software
• Testing methodologies
• Formal testing with test plan and review
• Typically, tests are semi automated and manually
  executed
• Automation yields regression test capability

11

• Phase Acceptance Test
• Testing is focused on externally visible features
• User Interface (Human to Computer Interface)
• Operational focus
• Interfaces with external systems in customer
  domain
• Other collaborative systems
• Customer network, etc.
• Tests are executed on target hardware and
  software usually at the customers site
• Testing methodologies
• Formal testing with test plan and review
• Typically, tests are manually executed

12
Some Software Testing Models

• Developer level testing
• Applicable to small teams
• Community testing
• Beta versions of a product are released to a
  volunteer community for functionality and user
  level testing
• Used by gaming community prior to product
  releases
• Microsoft uses this approach as one of their test
  phases
• Collaborative testing
• Developers and customers test together
• Project level testing
• Used by most large projects

13
When Should Software Testing Begin?

• The software test cycle should begin during the
  project requirements phase
• Requirements review should result in the first
  level of test specifications
• Similar to the Agile software development
  process, test cases should be written and
  reviewed before the software is developed

14
When is Testing Finished?

• When the schedule says youre finished?
• When you run out of money?
• When the boss says to release it?
• When the testers say youre done?
• When the customer says youre done?
• When the law suits are filed?

15
Requirements Verification

• Raytheon uses requirement verification and
  statistical mechanisms to determine when testing
  is complete
• Requirements are decomposed and allocated to
  software components
• Requirements are sold off with during formal
  testing events with stakeholders (customers)
  present

16
Requirement Decomposition
Parent requirement decomposes into three child
requirements
Operational Requirement
The system shall provide three factor user
authentication
Req 1
17
Requirements Role in Testing

• Software is tested against requirements
• Before a parent requirement can be sold off,
  all children requirements must be fulfilled
• Before a child requirement can be sold off, all
  of its siblings must themselves be sold off
• The failure of any requirements will prevent the
  parent from being fulfilled

18
Requirement Decomposition
Parent requirement decomposes into three child
requirements
Operational Requirement
The system shall provide three factor user
authentication
Req 1
Req 1
19
Requirements Traceability

• Software test cases are built from system use
  cases
• Scenarios (rainy day and sunny day) are used to
  test various threads through the software
• Every scenario is traced to parent requirements
• A scenario may have more than one parent
  requirement
• Test cases must demonstrate interface level
  testing
• All system and software level interfaces are
  documented and tracked
• These are known as IRLLs, IRLEs, and IRLMs
• IRLL Interface Requirement Logical Level
• IRLE Interface Requirements Electrical Level
• IRLM Interface Requirements Mechanical Level
• Test cases and requirements are stored in DOORS

20
Software Source Code Control

• The ability to save and retrieve versions of
  software is a key component to success in medium
  to large software projects
• Software can be versioned for new features or
  software load lines
• Changes made to software modules due to bug fixes
  can be isolated
• Enables developers to work on next release while
  maintaining current versions
• Software builds can include or exclude changes

21
Using Source Control Systems
Source Code Version 1
SW Build Specification
Mod A-V1.1 Mod X-V7 Mod Z-V2
Put (ModA V1)
Source Code Control System
Source Code Version 1
get(ModA V1)
New Feature
put(ModA V2)
Source Code Version 2
Software Build
get(ModA V1)
Source Code Version 1
put(ModA V1.1)
Bug Fix
Source Code Version 1.1
22
Trouble Reporting and Bug Tracking

• Software bugs are documented using formal
  mechanisms
• CMTS Change Management Tracking System (MRs)
• WSTR Web System Trouble Report (STRs)
• Bugs are considered addressed when software has
  been modified, tested, and submitted to the
  load-line
• Bugs are considered fixed when they have been
  verified
• Bug fixes are pulled into the software load via
  the source code control system
• Bug reports and time based metrics guide
  management in software release decisions

23
(No Transcript)
24
Test Methods

• Manual
• Usually applied during unit test
• Semi Automated
• Applied during integration test
• Fully Automated
• System test

25
Test Automation

• Test automation can provide a project with huge
  dividends
• Test scripts
• Drive test automation engines
• Can be used in unit testing
• Can be reused by multiple test organizations
• Regression tests can be constructed from reusable
  test scripts

26
Metrics Drive Behavior

• An organization or individuals behavior is
  directly related to the metrics they are measured
  by
• Defects found and fixed
• On time delivery
• Productivity
• SLOC counts
• Defect rate at delivery
• Complexity indices
• Customer satisfaction

27
Best Practices?

• No real silver bullets
• Process is the glue that holds a project together
• The right solution depends on the project but is
  often a combination of methods/processes and
  tools
• Just remember that customer satisfaction trumps
  all!!

28
BACKUP SLIDES
29
ABSTRACT
In the world of software development, process is
the glue that holds the project and team
together. Too little, the pieces fall apart too
much and it can become a sticky quagmire
resulting wasted effort, time, and money. The
challenge facing many projects is finding just
the right balance point on the process continuum.
As a project progresses through its life cycle,
the balance point will shift and the project will
need to continuously review and adjust to
maintain optimal performance.   A typical
software project life cycle is comprised of six
distinct phases Requirements, Design, Code,
Test, Deployment, and Support. Most projects
place a significant amount of emphasis on the
front-end phases of the life cycle e.g.
requirements, and design. However, industry data
clearly demonstrates that the largest percentage
of a projects budget is spent during the final
phases, namely test, deployment, and support.
These phases encapsulate much of the rework
experienced by the project.   This talk will
describe the process philosophies and tools used
by Raytheon to tackle large-scale multisided
software development projects.  
30
Tools

• The right tool can result in huge pay backs
• Source Code Control Tools
• Trouble Report tracking
• Stress testing
• Performance testing

31
Testing Methodologies

• Ad-hoc
• Non structured testing with or without test
  artifacts
• No test plans
• Non repeatable tests
• Formal testing
• Test plans
• Test Artifacts
• Defect tracking

32
Software Phases(Spiral Model)
Spiral Development Experience, Principles, and
Refinements Spiral Development Workshop
February 9, 2000 Barry Boehm, edited by Wilfred
J. Hansen
33
The NATO Sea Sparrow Surface Missile System
(NSSMS) is a quick reaction, point defense
system designed to protect the ship it is on. It
is very, very effective against enemy aircraft,
low flying sea skimming missile and even bomb
laden inflatable boats. It was first developed
by Raytheon in the '70's and installed on over
150 ships world-wide. In the mid '90's the Navy
contracted Raytheon to re-architect the system so
that it would be capable of firing the Navy's
newest missile the Evolved Sea Sparrow Missile
(ESSM). The Navy's brand new nuclear powered
aircraft carrier , the USS Ronald Reagan, has 6
Seasparrow radars 3 Seasparrow missile
launchers. The movie shows the Navy doing
Qualification Testing on these systems.
34
Software Phases(Process Glue)
Requirements Review Do the requirements support
the customers intent? Are they testable? Are
they complete (sunny day/ rainy day) Artifacts
Review documentation Requirements placed under
documentation control
Requirements
Design Review Does the design support
requirements? Artifacts Review documentation
Design
Code Review Does the code support requirements
and implement the design?? Artifacts Review
documentation Code under source code control
Code
Test Review Test plans Do the tests
adequately exercise the requirements? Test
Results Did the tests pass? Artifacts Review
documentation Test plans System Trouble Reports
(STR) or Modification Requests (MR)
Test
Deploy
Test Review Customer Acceptance Test Test
Results Did the tests pass? Artifacts Review
documentation System Trouble Reports (STR) or
Modification Requests (MR)
Support
Customer Survey Is the system meeting customer
expectations Artifacts - Software Updates STRs