The Cost of Inadequate S/W Testing

1
The Cost of Inadequate S/W Testing

• The Economic Impacts of Inadequate Infrastructure
  for Software Testing
• National Institute of standards Technology
  Planning Report 02-3
• May 2002
• www.nist.gov/director/prog-ofc/report02-3.pdf
• An improved infrastructure for testing has the
  potential to affect developers and users by
• removing more bugs before the s/w product is
  released.
• Detecting bugs earlier in the s/w life-cycle.
• Locating the source of bugs earlier and with more
  precision.

2
The Costs of Inadequate S/W Testing

• The study investigates the economic impacts of an
  inadequate infrastructure for s/w testing in the
  U.S.
• Study conducted surveys with both developers and
  industry users.
• Two industry groups were selected for detailed
  analysis.
• Findings from these two groups were then used to
  estimate total economic impacts for U.S.
• Note - Impact estimates do not reflect costs
  associated with mission critical s/w where
  failure can lead to extremely high costs such as
  loss of life or catastrophic failure.

3
Economic Impact on Automotive and Aerospace
Industries

• Study focused on the use of Computer-aided design
  (CAD), computer-aided manufacturing (CAM),
  computer-aided engineering (CAE), and product
  data management (PDM).
• Interviews were conducted with 10 s/w developers
  (vendors), and 179 users of these products.
• Users reported an average of 40 major and 70
  minor bugs per year in their CAD/CAM/CAE/PDM
  software systems.

4
Cost Impacts on Developers and Users

• Developers (vendors) - Costs Impacts of .373
  (bil)
• Potential cost savings from feasible
  infrastructure improvements - .158 bil)
• Software Users
• Automotive - Cost impacts of 1.230 (bil).
  Potential savings of 370 (bil)
• Aerospace - Cost Impacts of .237(bil).
  Potential savings of .055 (bil)
• Total Cost Impacts of 1.840(bil). Total
  Potential savings 0f .598 (bil)
• Users account for approx.. three-quarters of
  total impact, Developers account for the
  remaining one- forth of costs.

5
Economic impact on Financial Services Sector

• Study focused on developers and users of
  Financial Electronic Data Interchange (FEDI) and
  clearinghouse s/w, as well as the s/w embedded in
  routers and switches that support electronic data
  exchange.
• Four s/w developers, and 98 users.
• Users reported an average of 40 major and 49
  minor bugs.
• Approx.. 16 of bugs attributed to router/switch
  problems, and 48 to transaction s/w problems.
  The remaining 36 of errors were unknown.

6
Cost Impacts on developers and Users

• Developers
• Routers and Switches - Cost impact of 1.898
  (bil). Potential savings from feasible
  infrastructure improvements of .975 (bil)
• FEDI and Clearinghouse - Cost impact of .438
  (bil). Potential savings of .225 (bil)
• Software Users
• Banks and savings institutions - Cost impacts of
  .789(bil). Potential savings of .244 (bil)
• Credit Unions - Cost impacts of .217 (bil).
  Potential savingsof .068 (bil)
• Total cost impacts of 3.343(bil). Total
  potential savings of 1.513 (bil)
• Developers account for 75 of costs, with users
  the remaining 25.

7
National Impact Estimates

• The per-employee impacts of these sectors were
  extrapolated to develop an estimate of the
  economic impacts of an inadequate s/w
  infrastructure for s/w testing for the total US
  economy.
• Developers - Cost impacts of 21.2 (bil).
  Potential savings of 10.6 (bil).
• Users - Cost impacts of 38.3 (bil). Potential
  savings of 11.7 (bil)
• Total cost impacts of 59.5(bil). Potential
  savings of 22.2 (bil).
• Developers accounted for about 40 of total
  impacts, with Users accounting for the remaining
  60.