Evaluating the Efficacy of Test-Driven Development: Industrial Case Studies

1
Evaluating the Efficacy of Test-Driven
Development Industrial Case Studies

• -Joe Finley

2
Test- Driven Development

• First use- 1960 NASA Project Mercury
• Test code written prior to implementation code.
• Highly Iterative process.
• The goal is to improve quality.

3
Benefits of Test Driven Development

• Efficiency and Feedback Test-then code gives
  continuous feedback
• Low-Level design Tests provide a specification
  of low-level design decisions.
• Reduction of defects injected Patched code
  requires running automated test cases.
• Test Assets Requires writing code that is
  automatically testable. Regression testing
  assets are already built.

4
Related Case Studies

• Empirical study at IBM 40 fewer defects in
  functional verification and regression tests than
  a baseline prior product without reduced
  productivity.
• John Deere, Rolemodel Software and Ericson Small
  Java program using TDD while the control group
  used a waterfall-like approach.
• TDD programmers passed 18 more tests.
• TDD programmers used 16 more time (control group
  did not write automated tests).
• Academic studies
• 1.) TDD-variant (Muller and Hagner) Write
  complete automated unit test cases before any
  production code. Conclusion Not higher quality.
• 2.) Test-First and Test-Last (Erdogmus) 24
  undergrads improved productivity but not quality.
  Conclusion effectiveness of Test-first depends
  on backing up code with test cases.
• 3.) XP 11 undergrads from a testing
  perspective, 87 stated that the execution of
  test cases strengthened their confidence.

5
Microsoft Case Studies - Defect Measurement

• Defect density measured a person makes an
  error that results in a physical fault (or
  defect) in a software element. When this element
  is executed, traversal of the fault/defect may
  put the element (or system) into an erroneous
  state. When this erroneous state results in an
  externally visible anomaly, we say that a failure
  has occurred
• The above definition for defect is used and
  defects are normalized per thousand lines of code
  (KLOC)

6
Microsoft Case Studies

• TDD evaluations done in two different divisions
  Windows and MSN
• Defect density used to measure quality.
• Measure development time increase due to TDD
• Use of CppUnit and NUnit framework indicate
  generalization of results across languages.
• Both project managers report to same manager.

7
Setup Context Factors

• Project A Windows networking team, Project B -
  MSN
• Expertise of project B crew is lower than that
  of project A
• Project A uses C while project B uses C.

8
Product Measures

• Comparable projects chosen by managers with
  similar responsibility.

9
Outcome Measures

• Project A took at least 10-20 more time than
  the comparable project over project B.
• Project B was 5 times larger than A.
• Project Bs comparable non-TDD project had 4.2
  times as many defects / KLOC
• Comparable non-TDD project in Project A had 2.6
  times as many defects / KLOC.. Less due to
  expertise? or size of project?

10
Threats to validity

• TDD developers may have been motivated to produce
  higher quality code since it was a new process.
• TDD projects may have been easier to develop.
• Analysis needs additional repetition in different
  contexts before results are generalized.

11
Conclusions and future work

• Future Work
• Additional research in industry in differing
  contexts.
• Cost-benefit economic analysis on the utility of
  TDD.

12
Questions ?