Classic Testing Mistakes: Revisited

1
Classic Testing MistakesRevisited
Contributing peer reviewers James Bach Paul
Carvalho Michael Kelly Harry Robinson

• Matthew Heusser
• mheusser_at_charter.net
• Presented at the Indiana QA Conference
• Indianapolis, Oct 13th, 2006

2
Organization

• Classic Mistakes A different approach
• The mistakes enumerated
• Test Management Mistakes
• Test Automation Mistakes
• Development Mistakes
• Test Strategy Mistakes
• Root Causes
• What to do tomorrow

3
(No Transcript)
4
(No Transcript)
5
Classic Mistake 1De-humanize the test process
AKA Management by Spreadsheet, Management by
Email, Management by MS Project
Test Management Mistakes
6
Classic Mistake 2 Testers Responsible for
Quality
Its strange that QA let that bug slip through
Test Management Mistakes
7
Classic Mistake 3IVV Determines ship date
Do they really?
Test Management Mistakes
8
Classic Mistake 4Task-based status reporting

• Examples
• Testing is on schedule
• Testing should be done by Tuesday
• Consequences
• Loss of credibility
• Bad information for decision makers

Test Management Mistakes
9
Classic Mistake 5Evaluating testers by bugs
found

• and developers by number of bugs injected
• Consequences
• Friction
• Focus on easy-to-find yet trivial bugs
  (usability)
• Information hiding

Test Management Mistakes
10
Classic Mistakes 6Inappropriate Models for Test
Improvement

• NO

11
Classic Mistake 7Lack of test training for
developers

• Testing is a skill.
• It wont appear like magic.

Development Testing Mistakes
12
Classic Mistake 8Separate devs and testers

• To create friction, emphasize division
• Anything that increases the length of the
  feedback loop is bad.
• To improve get rid of waste and tighten the
  feedback loop.

Development Testing Mistakes
13
Mistake 9When late, add Test Automation

• Someone has to learn the tool
• Someone has to record the scripts

Test Automation Mistakes
14
Mistake 10Mine Field Test Automation
Test Automation Mistakes
15
Mistake 11Hiring for test tool skills

• Technology skills can be taught
• Talent cant
• The Hit the Ground Running Argument

Test Automation Mistakes
16
Classic Mistake 12Insufficient diversity in
test strategy

• Examples
• Only requirements based testing
• Only coverage testing
• Consequence
• Missing entire classifications of defects

Test Strategy Mistakes
17
Classic Mistake 13Over-reliance on scripted
testing
All the testing we did, meticulously pulling down
every menu and seeing if it worked right, didn't
uncover the showstoppers that made it impossible
to do what the product was intended to allow.
Trying to use the product, as a customer would,
found these showstoppers in a minute. - Joel
Spolsky, JoelOnSoftware.com
Test Strategy Mistakes
18
Mistake 14Untrained exploratory Testing

• Just think creatively
• Try to break it
• Exploratory Testing is a discipline

Test Strategy Mistakes
19
Classic Mistake 15 Test Engineers and
Executors
Test Strategy Mistakes
20
Classic Mistake 16Vacuous Documentation

• Examples
• The issue resolution document
• Physical signoff/check marks
• Elaborate test case templates
• Consequence
• Time spent documenting is time not spent testing

Test Strategy Mistakes
21
Mistake 17Trying to fix things beyond your
reach
The Meta-Mistake
22
Root Cause 1Lack of Systems Thinking in Testing

• The law of unintended consequences

Root Causes
23
Root Cause 2Translation Problems
Example - You need to completely test this
module
Root Causes
24
Root Cause 3Process Myopia

• Example
• The infamous Issue Resolution Document
• We dont do things that way here
• Elevating process over skills
• Solutions
• The ear of the king / History Lessons
• If your boss doesnt care ignore it

Root Causes
25
A new methodology

• Root Cause Analysis
• Pareto Analysis
• Drive out waste/tighten the feedback loop
• Then worry about better practices
• (Image from Rapid Development, (c) 1996 by Steve
  McConnell. Used with permission from the author)

What to do tomorrow
26
What to do tomorrow

• Discuss
• QA

27
Where to go for more

• Classic Mistakes in Software Testing, Brian
  Marick, STQE, 1997
• Rapid Development, Steve McConnell
• An Introduction to general systems thinking,
  Gerold Weinberg, 1975
• Lessons Learned in Software Testing, Kaner, Bach,
  Pettichord

28
Bonus Section
29
Root Cause 4Technology Myopia

• Example
• Use XML on the next project
• I just bought 5 copies of WinRunner
• Solution
• If youre technical, they need you to do it
• If youre a manager, focus on business impact and
  risk

Root Causes
30
Root Cause 5Fred Taylor in the organization

• Examples
• Factory Mentality
• High Specialization
• Mixing of skill sets is verboten
• Solutions
• Peopleware, or anything by Weinberg
• First Break all the rules Jim Collins
• Lead. Insulate your team.

Root Causes
31
Root Cause 6Pressure for short-term results

• Example
• Ship to make 4th quarter numbers
• Putting off problems instead of addressing them
• Solution
• Save your team
• Professionalism means something
• The Quake Example

Root Causes