Id like some more please...

1
Id like some more please...
EE491.20
Maj JW Paul
2
We are what we measure. It's time we measure
what we want to be.
3
Stages from Ch 13 KFN

• first functionality
• almost alpha Maurice
• alpha Oldford
• pre-beta
• internal beta Girard
• outside beta McLellan
• UI freeze Toenders
• pre-final Pollard
• final integrity test Hutter

4
Myth vs Reality

• Myth
• testing is too expensive
• Reality
• pay me now or pay me MUCH more later
• failures in operational systems can cause severe
  secondary problems
• proper testing is very cheap by comparison, even
  when done manually
• efficient testing of large or complex systems
  needs some automated support

5
Review

• OO testing same or different?

acceptance testing (use cases) still a human
endeavor
inheritance (a method for bugs to
propagate) encapsulation (observation/interaction)
polymorphism (multiple test cases)
6
Todays Class

• Bug Classification
• More on Metrics

7
Bug Classification
8
Why classify bugs (defects)

• to help determine trends
• identifying specification errors
• estimate release date
• to learn (from your mistakes...)
• tracking
• to to fix them
• assigning, getting more resources...

9
Some Classifications

• Logic
• Compatibility
• Specification issues
• These tell us what type it is
• project needs to know its status

10
Most use bug status

• pending
• fixed
• irreproducible
• deferred
• as designed
• withdrawn by bug reporter
• need more info
• disagree with bug report
• duplicate of existing bug report

11
Sample bug report
ID 1203 Project Bee Flogger 2.0 Area FTP Client
Title Uploading file causes FTP server to dump
core Assigned To CLOSED Status CLOSED (RESOLVED
- FIXED) Priority 2 - Must Fix Fix For 2.0
Alpha Version Build 2019 Computer Jills iMac,
Mac OS 9.0, 128M RAM, 1024x768 millions of colors
Description 11/1/2000 Opened by Jill M Start
Bee Flogger Create an unnamed document simply
containing the letter "a" Click on the FTP
button on the toolbar Try to ftp to your
serverBUG Observe the ftp server is no longer
responding. Indeed ps -augx shows that
it is not even running and there is a core dump
in /.EXPECTED No crash
From Joel on Software
12
Rules to live by

• 1. list minimal steps to reproduce (script, test
  data)
• 2. what you saw
• 3. what you expected to see
• can only be closed by opener (? resolved)
• fixed, won't fix, postponed, not repro,
  duplicate, or by design
• track versions
• bugs must be reported in database
• keep database minimal

13
The Return of the Metrics
Why collect Metrics?
14
Lord Kelvin on Measuring

• "In physical science the first essential step
  in the direction of learning any
  subject is to find principles of
  numerical reckoning and practicable methods for
  measuring some quality connected with it. I often
  say that when you can measure what you are
  speaking about, and express it in numbers, you
  know something about it but when you cannot
  measure it, when you cannot express it in
  numbers, your knowledge is of a meagre and
  unsatisfactory kind it may be the beginning of
  knowledge, but you have scarcely in your thoughts
  advanced to the state of Science, whatever the
  matter may be."

15
Model/Theory Checking

• Define a goal/theory
• Carry out your project
• Measure the results

• Fitness of Model
• Fitness of Task

• GQM, MC/DC, CMM, ...
• kloc, defect rate, inspection rate, ...
• (many software models metrics to choose from)

16
MetricsThe Dark Side...
17
Generic bug tracking metrics

• Defect find/fix rate
• Percent of tests running/passing
• Complex model (COCOMO)
• Trying to measure project status, program
  readiness for release, milestone status, etc

18
Defect find/fix rate

• Simple ratio - good if lt 1
• Assumes
• All defects that are found are reported
• Goal is fixing all known defects (can be done)
• When defects are fixed, product is ready
• True stories
• Bug bonus
• Report after fixing (or delay)
• Pocket Lists

19
of tests running/passing

• What does running/passing mean
• Assumes
• All tests are known at start
• Everyone agrees on what the following mean
• a test running passing
• True stories
• Combining tests for one failure
• Redefining expected output as it is a known
  failure

20
Complex Model

• 5 X 2 Y Z
• Assumes
• We can accurately measure X, Y, and Z
• Similarity to previous project
• True stories
• Cooking the numbers
• Making counts match expected results
  (not reporting bugs, inflating minor bugs,
  etc)
• Declaring done based on model

21
Kaners Measurement Factors

• purpose of the measure
• scope of the measurement
• attribute to be measured
• appropriate scale for the attribute
• natural variation of the attribute
• instrument that measures the attribute
• scale of the instrument
• variation of measurements
• relationship between attribute instrument
• probable side effects

22
Questions?
23
Next Class

• Guest Lecture - Bill Kelly