Definitions and Concepts of Testing and Quality

1
Definitions and Concepts ofTesting and Quality
What is Quality ?
What is Testing?
How Related?
2
Introduction to Testing Software

• Testing is a relatively new discipline although
  programmers always debugged their programs.
• Testing was conducted to show that the software
  works.
• In the 1970s Glen Myers wrote a book, Art of
  Software Testing (1978).
• He believed that the main purpose of testing is
  to find faults.
• Are these (works .vs. faults) opposing views?
• Why should we and why do we Test ?

3
Why Test ?

• We cant assume that the software works
• How do we answer the question Does this
  software work?
• Is it reliable ?
• Is it functional ?
• Complete
• Consistent
• Is it responsive ?
• In general, what is the Quality of our software?
• How do we answer this?

How would you answer this about the program that
you wrote? How would you answer this about the
program your friend wrote?
4
Quality of Software?

• Depends on what we include as software
• Just the executable code ----- How about ?
• Include the pre-populated data in the database
• Include the help text and error messages
• Include the source code
• Include the design document
• Include the test scenarios and test results
• Include the reference manuals
• Include the requirements document
• When we talk about quality how much of the above
  do we include?
• How do we test these different artifacts?

5
Testing

• Traditionally, testing includes executing the
  code. (assuming - code is the main software
  artifact)
• What do we do with the non-executable software
  artifacts?
• Reviews and inspections
• Expensive in terms of human resources
• A lot to maintain and keep updated
• Can we prove that the software works or is
  defect free?
• Theoretically, given an arbitrary program we can
  not show that it has no bug.
• We can use formal proofs to show the behavior of
  code

6
An Informal Survey in the 90s

• Professionals taking a course in testing
• 60 were new to testing
• 20 had 1 to 5 years of experience in testing
• 20 expert testers
• Metric used in testing
• Most regularly used Counting bugs found and
  ranking them by severity
• Small number used bug find rate or bug fix rate
• Formal Methods used
• Almost none in inspection or formal analysis
• Test tool
• 76 had been exposed to some automated test tool
  
• Test definitions
• Most of the practicing testers could not supply
  good definitions of testing terms they just did
  the work!

7
Historical Progression of Software Quality
-Large Host Centralized Systems -Single
Vendor(hdw,sw,serv) -Long term investment(10
yrs) -Single platform -Systems ran by computer
professionals Product Reliability Quality
was required and expected
-PC and Desktop Computing became
ubiquitous -multiple vendors -quick product
development shorter term investment -systems
ran by non-computer individuals New product
was fashionable reboot was acceptable.
1980s Before
-Web availability to many -Business conducted on
the Web -Software and systems are not hobbies
but a business again Product Reliability
Quality is once again important
1990s
Late 1990s 2000s
8
Current State on Testing

• Software is written by many with the
  entrepreneurial spirit
• Speed to market
• New innovation is treasured
• Small organization that cant afford much more
  than coders
• Embracing Agile process and mistaking it as
  fast production regardless of what
• Not much documented (requirements/design)
• Hard to test without documented material
• Lack of Trained/Good/Experienced Testers
• Testers are not rewarded as equals with designers
• Improvement tools and standards making
  development easier and less error prone

Why ?
9
Quality

• What is Quality?
• Some common comments
• I know it when I see it
• Reliable
• Meets requirements
• Fit for use
• Easy to use
• Responsive
• Full function / full feature
• Classy and luxurious

How would you define quality?
10
Some U.S. pioneers on Quality

• Pioneers
• Joseph M. Juran
• Quality gtFitness for use
• W. Edward Deming
• Quality gt Non-faulty system
• More recently
• Philip Crosby
• Quality gt conformance with requirements
• Achieve quality via prevention of error
• Target of Quality is zero defect
• Measure success via cost of quality

11
Quality

• Quality is a characteristic or attribute
• Needs to be clearly defined and agreed to
• May have sub-attributes
• Needs specific metrics for each of the
  sub-attributes
• Needs to be measured with the defined metric(s)
• Needs to be tracked and analyzed
• Needs to be projected
• Needs to be controlled
• Testing and Measurement are two key activities
  that would help us manage quality.

Note Hutcheson, in your text book states that
quality is a metric and that it measures
excellence Discuss --- agree / disagree why?
What is a Metric?
12
Some Precept Concerning Quality QA

• Quality Requirements Does not Dictate Schedule
• Market dictates schedule (especially for small
  companies)
• For large and multiple release software, quality
  is still a factor and may affect schedule ----
  albeit seldom
• Software development process today incorporates
  both the need for speed and quality
  (incorporating the notion of service cost versus
  rewrite for a replacement new product.)
• Quality does not require zero defect
  Reliability
• Commercial (non-life critical or mission
  critical) products are not developed with this
  goal in mind. They are much more market driven
  --- market does not demand zero defects.
• Focus on proper support
• Focus on main functions and heavily used areas
  (not all defects are the same)
• Focus on customer productivity (e.g. easy to
  learn and use)
• Zero Defect is very expensive proposition ( time
  resource)

13
Some Precept Concerning Quality QA(cont.)

• Users Do Not Always Know What They Want
• Users may not know all the requirements
  (especially for large, complex systems which
  require professional or legal knowledge.)
• Users may not have the time or interest to
  really focus on the requirements at the time
  when asked (timing problem). Users have their own
  fulltime jobs.
• Users may not know how to prioritize needs from
  wishes
• Users may not know how to articulate clearly all
  the requirements. (They are non-software
  development people.)
• Developers may not listen well or properly
  interpret the users statements. (They are not
  industry specialists.)

Requirements is a key factor in software
development ---- why? How does it affect software
quality? ----- think about definitions of Quality
14
Some Precept Concerning Quality QA(cont.)

• Requirements are not always Clear, Stable, and
  Doable
• Not all requirements are technically feasible
  sometimes the desired new technology needs to
  be prototyped first.
• Sometimes the requirements are changed, causing
  re-design or re-code without proper assessment of
  schedule impact.
• Requirements are not always reviewed and signed
  off, but sometimes given in verbal form ---
  especially small changes.
• People mistake iterative development to mean
  continuous change of requirements.

Whats the danger here? cost, schedule, quality
15
Some Precept Concerning Quality QA(cont.)

• Customers often go for New and Exciting Product
• If the product has all the needed features it
  would sell is not necessarily true people often
  want new features.
• Reliability is not always enough sometimes
  customers will sacrifice quality for new and
  exciting features.
• Recall the story of IBM OS/2 operating system and
  Microsofts operating system (even though both
  was commissioned by IBM).
• IBM went for Reliability of the old Host Machines
  for desktop PCs
• Microsoft went for exciting individual user
  interfaces

Over emphasis of exciting features is one
reason why we are regressing in software quality
in the Last ten years !
16
Some Precept Concerning Quality QA(cont.)

• Price and Availability is sometimes more
  important to customers (especially in commodity
  level software) than Product Maturity.
• At the commodity level software, the customers
  are individuals who wants the product now at an
  competitive price. (much like shopping for a home
  product such as a T.V.)
• Sophisticated and full feature software needs to
  be balanced and sometimes traded off for price
  and speed.
• Customers dont always want all the functions and
  product maturity they think they require ---- if
  the price is right!

17
Controlling Quality

• As software size and complexity increased in the
  1960s and 1970s, many software projects started
  to fail.
• Many software did not perform the required
  functions
• Others had performance (speed) problems
• Some had large number of defects that prevented
  users from completing their work some just flat
  out wont even install !
• Software Quality Crisis was recognized and
  Quality Assurance was born.

18
Software Quality Assurance (QA)

• Software QA focused on 2 main areas
• Software product
• Software process
• The focus on the process areas borrowed many
  techniques from the traditional manufacturing
  area
• Concept of reliability (number of defects, mean
  time to failure, probability of failure, etc.
  metrics)
• Concept of process control in terms of looking at
  repeatability of process--- repeatable process
  produces similar product (or controllable
  results).

19
QA, Process Control, and Documentation

• A period of heavy emphasis on software
  development process and excessive documentation
  dominated QA --- initially this improved the
  software crisis.
• Process included multiple steps of reviews
• Process included multiple steps of test
  preparation, testing, and test result analysis
• Process was controlled by many documents and
  document flow which also improved project
  communications
• But ----- the price was speed and innovation.

20
Software is NOT manufacturing (or is it?)

• Software Development is extremely labor intensive
• People are not uniform like machines used in
  manufacturing.
• Software Development often requires innovation
• Every software seems to be a one of its kind
  although more and more are becoming standardized

21
Some ideas of adjusting the QA

• Hutchesons Goals of QA
• Quality is defined as customer satisfaction
• Quality is achieved by constant refinement
• Quality is measured by profit
• Quality process should produce a hit everytime
• Hutcheson Approaches to Quality Product
• Be first to market with the product
• Price the product correctly
• Ensure the product has both required functions
  plus wishful features
• Ensure the bugs are kept to minimum ---- less
  than your competitors bugs.

Do you agree? ---- the end product may have many
bugs but that may
be ok if it is less than competitors and if the
customers dont
mind ------ what about minimizing fix cost by
having less
bugs? ( the author forgot support business)
22
Areas of Improvements for QA Process and Control

• Automate Record Keeping
• Improve Documentation Techniques
• Use trained testers and improve on their tools
  and methodologies

23
Automate Record Keeping

• Many records are kept and should be automated
  with on-line forms
• Test plan
• Test schedule
• Test scripts
• Test results
• Etc.
• The information often should be available to a
  set of people
• Repository or Configuration Management
• Collaborative web-sites

24
Improve Documentation Techniques

• Improve the creation and maintenance of documents
  via an on-line repository or configuration
  manager
• Centrally protected data
• Sharable data
• Managed data (data relationships are managed)
• Improve the usability and productivity by
  providing more and better visualization of data
• Replace numbers with graphs and figures
• Replace words with pictures

25
Improve Testers Tools and Methodology

• Test Methodology Improvements
• Test coverage analysis
• Test case generation
• Test-Fix-Integrate Process
• Test results analysis
• Test metrics definition and measurements process
• Etc.
• Test tools improvements
• Test coverage computation
• Test trace
• Test script generator
• Test result records keeping and automated
  analysis
• Build and integration (daily builds)
• Etc.