Software initiatives 4 Quality Standards

1
Software initiatives 4Quality Standards

• See Word 97 file
• Software initiatives 4

2
Quality (BURKS)

• The totality of features and characteristics of a
  product or service that bear on its ability to
  satisfy stated or implied needs. Not to be
  mistaken for "degree of excellence" or "fitness
  for use" which meet only part of the definition.
• There is no single, unambiguous definition.

3
Quality Assurance (BURKS)

• A planned and systematic pattern of all actions
  necessary to provide adequate confidence that the
  product optimally fulfils customer's
  expectations.
• This term can be used generically to refer to all
  activities that affect quality in any way.
• It also has a number of more specific
  definitions, such as the label assigned to
  quality assurance departments and to quality
  assurance personnel.

4
Quality Control (BURKS)

• The assessment of product compliance.
  Independently finding deficiencies assures
  compliance of the product with stated
  requirements.
• This term can be used generically for the same
  concepts as "quality assurance" or more
  specifically to specific combinations of
  inspections and test steps applied to software
  projects.

5
Quality Standards (BURKS)

• "The nice thing about standards is that there are
  so many of them to choose from",
• Standards are necessary for interworking,
  portability, and reusability.
• They may be de facto standards for various
  communities, or officially recognised national or
  international standards.

6
Standards?

• Every country has a standards organisation.
• In addition, many industries and technologies
  also have standards bodies, such as the well
  known standards published by the IEEE (Institute
  of Electrical and Electronic Engineers) and the
  European Computer Manufacturers Association
  (ECMA).
• Many large corporations have in-house standards.
  Indeed, for software quality the internal
  standards used at companies such as ATT, IBM,
  Hewlett Packard, Motorola, and many others are
  among the most effective in the world.
• In some cases, the in-house standards pioneered
  entirely new concepts such as Motorola's well
  known Six-Sigma quality standard.

7
Some standards-issuing organisations

• Allied Quality Assurance Publications (AQAP)
  produced by NATO
• American National Standards Institute (ANSI)
• British Standards Institute (BSI)
• Department of Defence (DoD) in the United States
• Department of Defence (Def) in the United Kingdom
• Deutsches Institut fur Normung (DIN)
• European Computer Manufacturing (ECMA)
• European Space Agency (ESA)

8
Some more standards-issuing organisations

• Federal Information Process Standards (FIPS)
• International Organisation for Standards (ISO)
• Institution of Electrical Engineers (IEE)
• Institute of Electrical and Electronic
  Engineering (IEEE)
• Japanese Industrial Standards Committee (JISC)
• Japanese Union of Scientists and Engineers (JUSA)
• Software Publishers Association (SPA)
• Various in-house standards and guidelines by
  specific companies

9
Oh Dear!

• On the whole, the software standards associated
  with quality have had only a marginal effect on
  quality.
• Generally speaking, software standards are based
  on the subjective views of the standards
  committee, and seldom have even much solid
  empirical data behind them.

10
European Software Initiatives

• The combined software population of Western
  Europe and the U.K. is somewhat larger than the
  software population of the United States and very
  energetic in terms of quality initiatives,
  standards, conferences, and Internet
  communication channels.
• Quality initiatives such as the ISO 9000-9004
  standards, now have global impact.

11
The Good

• The same industries tend to achieve the best
  quality results in every industrialised country
  in Europe, North America, South America, the
  Middle East, and the Pacific Rim.
• These industries share a common characteristic
  that their software supports large and complex
  physical devices which need close to zero-defect
  software in order to operate successfully.

12
The Good

• Aerospace manufacturers
• Commercial systems software manufacturers
• Computer manufacturers
• Defence system manufacturers
• Medical instrument manufacturers
• Telecommunication manufacturers

13
The Bad

• Several industries where software quality lags in
  the United States appear to be rather good in
  much of Europe insurance and banking software,
  for example.
• In the USA the insurance and banking software
  development practices tend to be somewhat less
  formal and more casual than the six industries
  with the best quality results.

14
The Bad

• The financial and insurance industries have also
  been rushing to get into the
• client/server world,
• where quality is often marginal and even worse
  than equivalent mainframe applications.
• By contrast, financial and insurance software in
  Europe has a higher probability of rigor during
  development, more care in quality control, and a
  greater use of non-test defect removal methods
  such as inspections.

15
The Ugly

• The quality of government software seems somewhat
  better in several European countries than here in
  the United States.
• Software produced by some of the civilian
  agencies of the Federal Government in the United
  States tends to lag the private sector in terms
  of defect prevention and defect removal
  operations, and hence gets deployed with
  substantially more errors than commercial norms.

16
The Ugly

• By contrast, government-produced software in
  Belgium, Italy, France, Sweden, and the U.K. is
  sometimes equivalent to information systems
  software produced by public company or their
  outsourcing partners.

17
Differences

• Requirements and design approaches are often more
  formal in Europe, and some of the European
  front-end approaches are technically very
  sophisticated
• On the whole, formal methodologies are used more
  often than in the United States, and much more
  often in the information systems and
  client/server domains where U.S. practices tend
  to be rather careless.
• The overall result is that European MIS software
  projects often run about 10 to 20 below
  equivalent U.S. projects in requirements and
  design errors.

18
Differences

• The patterns of programming language usage tend
  to differ between the U.S. and Europe.
• Logic-based languages such as ALGOL, Prolog, and
  Miranda have a higher frequency of use as do
  strongly-typed languages such as PASCAL and
  Modula-2.
• The usage of Ada83 for civilian projects is far
  more common in Europe than in the United States.
• Programming language research is very strong in
  much of Europe.

19
Differences

• Usage of commercial software cost estimating
  tools and other software project management aids
  are less common in Europe than in the United
  States, although both sides of the Atlantic are
  now experiencing rapid growth in project
  management technologies.
• Usage of function point metrics for data
  normalisation is roughly equal between the United
  States and Europe in terms of frequency of use.

20
Differences

• Membership in various function point users'
  groups is growing at about 50 per year in every
  industrialised country, while usage of "1ines of
  code" is declining by about 10 per year.
• In Europe, large function point associations have
  been formed in France, Italy, the Netherlands,
  the United Kingdom, and also a multi-country
  Scandinavian function point group.
• Function point usage is growing but not yet
  formally organised in Austria, Belgium, Germany,
  Greece, Portugal, or Spain.

21
Differences

• Although the factor is more important for
  productivity studies than for quality studies,
  the rather lengthy European vacation periods
  (running to more than 30 days per year in some
  cases) tend to favour countries with more working
  days per year, such as the United States, Japan,
  and India.

22
Differences

• Although a difficult topic to explore since many
  tracking systems don't record it, the amount of
  unpaid overtime applied to software projects
  (i.e., unpaid work in the evenings, on weekends,
  and on public holidays) appears low in Europe and
  the United Kingdom compared to the United States
  and Japan.
• Unpaid overtime is so often part of software
  projects that it can exert an overall
  productivity influence that approaches 20.

23
Differences

• One of the problems of doing direct international
  comparisons between the United States and Europe
  is that the metrics are sometimes different.
• A survey of refereed software engineering
  journals carried out a few years ago noted that
• one-third of the articles using "lines of code"
  were based on physical lines
• one-third were based on logical statements
• the remaining third did not even state the basis
  of the lines of code count.
• almost 20 of the articles did not even identify
  the languages from which the study was based.

24
Differences

• Of course, there are function point variants too.
  
• Between the United States and Europe, at least a
  dozen function point variations can now be
  enumerated.
• On a global basis, however, standard IFPUG
  function points have about 90 penetration in
  terms of published books, articles, and data.

25
ISO 9000 (BURKS)

• A set of international standards for both quality
  management and quality assurance that has been
  adopted by over 90 countries world-wide.
• The ISO 9000 standards apply to all types and
  sizes of organisations.
• Documentation is at the core of ISO 9000
• Say what you do.
• Do what you say.
• Write it down.

26
ISO 9000

• The standards require
• a standard language for documenting quality
  processes
• a system to manage evidence that these practices
  are instituted throughout an organisation and
• third-party auditing to review, certify, and
  maintain certification of organisations.

27
ISO 9000

• Say what you do.
• Do what you say.
• Write it down.

28
Used where?

• The ISO standards are used intermittently in the
  United States, Japan and the Pacific Rim, South
  America, the Middle East, and Eastern Europe.
• The ISO standards have had far and away their
  greatest deployment in Europe and the United
  Kingdom.
• Indeed, they are now often required for marketing
  products within the European Community (EC).

29
Used where?

• Not every company utilises the ISO standards in
  Europe, but the penetration is probably now in
  excess of 20 in Western Europe and the U.K. as
  opposed to perhaps 3 elsewhere based on
  interviews with software managers and QA
  personnel.
• A few U.S. companies that produce only domestic
  software respond to questions about ISO standards
  with a question of their own "What are they?"

30
ISO 9001 and ISO 9000.3

• ISO 9001 is the primary standard on how products
  are built, and a guideline
• ISO 9000.3 explains the ISO approach in a
  software context.

31
But ...

• So far as can be determined, there is not yet
  any tangible or perceptible difference in the
  software quality levels of companies which have
  been certified when compared to similar companies
  in the same industries which have not been
  certified.

32
More But ...

• Currently the most tangible aspects remain
• the high costs of the ISO certification process,
• the large volumes of planning and control
  documents, and
• the time required to achieve certification.

33
Motorolas 2 Objections

• ISO certification was used for marketing purposes
  with an implied assertion that achieving ISO
  certification meant high quality levels
  regardless of the lack of empirical data
  demonstrating that point. Motorola's Vice
  President of Quality, bluntly stated that was
  false advertising.
• The ISO standards themselves lacked some of the
  key quality elements found in Motorola's own
  quality standards and could be viewed as a step
  back rather than a step forward in terms of
  quality methodology.

34
What?

• Some ISO users have observed that it is not the
  place of the ISO 9000-9004 standards to actually
  improve quality.
• Their purpose is simply to ensure that effective
  practices are followed, and are not in and of
  themselves a compilation of best current
  practices.

35
Does ISO 9001 ...

• reduce software defect potentials below 5 per
  function point?
• raise defect removal efficiency levels above 95
  ?
• reduce the number of had test cases below 15 ?
• reduce the rate of "bad fix" injections below 7
  ?
• increase reliability under field conditions?
• increase customer satisfaction levels?
• have any affect on data quality?
• reduce the number of cancelled or delayed
  projects?
• reduce the incidence of litigation for breach of
  contract?
• reduce the incidence of litigation for poor
  quality control?

36
Data?

• Software standards groups should consider the
  methods through which medical and pharmaceutical
  standards are set.
• Although medical standards are not perfect, their
  concept is that a therapy should be proven to be
  effective and its side effects understood before
  it becomes a standard.
• It would be a strong statement that software is
  approaching the status of a real profession if
  software standards included information similar
  to medical and pharmaceutical standards on
  efficacy, counter-indications, known hazards, and
  potential side effects.

37
Data?

• Software standards are normally totally published
  in a way that is totally devoid of empirical data
  regarding their effectiveness when the standards
  are first promulgated.
• The fact that there may possibly be negative or
  harmful side effects from following the standards
  is usually not even discussed, although negative
  side effects are a major section of medical and
  pharmaceutical standards.
• Standards organisations should collect data, and
  should encourage auditors and assessors to move
  toward quantification and empirical results
  rather than subjective opinions.

38
TickIT

• TickIT is a method of evaluating software quality
  that originated in the late 1980s by the BCS.
• The TickIT approach is based on ISO 9000.3 and is
  intended to serve as a consistency check on
  auditing ISO standards as applied to software.
• The TickIT concept includes both training and
  certification of auditors, who are then qualified
  to perform ISO audits.
• The TickIT approach includes on-site interviews
  and assessments and envisions the production of a
  number of quality-related plans and reports.

39
TickIT

• TickIT is widely used throughout the United
  Kingdom, and especially so for projects involving
  the government.
• Since the ISO standards did not originate for
  software, there is a strong need to customise or
  tailor the ISO standards and audits to the
  software world. The TickIT concept is to serve as
  a translator of ISO philosophy into the software
  domain.
• Empirical data on the effectiveness of the TickIT
  approach is sparse.
• The TickIT concept and the ISO standards ignore
  both functional metrics and defect removal
  efficiency measurement.

40
SPICE

• "Software Process Improvement and Capability
  Evaluation."
• The SPICE approach is a composite software
  assessment method based in part on
• the Software Engineering Institute (SEI)
  capability maturity model (CMM)
• the Bell Northern Research TRILLIUM assessment
  approach and
• the International Standards Organisation (ISO)
  9000-9004 standard series.

41
SPICE

• The SPICE approach is moving toward an
  international standard for software process
  assessments that is intended to improve the
  relationship between software contractors and
  clients.
• Many of the SPICE concepts are derived from the
  needs of large corporations and large government
  agencies whose software contracts might run to
  many millions of dollars for key contracts, and
  billions of dollars in aggregate.

42
SPICE

• SPICE is a bold and ambitious program, and like
  many bold programs, also has some risks
  associated with it.
• It is premature to judge the overall impact of
  SPICE since it is still evolving.
• The direction is promising, but when so many
  organisations are involved in standards the
  results are often cumbersome and unsuitable for
  small projects and small companies.

43
Software development in 1984

• More than half of large software systems were
  over 12 months late.
• The average costs of large software systems was
  more than twice the initial budget.
• The cancellation rate of large software systems
  exceeded 35.
• The quality and reliability levels of delivered
  software of all sizes were poor.
• Software personnel were increasing by more than
  10 per year.
• Software was the largest known business expense
  that could not be managed.

44
Software Engineering Institute

• In order to explore software issues, and
  especially those topics associated with defence
  contracts, the Department of Defence in 1984
  funded a new software research facility.
• This was the essential origin of the Software
  Engineering Institute.
• The SEI is located on the campus of Carnegie
  Mellon University in Pittsburgh.
• It is a non-profit software research organisation
  funded primarily by the US government through the
  Defence Advanced Research Project agency (DARPA).

45
SEI Assessment Approach

• The SEI assessment data is collected by means of
  on-site interviews using both a standard
  questionnaire and also observations and informal
  data.
• Once collected, the assessment data is analysed,
  and used to place a software organisation on one
  of the five plateau of the now well-known SEI
  Capability Maturity Model
• CMM

46
System Size?

• The SEI assessment approach originally dealt
  primarily with the software processes and
  methodologies used by very large companies that
  produced very large systems.
• It was derived from the best practices used by
  leading corporations such as IBM and ITT which
  employ from 5,000 to more than 25,000 software
  professionals, and which could safely build
  systems in excess of 1,000,000 lines of code or
  10,000 function points.

47
SEI CMM
48
CMM

• The Software Engineering Institute's model of
  software engineering that specifies five levels
  of maturity of the processes of a software
  organisation.
• CMM offers a framework for evolutionary process
  improvement.
• Originally applied to software development
  (SE-CMM), it has been expanded to cover other
  areas including Human Resources and Software
  Acquitition.

49
Focii - key process areas Level - 1 Initial

• Heroes
• None.

50
Level 2 - Repeatable

• Project Management
• Software Project Planning, Software Project
  Tracking and Oversight, Software Subcontract
  Management, Software Quality Assurance, Software
  Configuration Management, Requirements
  Management.

51
Level 3 - Defined

• Engineering Process
• Organisation Process Focus, Organisation Process
  Definition, Peer Reviews, Training Program,
  Inter-group Coordination, Software Product
  Engineering, Integrated Software Management.

52
Level 4 - Managed

• Product and Process Quality
• Software Quality Management, Quantitative Process
  Management.

53
Level 5 - Optimising

• Continuous Improvement
• Process Change Management, Technology Change
  Management, Defect Prevention.

54
P-CMM

• The SEI assessment program was criticised for
  concentrating on too narrow a set of topics, and
  ignoring other topics such as personnel,
  compensation, hiring practices, and the like.
• In 1995 the SEI filled a notable gap by
  publishing a People Capability Maturity Model or
  P-CMM.
• The lead researcher on the P-CMM was a cognitive
  psychologist, Dr. Bill Curtis.
• He had formerly headed the SEI assessment practice

55
Level 1 initial

• Characterised by random or chaotic development
  methods with little formality and uninformed
  project management.
• Some smaller projects may be successful due to
  the capabilities of individual staff members,
  large systems are often failures and the overall
  results are marginal to poor.
• In terms P-CMM, such organisations are asserted
  to be deficient in training at both the technical
  staff and managerial levels.

56
Level 1 initial

• SEI does not recommend any key process areas for
  Level 1 organisations on the grounds that they
  are not sophisticated enough to deploy methods
  consistently and will probably botch things up if
  they try.
• Moving from Level 1 to Level 2 is the most
  difficult transition point.
• For a large company with perhaps 1,000 software
  personnel, the move from Level 1 to Level 2 can
  take from 18 to more than 36 months.

57
Level 2 repeatable

• Level 2 or Repeatable organisations have
  introduced at least some rigor into project
  management and technical development tasks.
• Approaches such as formal cost estimating are
  noted for project management, and formal
  requirements gathering are often noted during
  development.
• Compared to the initial level, a higher frequency
  of success and a lower incidence of overruns and
  cancelled projects can be observed.

58
Level 2 repeatable

• In terms P-CMM, the Level 2 organisations have
  begun to provide adequate training for managers
  and technical staff.
• Further, they have become aware of the importance
  of professional growth and the need for selecting
  and keeping capable personnel.

59
Level 2 repeatable

• The key process areas which SEI recommends for
  Level 2 organisations include requirements
  management, project planning, resource tracking,
  quality assurance, and configuration management
  plus subcontract management for projects that
  utilise multiple contractors.
• The key process areas for Level 2 of the People
  CMM include compensation, training, staffing,
  communication, and work environment.

60
Level 2 to 3?

• Moving from Level 2 to Level 3 is not as
  difficult as the first move, since organisations
  have usually become used to the idea that change
  will be necessary.
• However, the same size organisation illustrated
  previously with 1000 software personnel, should
  not expect to go from Level 2 to Level 3 in less
  than a year, and it may take more than 18 months.

61
Level 3 defined

• Level 3 or Defined organisations have mastered a
  development process that can often lead to
  successful large systems.
• Over and above the project management and
  technical approaches found in Level 2
  organisations, the Level 3 groups have a
  well-defined development process that can handle
  all sizes and kinds of projects.

62
Level 3 defined

• In terms of the P-CMM, the Level 3 organisations
  have developed skills inventories and are now
  capable of selecting appropriate specialists who
  may be needed for critical topics such as
  testing, quality assurance, web mastery, and the
  like.
• Level 3 organisations should be capable of
  effective software reusability programs.

63
Level 3 defined

• The key process areas which SEI recommends for
  Level 3 organisations include establishing an
  effective organisational infrastructure, training
  of both managers and technical staff, inter-group
  co-ordination, and the utilisation of formal
  design and code inspections.
• The key process areas for Level 3 of the P-CMM
  include career development, competency-based
  practices, work force planning, and analysis of
  the knowledge and skills that are needed by the
  organisation.

64
Level 3 to 4?

• Organisations that achieve Level 3 sometimes rest
  on their laurels, since Level 3 is the minimum
  level needed to bid on some U.S. defence
  projects.
• Organisations that are pursing "best in class"
  goals will find Level 3 to be only a way station.
  
• For these energetic groups, moving from Level 3
  to Level 4 can sometimes be accomplished in less
  than 12 months.

65
Level 4 managed

• Level 4 or Managed organisations have established
  a firm quantitative basis for project management
  and utilise both effective measurements and also
  effective cost and quality estimation.
• In terms of the People CMM, the Level 4
  organisations are able to not only monitor their
  need for specialised personnel, but are actually
  able to explore the productivity and quality
  results associated from the presence of
  specialists in a quantitative way.

66
Level 4 managed

• Long-range predictions of future needs are also
  part of the Level 4 P-CMM assertions. In
  addition, Level 4 organisations would make
  extensive use of mentoring.
• The key process areas which SEI recommends for
  Level 4 organisations include quantification of
  defect levels, productivity levels, and
  activity-based costing.
• Level 4 organisations should be approaching 50
  levels of software reuse for both design, code,
  and testing materials.
• The key process areas for Level 4 of the P-CMM
  include mentoring, team building, organisational
  competency, and the ability to predict and
  measure the effect of specialists and teams in a
  quantitative manner.

67
Level 4 to 5?

• Moving from Level 4 all the way to Level 5 is not
  easy to quantify because the actual criteria for
  Level 5 are somewhat abstract.
• There is no reason to assume that a transition
  from Level 4 to Level 5 should take more than 12
  months.

68
Level 5 optimising

• Level 5 or Optimising organisations are assumed
  to have mastered the current state-of-the-art of
  software project management and development.
• Such groups are expected to be the pioneers who
  are capable of building entirely new methods that
  advance the state of the art beyond current
  levels.
• Although not explicitly stated, Level 5
  organisations should be approaching or exceeding
  a volume of about 75 of reusable design, code,
  and test materials.

69
Level 5 optimising

• In terms of the P-CMM, the requirements are an
  extension of Level 4 and hence different more in
  degree than in kind.
• The P-CMM stresses both coaching and rewards for
  innovation at Level 5.
• The key process areas recommended for Level 5
  include defect prevention and advancing the
  fundamental software engineering and management
  technologies, plus rapid and effective technology
  transfer and deployment of improvement
  approaches.
• The key process areas for the P-CMM include
  encouragement of innovation, coaching, and
  personal competency development.

70
Same again?

• When the SEI capability maturity model was first
  published it was asserted that both quality and
  productivity levels would improve at the higher
  levels.
• This is an area where SEI shares the bad habits
  of the rest of the software industry. The SEI
  assertions about quality and productivity were
  initially made without any kind of study or
  empirical evidence being collected.
• In fact, even in 1997 the SEI assessment approach
  itself still does not include the collection of
  quantitative information on either productivity
  rates or quality levels.
• This is a serious flaw in the SEI assessment
  procedure.

71
Level overlaps
72
Six-Sigma Quality Levels

• The Motorola Corporation has received substantial
  interest in its famous six-sigma quality
  program for hardware and manufactured components.
  
• The phrase refers to defect densities of 3.4 in
  a million.
• Since the six-sigma definition is hard to
  visualise for software, an alternative approach
  would be to achieve a cumulative defect removal
  efficiency rate of 99.999999 prior to delivery
  of a product.
• This method is not what Motorola uses but it
  helps to clarify what would have to be done to
  achieve six-sigma results for software projects.

73
Six-Sigma Quality Levels

• Since the current U.S. average for software
  defect removal efficiency is only about 85, and
  quite a few software products are even below 70,
  it may be seen that software producers have some
  serious catching up to do.
• As it happens, there are a few software products
  which appear to have achieved six-sigma quality
  levels.
• In 30 years Capers Jones has observed four
  projects that achieved six-sigma results in their
  first year of deployment out of a total of almost
  10,000 projects.

74
Six-Sigma Quality Levels

• Surprisingly, the prognosis for achieving
  six-sigma quality levels for software is fairly
  good.
• The best of the available technologies today can
  approach this level if
• a full suite of defect prevention and
• defect removal operations are carried out
  synergistically.
• Indeed, for software projects where reuse of
  certified materials top 75 by volume, and where
  formal inspections are used to augment testing,
  six-sigma quality levels should be achieved
  routinely.

75
Six-Sigma Quality Levels

• That does not mean that the software industry is
  ready to try.
• There are both sociological and technical reasons
  that quality is held back, such as
• the widespread lack of appreciation of the
  correlation between high quality and short
  schedules and
• the lack of commercial reusable materials
  certified to zero-defect levels.