Writing Quality Specifications July 9, 2004

1
Writing Quality SpecificationsJuly 9, 2004
TGDC Meeting

• Mark Skall
• Acting Director, Information Technology
  Laboratory
• National Institute of Standards and Technology
• mark.skall_at_nist.gov

2
Background

• NIST works with industry and Federal agencies to
  develop standards and tests to improve the
  quality of software and achieve interoperable
  solutions
• NIST has many years experience with
• Formal standards organizations and Consortia
  (W3C, OASIS, ANSI, ISO, IEEE, HL7)
• Helping to develop good specifications
• Developing conformance test suites, tools,
  reference implementations
• Developing validation and certification testing
  programs

3
Moving Towards Trust and Confidence
Specification
Requirements
Implementation
Conformance
Testing activities
4
Good Specifications are the Key

• Goal is correct, reliable software
• Requirements are captured in a specification
• Spec must use appropriate language to designate
  requirements
• Spec must be precise, unambiguous, and testable
• Spec must contain a conformance clause
• Ideal spec would be defined in a formal language
  not English

5
Specs Must Use Appropriate Language to Designate
Requirements

• MUST, MUST NOT, SHALL, SHALL NOT are
  examples of mandatory requirements
• SHOULD, SHOULD NOT, MAY, RECOMMEND are
  not mandatory, but are often used. If used, they
  have no bearing on conformance
• Good enough, Do the best you can is not
  appropriate language. They are not precise, and
  cant be tested

6
Specifications Must Be Precise

• A requirement is precise if one can determine if
  the requirement has been fulfilled
• An imprecise requirement
• Systems shall guard against viruses, trojan
  horses, and worms.
• A precise requirement
• An anti-virus program (from an approved list)
  shall be installed on ones system and anti-virus
  software checks and live updates shall be run
  daily.

7
Specifications Must Be Unambiguous

• The girl touched the cat with a feather
• Meaning
• (a) The girl touched (the cat with a feather) or
  
  
• (b) (The girl touched the cat) with a feather
• Does the girl or the cat have
  a feather?

8
Specs Must Be Testable

• If a requirement cannot be tested (i.e., a method
  cannot be devised to determine whether a
  particular requirement is met) one cannot verify
  that it has been satisfied
• Reformulate it so it can be tested or
• Eliminate the requirement from the standard
• Testable assertions will be derived from
  normative statements in the specification
  (must, must not, shall, shall not...)
• Unspecified, ambiguous, or imprecise requirements
  cannot be tested

9
Conformance Clause

• Conformance is the fulfillment of a product,
  process, or service of specified requirements
• A conformance clause is a high-level description
  of what is required of implementers and
  application developers. Its a section of a
  specification that states all the requirements or
  criteria that must be satisfied to claim
  conformance to the specification.
• Who (what types of entities) may conform
• How - What must they do to conform?
• It may, for conformance purposes, refer to
  functional subsets, such as profiles, levels, or
  other structures.
• Additionally it should specify the permissibility
  of extensions, options, and alternative
  approaches and how they are to be handled.

10
What Makes a Good Spec

• NIST is working with W3C to define how a spec
  should be written
• Addresses what a spec should contain with respect
  to conformance and testability
• Addresses
• How to define conformance
• How to sub-divide a spec
• Discretionary items
• Extensions
• Test assertions

11
Relevant NIST Efforts

• Guidance on writing better specifications
• W3C Quality Assurance effort
• OASIS Conformance Requirements for Specs document
• Conformance advisory
• Test methods and techniques
• Automatic Test Generation from formal
  specification
• Automatic Test Generation using XML Technologies
• Software Component Integration Testing
• Adaptable Conformance Test Method
• Develop XML conformance tests
• National Software Reference Library (NSRL) to
  determine whether software has been altered

12
Contact

• For more information, please contact
• Mark Skall
• mark.skall_at_nist.gov
• 301-975-3262
• http//www.itl.nist.gov/div897/