A%20Comprehensive%20Approach%20to%20Quality%20(COACH)

1
A Comprehensive Approach to Quality (COACH)

• IETF 57
• Vienna, Austria
• Monday, July 14, 2003
• 1300 1500
• http//www.drizzle.com/aboba/IETF57/COACH/

2
Agenda

• Preliminaries (1300 1305)
• Agenda Bashing
• Bluesheets
• Minutes
• Introduction (1305-1315)
• Existing RFC editorial guidelines - Scott Bradner
  
• http//www.ietf.org/rfc/rfc2360.txt
• Quality Overview and Framework (1315-1345)
• A Comprehensive Approach to Quality (15 minutes),
  Bernard Aboba
• http//www.ietf.org/internet-drafts/draft-loughney
  -coach-00.txt
• IETF Problem Resolution Processes (15 minutes) -
  Margaret Wasserman
• http//www.ietf.org/internet-drafts/draft-ietf-pro
  blem-process-00.txt
• Starting New Work (1345-1405)
• The BOF Process A Critique (10 minutes) - Leslie
  Daigle
• IESG Overload and Quality of WGs (10 minutes) -
  John Klensin
• http//www.ietf.org/internet-drafts/draft-klensin-
  overload-00.txt

3
Agenda

• The WG Process (1405-1415)
• Decision points/milestones in the WG process (10
  minutes) - Margaret Wasserman
• http//www.ietf.org/internet-drafts/draft-wasserma
  n-wg-process-00.txt
• The Review Process (1415-1440)
• Careful Additional Review of Documents (CARD) (15
  minutes) - Brian Carpenter
• http//www.ietf.org/internet-drafts/draft-carpente
  r-solution-sirs-01.txt
• The Review Process in Action The DCCP WG (10
  minutes) - Aaron Falk Allison Mankin
• Summary and Discussion (1440 1500)

4
Some Groundrules

• A mindset.
• The truth is out there - Mulder
• A reading List.
• People who have read at least three of the
  documents, please sit in the first three rows.
• A format.
• Strict time limit for each topic presentation.
• QA at the end.
• Expectations of Behavior.
• Please use the mike.
• Please state your name before speaking.
• Only one person at the mike at a time.
• Please do not interrupt the person at the mike.
• Please relinquish the mike when requested to do
  so by the chair.
• Please show courtesy to your fellow IETF
  participants.

5
A Comprehensive Approach to Quality (COACH)

• John Loughney
• Bernard Aboba
• Draft-loughney-coach-00.txt

6
Basic Principles

• IETF Working Groups are responsible for
  completing work
• And are accountable for the quality of that work.
  
• If Working Groups are to improve the quality (and
  timeliness) of the work
• It is necessary for them to plan for, and carry
  out, that improvement.
• If the IETF is to improve
• It is necessary to do post-mortems on the plans.

7
A Theory

• Many reasons why Working Groups deliver poor
  quality documents.
• Unlikely that the plan was to deliver poor
  quality documents all along.
• More likely that the level of effort, skill and
  experience required is beyond the capability of
  working group participants.
• We cant all be experts in everything.
• Poor quality not the result of bad intentions,
  but insufficient resources.
• Naïve optimism required to start the work
• But not enough to finish it.

8
What is a WG Quality Plan?

• A public document no more than 5 pages long
• Made available on the IETF web site.
• Developed in concert with the Working Group
  charter
• Revised when the WG charter or schedule changes
  significantly.
• Since each WG is different, no one size fits
  all quality plan.
• Sharing of best practices is encouraged.
• Signed off by the AD and the WG founders
• Reviewed by the IETF community

9
Potential Components of a WG Quality Plan

• WG charter the foundation.
• What the WG plans to do (documents), who will do
  it (chairs, editors, advisors), by when
  (milestones schedule).
• Challenge assessment whats in the way (reality
  check)
• The technical issues.
• The dependencies.
• Areas of skill, knowledge or experience that are
  lacking.
• Tools plan.
• WG mailing list.
• WG web site.
• Issue tracking tools.
• Revision control systems.
• Document production and build environments.
• Review plan.
• Checkpoints.
• Required reviewer skill set.
• Review process.
• Reviewer recruitment.

10
Challenge Assessment

• The Charter What the WG is trying to achieve,
  and the resources at its disposal.
• The Challenge Assessment what does the WG need
  to do in order to have a high probability of
  completing the work on schedule and with high
  quality?
• Some questions
• Are the goals achievable?
• With infinite resources? With the resources
  available?
• What are the key assumptions?
• What are the risks?
• Within the core area of expertise of the WG?
• Outside the core area of expertise?
• How can the risks be mitigated?

11
The Tools Plan

• The technology and host for the Working Group
  mailing list.
• Anti-spam plan.
• Archive access.
• The Working Group web site.
• Permanence? BW limits?
• Document production system.
• Tools templates
• Build environment
• Revision control system.
• Issue tracking and reporting system.

12
Issue Tracking Reporting

• An important tool for
• Tracking issues encourages participants to raise
  issues, knowing that they wont be forgotten
• Focusing WG discussion encourages discussions
  that lead to document improvements.
• Measuring progress
• Provides metrics for assessing current status
• Open/resolved
• Accept/reject
• Days to resolution
• Enables estimates of the task remaining

13
The Review Plan

• Challenge Assessment review.
• Review of the WG Charter
• Should include reviewers from outside the area of
  the Working Group
• Work item review.
• Review of documents before they become WG work
  items.
• Does the architecture make sense?
• Are there major issues lurking?
• Midterm assessment.
• Is the document on the right track? Has something
  important been missed?
• Late review.
• Is the document ready for WG/IETF last call?
• More on this later

14
An Observation

• Most documents are competent in the Area in which
  the WG resides.
• Transport Area WGs understand transport.
• Security Area WGs understand security issues.
• Problems, when they occur, are generally in areas
  outside of the core competence of the WG
• Internet Area documents with security issues.
• Security documents with transport issues.
• Suggestion Cross-Area review is an important
  aspect of the review plan.

15
The Post Mortem

• No more than three pages long.
• Quality assessment.
• An evaluation of the quality of working group
  output, written by the Area Director.
• Challenge Assessment.
• The unforseen challenges that the Working Group
  encountered.
• Tools assessment.
• An evaluation of the tools that were used and how
  well they worked.
• Reviews assessment.
• An assessment of the review process.
• Recommendations.
• Recommendations to future working groups looking
  to avoid similar problems.

16
Required Materials(What a WG might produce?)

• An archive of sample document build environments
  and templates.
• A website covering issue tracking and reporting
  tools.
• A document on use of tracking tools for document
  management, including metrics and reporting.
• A document on the review process, including
  Challenge Assessment.
• A website including sample quality plans and
  post-mortems.

17
Summary

• Quality doesnt just happen, it can be planned.
• The WG charter provides what, who, when.
• The WG quality plan provides the how.
• The resources available need to match those
  required to finish the job.
• It isnt what we know, its what we think we know
  that isnt so.

18
Feedback?