Benchmarking%20Methodology%20WG%20(bmwg)%2064th%20IETF%20

1
Benchmarking Methodology WG (bmwg)64th IETF
Vancouver, Canada

• Monday, November 7, 2005, 1510-1710 (Oak)
• Chairs
• Kevin Dubray (kdubray_at_juniper.net)
• Al Morton (acmorton_at_att.com)
• If you are not subscribed to the BMWG mailing
  list and would like to be, put an asterisk () by
  your name when you sign the blue attendance
  sheet.  

2
BMWG Agenda

• Note-Taker(s), IPR, and Blue Sheets
• WG Status with some discussion (Chairs, 20
  min)
• IPsec Benchmarking Terms New Methods (Kaeo, 20
  min)
• IGP-Dataplane Convergence I-Ds. (Poretsky, 15
  min)
• Core Router Accelerated Life Testing I-Ds
  (Poretsky, 15 min)
• Milestones Status (Chairs, 5 min)
• Revised Charter Discussion (Chairs, 15 min)
• New Work (Chairs, Poretsky, 15 min)

3
BMWG Activity

• AD/IESG Review
• this space is blank for now
• WG Last Call
• ltdraft-ietf-bmwg-benchres-term-06.txtgt, WGLC
  ended 10/10
• ltdraft-ietf-bmwg-igp-dataplane-conv-term-08.txtgt
  LC ended 10/26
• ltdraft-ietf-bmwg-igp-dataplane-conv-meth-08.txtgt
• ltdraft-ietf-bmwg-igp-dataplane-conv-app-08.txtgt
• ltdraft-ietf-bmwg-dsmterm-11.txtgt, begin WGC
  shepherding
• ltdraft-ietf-bmwg-ipsec-term-07.txtgt will proceed
  with Meth
• I-Ds
• ltdraft-ietf-bmwg-ipsec-meth-00.txtgt new
• ltdraft-ietf-bmwg-hash-stuffing-04.txtgt WGLC
  requested.
• ltdraft-ietf-bmwg-acc-bench-term-07.txtgt Ready for
  WGLC?
• ltdraft-ietf-bmwg-acc-bench-meth-04.txtgt Ready for
  WGLC?

4
BMWG Activity

• I-Ds
• ltdraft-ietf-bmwg-acc-bench-meth-ebgp-00.txtgt,
• ltdraft-ietf-bmwg-acc-bench-meth-opsec-00.txtgt,
• ltdraft-ietf-bmwg-bgpcmeth-00.txtgt expected soon?

5
BMWG Activity

• Expired BMWG I-Ds
• ltdraft-ietf-bmwg-fib-meth-03.txtgt, No Prog
  Dead
• ltdraft-ietf-bmwg-benchres-method-00.txtgt Pending
  term
• New Work proposals.
• LDP Convergence expired, but
• draft-poretsky-protection-term-00.txt (adopt?)
• draft-poretsky-mpls-protection-meth-04.txt
  (adopt?)
• RFC Editor none.
• Charter Update
• Need more discussion.
• New RFC

6
Revised Milestones

• Fmr Past Due Milestones
• Dec 05 IPsec Device Benchmarking Terminology to
  AD Review
• Apr 06 Net Traffic Control Benchmarking
  Terminology to AD Review
• Apr 06 Methodology for FIB related Router Perf.
  Bench. to AD review.
• Apr 06 IGP/Data-Plane Terminology I-D to AD
  Review
• Apr 06 IGP/Data-Plane Methodology and
  Considerations I-Ds to AD Review
• Apr 06 Router Accelerated Test Terminology I-D
  to AD Review
• Jul 06 Basic BGP Convergence Benchmarking
  Methodology to AD Review.
• On Track, but running late
• Feb 06 Router Accelerated Test Method. and
  Consid. I-Ds to AD Review
• Feb 06 Hash and Stuffing I-D to AD Review
• Dec 05 Net Traffic Control Benchmarking
  Methodology to AD Review (no change)
• New
• Apr 06 Router Accelerated Test Method for EBGP
  to AD Review
• Apr 06 Router Accelerated Test Method for
  Operational Security to AD Review
• Dec 05 IPsec Device Benchmarking Methodology to
  AD Review

7
Charter Text (Goal/Scope)

• Current goal is very broad
• to make a series of recommendations concerning
  the measurement of the performance
  characteristics of various internetworking
  technologies
• further, these recommendations may focus on the
  systems or services that are built from these
  technologies.

8
Charter Text (continued)

• Should we tighten this coverage?
• to make a series of recommendations concerning
  the measurement of the performance
  characteristics of various internetworking
  technologies, primarily those developed by the
  IETF
• Could also say preferably or exclusively or
  ??

9
What BMWG benchmarks are not, or cannot be

• (much discussion last time)
• Conformance tests
• IETF consensus avoids legal entanglements (?)
• Benchmarking methodology assumes compliance, and
  the procedures should make it difficult to cheat.
• Reliability tests?
• How well do these tests fall under our benchmark
  umbrella?
• Acceptance tests
• Implication of success criteria must be avoided
• Operational Network tests (not controversial)
• Division of Labor between other organizations.

10
Backup
11
Charter Text

• Central Tenet
• Notion of a Benchmark
• BMWG Benchmark
• A prescribed measurement thats a basis for
  performance evaluation AND comparison.

12
Benchmark Required Attributes?

• Relevant and meaningful.
• Unambiguous specification of an attribute (or
  attributes) being examined, its collection and
  reporting.
• Universal applicability within class. (No bias
  toward or against a specific implementation.)
• Repeatability.