BGP Convergence Measurement Issues

1
BGP ConvergenceMeasurement Issues

• Susan Hares, NextHop
• Padma Krishnaswamy, NextHop
• Marianne Lepp, Juniper Networks
• Alvaro Retana, Cisco
• Howard Berkowitz, Gett Communications
• Elwyn Davies, Nortel Networks

2
Convergence?
Flapping
AS
AS
He's Dead, Jim
AS
AS
AS
AS
3
Convergence Within an AS
Single AS
R
R
R
4
Convergence Within a Box
Single Box
Tester
Tester
Routing/Control
Forwarding
5
Convergence for BMWG

• Box wide
• eBGP initially
• Control Plane initially
• Black box
• Specify begin and end of convergence measurement
• Specify measurement point

6
Send a packet stream from TR 3 Measurements

• Convergence 1 1st packet sent from Test
  Generator to 1st packet received by Test
  Collector
• Transmission in and out plus process time of 1st
  packet
• Convergence 2 Last Packet sent from Test
  Generator to last packet received by Test
  Collector
• Transmission in, queuing, processing of preceding
  updates, tail end processing, transmission out of
  last packet
• Convergence 3 1st packet sent from Test
  Generator to last packet received by Test
  Collector
• Transmission in and out (relative to DUT), plus
  back-up in BGP update and processing of entire
  stream

7
Measurement 1-3 Factors

• Packing matters
• Influences number of packets in the train
• Attribute packing
• Classification speed
• Packetization triggers
• IBGP synchronization turned off
• Turn off Minimum Route Advertisement Interval
  Timers
• Smoothing in BGP to avoid self-synchronization in
  the Network

8
BGP Convergence Depends On

• Route mixtures
• Packet packing
• Timers
• TCP implementations
• Peers types, number of peers, and connectivity
• BGP-specific functionality
• Eg. Confederations, use of route reflectors, etc.
• Topology
• Vantage point within the network
• Policy

9
Benchmarking Convergence Approach

• Must be repeatable
• Must be consistent
• Must be specifiable
• Must take into account
• Route mixture (data)
• Peers types and connectivity
• BGP-specific functionality
• Topology

10
Goals

• Provide a baseline of expected performance in
  todays network.
• Test different vendor implementations fairly
• Design tests that can be replicated
• Good results require good data
• The amount, type and composition of the
  information advertised to the DUT has an impact
  on the convergence.

11
Route Mixtures
12
Modeling Route Mixtures Why not just use a
feed?

• The route mixture is highly dependent on the
  vantage point Tier 1 ISP, Enterprise, Access,
  etc.
• Problems with Looking Glass
• Vantage point
• Need to test tables larger than current live
  tables
• Needs to be repeatable, consistent, and
  specifiable

13
Route Mixture

• Factors that describe the BGP Table composition
  and timing
• Prefix distribution
• Node distribution and levels on tree
• AS mixtures and path lengths
• Attribute distribution (nexthop, communities,MED,
  localpref)
• Packet packing
• Update sequencing (timing)
• Packet trains

14
Prefix Distribution

• Example A table of all /32s is not
  representative of the real world
• Prefixes are distributed across dozens of prefix
  lengths
• For IPv4, the distribution is spread out through
  the Class A, B, and C address spaces.
• For IPv6, there is no data
• Need to describe prefix distribution per prefix
  length
• Better characterization for IPv4 if Class also
  taken into account
• Analyze current Internet table to determine
  prefix distribution characteristics

15
Prefix Distribution

• Example percentages of prefix distribution
•    Mask   Overall   Class A   Class B   Class C
•      16   0.08114   0.00076   0.06637  
  0.01401     17   0.00912   0.00030   0.00142  
  0.00741     18   0.01813   0.00093   0.00113  
  0.01607     19   0.05910   0.00378   0.00196  
  0.05336     20   0.03372   0.00152   0.00151  
  0.03070     21   0.04128   0.00085   0.00127  
  0.03915     22   0.05574   0.00171   0.00226  
  0.05176     23   0.07878   0.00235   0.00450  
  0.07193     24   0.53355   0.00892   0.02366  
  0.50097

Total prefix length distribution.
IPv4 sample distribution across classes.
16
IP v6 Prefix Distribution

• Example percentages of prefix distribution
•    Mask   Overall   3FEE 2001 other
•   0-10    0.08114   0.00076   0.06637  
  0.01401  11-20   0.00912   0.00030   0.00142  
  0.00741  21-30 0.01813   0.00093   0.00113  
  0.01607  31-40   0.05910   0.00378   0.00196  
  0.05336  41-50   0.03372   0.00152   0.00151  
  0.03070  51-60   0.04128   0.00085   0.00127  
  0.03915  61-70   0.05574   0.00171   0.00226  
  0.05176  71-80   0.07878   0.00235   0.00450  
  0.07193  81-90  0.53355   0.00892   0.02366  
  0.50097
• 91-100  0.53355   0.00892   0.02366   0.50097
• 100-110 0.53355   0.00892   0.02366   0.50097
• 111-128 0.53355   0.00892   0.02366   0.50097

IPv6 sample distribution across currently
routed Addres space
Total prefix length distribution.
17
Node Distribution

• Is tree dependent
• Width and depth of table are important
• Route mixtures should exercise various choices of
  trees
• A route mixture that minimizes the number of
  nodes is not accurate
• A route mixture that maximizes the spread of
  prefixes creates is not accurate

18
Node Distribution
Levels
Nodes
19
IP v6 Node Distribution
Levels
ROOT
3FEE
2001
200101
200102
3FEE0100
3FEE2000
3FEE0101
3FEE0101
20010201
2001020102
2001020101
3FEE010101
Nodes
20
Node Distribution

• For example, the following tables both contain
  three Class A /32 prefixes
• Table A 1.1.1.1/32, 1.1.1.2/32, 1.1.1.3/32
• Table B 1.1.1.1/32, 2.1.1.1/32, 3.1.1.1/32
• Their distribution in a tree will be different.
• Table A represents a narrow distribution, while
  Table B represents a wide distribution.

Table B
Table A
21
Node Distribution Summary

• The width of the table must be measured per
  prefix distribution and length
• Need to determine how many nodes each
  address/prefix length combination use in a real
  table
• Solution Analyze current Internet table to
  determine node distribution characteristics

22
Route Components
23
BGP Attribute Distribution

• A BGP table contains many attribute
  combinations
• Analysis shows
• 11.75 of the routes have a unique AS_PATH
• 2.5 of the routes have some other unique
  attribute. 
• 0.25 of the table have both a unique AS_PATH and
  some other unique attribute

24
BGP Attribute Distribution

• Prefixes that share an attribute are not
  necessarily grouped together
• Analysis shows an average of two consecutive NLRI
  share the same attribute combination
• 1.0.0.0/8 AS_PATH 100 200
• 2.0.0.0/8 AS_PATH 100 200
• 3.0.0.0/8 AS_PATH 200 300
• 4.0.0.0/8 AS_PATH 200 300
• 5.0.0.0/8 AS_PATH 200 300
• 6.0.0.0/8 AS_PATH 100 200

25
Planes (control),Trains,and no Automobiles
26
Packet Packing

• Each packet has attributes and NLRIs
• Attribute packing is the ability to detect and
  pack NRLIs with the same attributes into a packet
• NLRI packing is
• the number of NLRIs per packet
• MPBGP not considered for 1st draft
• IPv6 packing is not different than IPv4
• Multicast packing (IP v4 and IP v6) may impact
  packing
• Specifics are affected by implementation

27
Update Sequencing (Timing)

• Parameters are
• Number of packets in a train
• Interval between packets in a train
• TCP parameters, traffic and implementations
  affect this

Packet 1
Packet 2
Packet 3
Packet 4
Packet train
Packet train
28
Timers

• Key timers
• Min-Route Advertisement Interval, Min AS
  Originations Interval --best setting still in
  debate
• Route Flap damping mechanisms
• Implementations vary
• Shorter prefixes get less damping
• RIPE 229 suggest parameters
• 1st Bgp Conv draft mandates route flap damping
  off
• TCP settings
• Operators need to give feedback

29
Peers, not Beers
30
Peer type matters

• EBGP vs IBGP
• EBGP
• 3rd party versus 1st party nexthop
• promiscuous versus specific peering
• IBGP - Route Reflection client and Confederations
  affect convergence patterns
• See ietf-idr-route-oscillations-01.txt
• Still single box but these affect work done by
  box

31
Multiple Peers in test Environment

• Peers can have staggered starts
• Most realistic
• Peers can all send simultaneously
• Most load on the router
• Peers can have staggered starts in groups

32
Sample topology with 4 Peers
tcpdump
tcpdump
TG1
DUT
TC
TG2
tcpdump
TG3
tcpdump
TG4
tcpdump
33
Peer Specifics

• Type of Peer
• Promiscuous/Specific
• Sequence
• Connection establishment
• Sending 1st data
• Spacing of updates
• Connection up/down

34
Timing Synchronization

• Consistency among timestamps taken by different
  devices is a requirement
• Should be at least 1 order of magnitude better
  than measured quantity
• For BGP convergence, we are time-stamping packets
• NTP? GPS? Other?
• Synchronization between measurements can a
  significant factor

35
Some Boxes workHarder than Others
36
BGP Protocol functions will impact convergence

• Route Reflections,
• Confederations
• Add/delete communities
• RFC 2547, Label switching
• Multi-protocol
• Route flap damping
• Min Route Advertisement

37
Parameters we suggest for Protocol Functions for
1st Document

• No Route Reflectors (no IBGP this version)
• No confederations
• No Add/Delete communities
• No 2547 VPNS or multicast
• Route flap damping OFF
• Min Route Advertisement Interval specified
• Min AS Origination Interval specified

38
Topology
39
Topology matters

• Exchange point topology
• N star topologies meshed for Route Reflection
• Confederations with particular topologies
• IBGP/EBGP mesh overlay
• Building blocks
• single link, line, mesh, partial mesh, star, wheel

40
Single link 1st Document
TR1
TRes
DUT
TR2
line
TRn
n gt 1
41
Line
DUT
TR1
DUT
DUT
Longer line
TRes
42
Mesh
DUT
DUT
TRes
TR1
DUT
DUT
mesh
TRes
TRes
43
Partial Mesh
DUT
DUT
TRes
TR1
DUT
DUT
TRes
TRes
44
References

• IETF51 BMWG talk http//www.ietf.org/proceedings/
  01aug/slides/bmwg-4/
• NextHop IETF51 talk http//www.ietf.org/proceedin
  gs/01aug/slides/bmwg-5/index.html
• Howards IETF51 talk http//www.ietf.org/proceedi
  ngs/01aug/slides/bmwg-6/index.html
• Recommendations for flap damping, Ripe 229
  http//www.ripe.net/ripe/docs/ripe-229.html
• BGP Convergence Terminology ID
  http//www.ietf.org/internet-drafts/draft-ietf-bmw
  g-conterm-00.txt
• BGP Convergence Methodology http//www.ietf.org/i
  nternet-drafts/draft-ietf-bmwg-bgpbas-00.txt

45
Thank You

• Questions?

46
Route Mixtures Matter!

• The amount, type and composition of the
  information advertised to the DUT has an impact
  on the convergence.
• Goal is to provide a baseline of expected
  performance in todays network.
• Test different vendor implementations fairly
• Design tests that can be replicated
• Good results require good data