Congestion Mitigation

1
Congestion Mitigation

• Trying to maximise performance between MFNs
  network and a peers network over some busy PNIs

2
Hello, Good Evening

• Joe Abley
• jabley_at_mfnx.net
• Toolmaker, Token Canadian
• Metromedia Fiber Network

3
Background

• There are frequent periods of congestion between
  our network and a peers network in Europe
• The peer is a major operator in the region, and
  evil forces are preventing us from simply
  lighting up new fibre
• We need to work with what we have in place already

4
Characteristics of Problem

• The peer is home to a lot of broadband
  subscribers
• Transient hot-spots in content hosted within
  MFNs network cause localised congestion in some
  PNIs, with other PNIs showing headroom
• We get paid for shifting packets (if we dont
  carry the packets, we dont get paid)

5
State of Play Day One
Location Capacity Utilisation
Amsterdam STM-1 75M
Frankfurt STM-1 140M
Vienna STM-1 90M
6
Goals

• Identify the important consumers of traffic in
  and beyond the peers network
• Once we can characterise the major traffic sinks,
  we can try and balance them out across our
  various PNIs
• Hopefully this will make the PNIs less sensitive
  to bursty traffic
• We expect to have to keep measuring and
  rebalancing

7
Tools

• Ixia IxTraffic
• Gumption
• Unix-based BGP speaker that participates in the
  IBGP mesh
• Gives us route history
• SeeFlow
• Smart NetFlow collector which talks to Gumption
• Awk
• We always have awk

8
Infrastructure

• NetFlow traffic to be collected in-band from
  GSR12012s
• Single IxTraffic box
• FreeBSD 4.5, i386, dual 700MHz P3, 2GB RAM
• Overkill
• Load average occasionally peaks above 0.07
• 10GB filesystem for storing routing and flow data
  in
• Located in Virginia
• MRTG-like thing (duck) which also lives in VA on
  a different box gives us nice visibility of
  congestion trends

9
Exporting Flow Data

• Flow switching needs to be turned on in a maint
  window, because it makes the routers belch
  impolitely
• All interfaces that can contribute towards
  traffic sent towards the peer get ip route-cache
  flow sampled
• See kiddie script! See router die!
• Export config is trivial
• ip flow-export source Loopback0
• ip flow-export version 5 peer-as
• ip flow-export destination 192.168.38.9 3871
• ip flow-sampling-mode packet-interval 1000
• Note low sampling rate of 11000.

10
Collecting Flow Data

• SeeFlow is configured to populate net2net and
  aspath matrices (buckets)
• We suspect that a lot of data is getting sunk
  within the peer network, hence net2net
• We could be wrong, and aspath matrices are cool,
  so we collect those too
• Buckets chew up about 50MB of disk per day (all
  routers)

11
Initial Discoveries

• All the traffic is being sunk within the peer
  network, and not in a downstream network
• Damn
• All the traffic is being sunk into a single /12
  advertisement
• Damn
• We need better granularity if we are going to be
  able to spread the demand across our PNIs

12
ASPATH Matrices

• jabley_at_nautilus seeasp -s 3320 ./dtag.agg
  more
• Facets
• TimeInterval 05/09/2002 150003.663492 -
  06/05/2002 045758.747866 PDT
• SuperDataFacet
• Facets
• RouterIpv4Addr 209.249.254.142
• RouterName pr1.fra1.de.mfnx.net
• Facets
• RouterIpv4Addr 209.249.254.195
• RouterName mpr2.vie3.at.mfnx.net
• Facets
• RouterIpv4Addr 216.200.254.246
• RouterName mpr1.ams1.nl.mfnx.net
• AS P PktsThru BytesThru PktsTo
  BytesTo PktsTotal BytesTotal
• ----- - ---------- ------------ ----------
  ------------ ---------- ------------
• 0 - 0 0 371.203M
  143.286G 371.203M 143.286G
• AAAA P 1.816M 333.567M 108.112M
  36.705G 109.928M 37.038G
• BBBB - 0 0 1.516M
  191.657M 1.516M 191.657M

13
Net2Net Matrices

• 172.184.0.0/13 -gt A.A.0.0/12
  13001989 5601247858
• 172.184.0.0/13 -gt A.A.0.0/12
  12983375 5592070913
• 62.4.67.0/24 -gt B.B.0.0/11
  9459634 1687041555
• 62.4.67.0/24 -gt B.B.0.0/11
  9443861 1677536483
• 172.176.0.0/14 -gt A.A.0.0/12
  7113026 2985029679
• 172.176.0.0/14 -gt A.A.0.0/12
  7099648 2977787074
• 62.80.115.0/24 -gt C.C.0.0/11
  6873518 1236318991
• 62.4.67.0/24 -gt A.A.0.0/12
  6689319 1180741686
• 62.4.82.0/24 -gt A.A.0.0/12
  6611879 1171430532
• 62.4.67.0/24 -gt C.C.0.0/11
  3469776 629221553
• 62.4.82.0/24 -gt C.C.0.0/11
  3433970 625422145
• 62.4.67.0/24 -gt D.0.0.0/13
  2422913 442942807
• 62.4.67.0/24 -gt D.0.0.0/13
  2407651 470778890
• 62.4.65.96/27 -gt A.A.0.0/12
  1981446 287218317
• 62.80.116.0/24 -gt E.E.0.0/15
  1802114 378062358
• 62.4.67.0/24 -gt F.F.0.0/14
  1510412 315282857
• 62.4.67.0/24 -gt F.F.0.0/14
  1421497 277014582
• 62.4.65.96/27 -gt B.B.0.0/11
  1341063 378931389
• 62.4.81.128/27 -gt B.B.0.0/11
  1330058 378268227

14
Destination Prefix Histogram

• destination net megabytes proportion
• A.A.0.0/12 21478 59
• B.B.0.0/11 6494 18
• C.C.0.0/11 4388 12
• D.0.0.0/13 1365 3
• F.F.0.0/14 1033 2
• G.G.0.0/15 416 1
• H.H.0.0/14 311 0
• I.I.I.0/21 160 0
• J.J.0.0/15 117 0
• K.K.0.0/19 89 0

15
Drilling down into A.A.0.0/12

• Ask peer to advertise longer prefixes within the
  /12, so we can measure the traffic per prefix
• Wait for response
• GOTO 10
• Maybe we can fix this ourselves

16
/home/dlr/bin/bgpd

• We injected 15 covered /16 prefixes into IBGP,
  with a NEXT_HOP that lay within the remaining /16
• All tagged no-export, to avoid messing with the
  peers public route policy
• Strictly local-use within AS6461

17
More Collection

• The increased granularity gives us better
  visibility into the traffic sinks within the peer
  network
• We will try to spread the traffic over the
  available PNIs so we can weather bursts of demand
  more effectively
• We will also continue to let the peer know what
  we are doing
• You never know, they may be listening

18
New Dest Prefix Histogram

• destination net megabytes proportion
• B.B.0.0/11 1912 14
• A.76.0.0/16 1530 11
• C.C.0.0/11 1516 11
• A.73.0.0/16 1120 8
• A.72.0.0/16 1024 7
• A.64.0.0/12 874 6
• A.74.0.0/16 683 5
• A.70.0.0/16 696 5
• A.68.0.0/16 601 4
• A.66.0.0/16 437 3

19
State of Play Day N
Location Capacity Utilisation
Amsterdam STM-1 110M
Frankfurt STM-1 105M
Vienna STM-1 85M
20
Conclusions

• Light more fibre is not always a realistic
  strategy
• You are not always your peers number one
  priority, so its nice to be able to take matters
  into your own hands
• Distributing the heavy traffic sinks across
  different PNIs makes bursty demand less
  unpleasant
• Routing plus flow data Power. Or something.