Miami NANOG Presentation

1
Operational Experience with
MPLS
Dave Siegel Vice President IP Engineering
2
Table of Contents

• Overview of hub architecture
• History of Network architecture
• Early challenges and how MPLS solved them
• Challenges with MPLS today
• VPN Deployment
• Architecture
• Capabilities (RFC2547, l2VPN)
• Provisioning aspects
• Customer experiences
• IPv6 deployment w/ MPLS
• How the network would behave without MPLS

3
WR1
WR2
OC-48/OC-192 OC-12/OC-48 OC-3/OC-12
CR1
CR2
AR1
AR2
PR1
RR1
Modems
ADMs
Ethernet Switches
GBLX
4
(No Transcript)
5
(No Transcript)
6
(No Transcript)
7
(No Transcript)
8
(No Transcript)
9
(No Transcript)
10
(No Transcript)
11
(No Transcript)
12
Early Challenges

• Hop Count
• Network diameter ranged from 14-18 hops
• GRE tunnels were not supported on GSR images
• Traffic Engineering
• Large numbers of DS3s and OC-3cs in metro
  regions proved difficult to manage with IS-IS
  metrics
• Future VPN Product

13
MPLS Solutions

• Hop Count
• MPLS had no-decrement-ttl option
• MPLS tunnels were in implementation phase for GSR
  images
• Traffic Engineering
• MPLS provided for much more efficient utilization
  of metro bandwidth

14
MPLS Solutions

• Hop Count
• Established Cross-country tunnels to mask main
  hops normally encountered in the core
• Traffic Engineering
• Established regional meshes of LSPs between
  devices

15
Multi-vendor networks

• Theory having multiple suppliers gives you
  best-of-breed, plus contingency plans if you have
  major problems with your primary supplier
• Reality Once a vendor is entrenched in your
  network, replacing them completely is simply too
  capital intensive
• Reality you have worst-of-breed, because you
  must wait for both of your vendors to have a
  compatible implementation of a feature before
  deployment.

16
Multi-vendor networks

• Early interoperability issues (circa 1999-2000)
• Penultimate hop (NULL label vs. strip)
• No-decrement-ttl issues (its a one-hop network!)
• Current Issues
• Fast Re-route
• Secondary LSP
• Auto-bw

17
Current Stats

• MPLS core LSP mesh
• 9900 tunnels make up the core mesh (100 core
  routers)
• 1200 tunnels between PRs that make up the IP-VPN
  Express route Product
• 11,100 tunnels total in the core
• Complexity requires automated management tools

18
MPLSrobot

• Bot components
• High-speed snmp poller
• Tunnel resize script w/ tons of knobs
• Graphing capability
• Path database
• Configuration push scripts
• Day-to-day challenges involve conditioning of
  collected data
• Run daily but configs pushed weekly

19
WANDL
20
Getting Started

• Remove roadblocks
• Look for features of your network design that
  increase complexity or introduce roadblocks to
  implementing MPLS
• Multiple ASs
• Multiple levels/areas in your IGP
• Lack TE support in your IGP

21
Getting Started

• Choose reasonable RSVP bandwidth
• Set bandwidth values on new tunnels to 0 Mbps,
  and then measure over 24 hours.
• Set tunnel bandwidth to observed peak some
  fudge factor (e.g. 95th tile peak 10)
• Do tunnel implementations slowly over timedont
  introduce too much churn in the network
• Tune link utilization with RSVP bandwidth values
  during transition

22
Follow our Roadmap!

• Q4 1998 MPLS lab trials begin
• Q1 1999 MPLS limited production trial begins
  (regional mesh ttl masking hack)
• Q2 1999 national LSP mesh between all CRs
  complete
• Q2 2000 global LSP mesh complete
• Q2 2001 RFC 2547 IP VPNs and L2-VPNs with
  DiffServ (2 CoSs)

23
Operational issues uncovered

• Through 1999, MPLS was blamed for a variety of
  outages and/or performance degradation issues,
  including
• High latency
• Loss
• Reachability
• Workarounds included bouncing LSPs
• Most of the time, CEF bugs were to blame!

24
Operational issues uncovered

• Except when WRED was to blame

25
Operational issues uncovered

• Training, Training, Training
• Cannot be underestimated
• Experience, Experience, Experience
• GX had 2 full years of experience with MPLS
  operationally before adding MPLS-based VPNs to
  the network

26
IP-VPN (ExpressRoute) architecture

• Objective is to provide as much isolation from
  the Internet as possible
• Separate ASN (not AS3549)
• Private Routers (PRs) not reachable from outside
  gblx.net (non-advertised address space)
• Full mesh of LSPs between all PRs
• Full iBGP mesh among all PRs

27
IP-VPN (ExpressRoute) architecture

• Secondary Objective is to provide as high a class
  of service as possible.
• LSPs have higher priority than LSPs for
  Internet Service so they always get the best
  (lowest latency) routes.
• ToS Bits are painted into a Business Class (vs.
  Best Effort for Internet Service) which is
  re-written into the EXP field

28
IP-VPN Customers

• Connected Even mix of RFC 2547 and l2-vpn
• Sales Funnel Majority (60) want L3-VPN
• Sales Funnel good mix between carrier and
  enterprise
• Largest customer is RFC 2547 with approximately
  50 circuits
• Market interest is still gaining momentum for
  this product set (ISP provided IP-VPN)

29
IP-VPN Provisioning pros/cons

• L2-VPNs are the easiest to engineer for the
  customer, but adds/deletes/moves require updating
  configs on every PR where the customer is
  connected.
• L3-VPNs require less configuration on the ISP
  side, but are preferred less due to the high
  level of CPE engineering coordination required.
• L3-VPNs were designed as a complete out-source
  of a customers routing, but in reality customers
  use this service in conjunction with another VPN

30
IPv6 deployment using MPLS

• GX has 3 routers located at native IPv6 exchange
  points
• Sure, you could use GRE tunnels to interconnect
  them over IPv4, but MPLS gives you
• Per tunnel utilization statistics
• Path info
• Scalability (as the product grows, you can add
  devices as an overlay network without impacting
  stability on existing platforms)

31
How the network would behave without MPLS

• WANDL simulations show that there would be no
  congestion in the network based on IGP TE with
  IS-IS, so MPLS is not needed today for TE.
• Bandwidth reservations for MPLS-based VPNs would
  not be as meaningful with large amounts of native
  IP traffic on backbone trunks.

32
Questions
33
THANK YOU
Additional questions may be
sent to dsiegel_at_gblx.net