Use of BGP and MPLS VPNs: A Case Study

1
Use of BGP and MPLS VPNs A Case Study

• Fred P. Baker
• CCIE3555

2
Contents

• Current Network
• The MPLS VPN project
• Routing Objectives
• What we did
• How we tested

3
Current Network
4
Current Environment

• Hub and spoke to 4 data centers
• Sites do not in general connect to 2 data centers
  due to cost and OSPF issues
• Generally place servers by geography
• You servers are in the data center your links are
  in
• Mostly Frame Relay to ATM interworking with some
  private lines
• 70 of some 350 remote sites have 2 links
• ATM PVC dual mesh between the data centers
• 12000 agent location network done by MCI with
  combination of DSL and Fractional T1

5
Address Space

• 10.0.0.0/8
• Mostly inside
• Some BP
• 192.168.0.0/16
• Used all over
• 172.16.0.0/12
• Extranet
• 167.127.0.0/16
• Public address space
• Used mostly by extranet
• Some legacy inside

6
Core

• ATM PVCs
• 2 10meg between each pair of data centers
• 2 routers on the core
• So 2 meshes

7
Allstate Core
8
10.0.0.0 address allocation/11 for core 1 per
data center
9
Allstate Data Center
10
Routing Protocol

• Single OSPF AS
• Cisco and OS/390 based routers only
• Firewalls now static routed
• Peer authentication soon

11
Remote sites

• ATT frame relay at the site
• ATM into the data center
• Some ISDN backup
• A remote site is connected to a single data
  center (for now)
• Servers and applications tend to have geographic
  affinity

12
Remote Site
13
Remote Site Switch Layer
14
Agent Broadband

• 10,000 locations
• Connected via IPSEC VPN
• WorldCom managed routers
• NO split tunneling
• IPSec Transport with GRE tunnel to Dallas and
  Hudson
• Agent PCs are 10...
• Agent access is via Allstate Internet Proxy

15
Overview
16
Agent Broadband in Data Center
17
Agent office
18
Internet/Extranet

• We do not use the default route
• There are 3 data center with ISP connections
• We code static routes to the firewalls (we dont
  trust firewalls running dynamic routing
  protocols) and redist to OSPF

19
The project
20
The project

• We use a single data network provider
• This is a single point of failure of that
  providers ATM/Frame networks
• Add a second data provider
• Initially to use for the dual attached sites
• Then convert 1 of the core ATM meshes to the
  second provider

21
Layer 2 vs Layer 3 provider

• Frame Relay is layer 2 connectivity
• The routers have a direct peering relationship
• Many providers are offering Layer 3
• Costs are the same or even less
• MPLS VPN is the data transport
• Many providers are using MPLS to move even layer
  2 networks
• You have a routing relationships with the
  provider not with yourself
• So More complex to configure and fix
• Not a simple OSPF network anymore

22
Which one we picked

• Layer 3
• DR becomes free do not need to run more PVCs to a
  DR data center
• The data center placement of servers assumption
  is changing
• Apps are being put to 1 DC
• Also there is more site to site traffic than we
  expect
• So we can reduce traffic on the ATM core
• And increase response time
• Do dual homed sites first convert 1 link to L3
• Single homed late

23
MPLS VPN
24
Route types

• CE customer Edge
• your router
• run BGP to provider
• Knows nothing about other customers or provider
  routes
• PE provider Edge
• Knows about all local customer VPNS
• Has multiple routing tables
• P providers
• Transport only
• No customer routes

25
Routing objectives

• Support load share from the home DC
• Remote site goes direct to non home DC over L3
• Remote site directly to remote site
• Reduce transit of the core
• Support a L3 provider in the core replacing 1 ATM
  mesh
• Do not use remote sites to transit traffic

26
Technical Objectives

• Limit the number of bgp attributes used
• Keep the remote site configuration simple
• Do not inject the default route unless you must
• How to inject the Internet routes

27
Routing protocol design
28
Dont forget the 3 rules of routing

• Longest subnet mask
• Lowest distance
• Best metric

29
BGP features we used

• As path
• Path length filters
• No export
• Backdoor
• If AS Paths are equal then router uses eBGP route

30
How to route

• Must look at the routes going BOTH ways
• Routes to
• Routes from
• The routes you advertise drags traffic to you
• The routes you take in is how you route back
• We load share by having each router use a
  different path, then send equal cost into IGP

31
Result

• Use MPLS VPN based L3 provider
• Remote sites 2nd link to L3
• Each data center connects to L3
• Will not use L3 to route between DCs due to QoS
  concerns

32
Routing

• Use BGP at remote sites
• Can use OSPF with SOME providers but not all
• BGP works much better
• Each site is 1 AS
• EACH data center is 1 AS
• This allows us to put an L3 provider in later
• BGP routes BETWEEN ASes
• Address ASes from private space
• This is ok because provider is a VPN

33
Route injection to/from BGP

• Allstate Data Center
• Explicit network statements to BGP
• Redist BGP to OSPF
• Remote site routes
• Redist from OSPF
• Decided that using network statements to complex
• BGP routers send just default route to any
  switches
• We will accept the extra LAN transit
• Internet routes
• Redist static

34
Internet routes

• There will be non BGP L3 switches between Inet
  and allstate core
• Redist static into OSPF already
• So just redist into BGP also
• Put internet router in same AS as datacenter
  (have to as no direct path)
• Use sync
• Send to L3 provider and to sites over L3

35
BGP to L3 provider (and then remote sites

• Data center side
• Send data center /11s
• Send internet routes
• Take routes from L3 provider
• Do not forward other eBGP learned routes
• Remote site side
• Send all local routes
• do not forward other learned eBGP routes
• Remember the no export to kill transit
• Receive all routes
• Want to take L3 when I can

36
DC to Remote site FR

• Send all bgp derived routes
• Do as prepend of the data center AS
• This makes AS path 2 for DC on FR and L3 paths
• This makes AS Path3 for DC to DC via ATM core so
  site to remote DC traffic over L3

37
Remote site to DC on FR

• Do as prepend of 1 AS at remote end
• Need this so FR and L3 paths have AS Path2 so we
  load share
• Filter routes with AS Path gt1
• I only want to send the local site routes up the
  FR link
• Do not want DC to send transit traffic to site

38
IBGP in the remote site

• Set next hop self
• Routers must have a shared Enet
• No redist of BGP to OSPF
• So cant use sync so cant transit a L3 switch
• Do not forward routes I learn via FR
• Do not want a transit from L3 up the FR link
• Do not want a transit to L3 from FR link
• Set no export attribute on routes from DC over
  the FR link
• This prevents site from passing them to L3
• Cannot AS path filter on IBGP because I want to
  pass the DC route via iBGP
• Why I use no export

39
Results
40
DC to DC

• Each site learns over ATM network with AS Path
  1
• Cannot route over L3 provider

41
Remote site to non home dc

• Non home DC sent via L3 AS Path 2
• Home data sends via FR AS Path 3 due to prepend
• Use if L3 down

42
non home dc to remote site

• Non Home DC learns remote site routes from L3
• Home data center sends only the /11 summary
• so longest match says L3

43
home dc to remote site

• Load share
• Routes from L3 have AS Path 2
• Routes from FR have AS Path 2 due to prepend
• So each router uses eBGP route

44
remote site to home dc

• Dont care as much about load share
• Routes from L3 have AS Path 2
• Routes from FR have AS Path 2 due to prepend
• So each router uses eBGP route

45
remote site to remote site

• Use L3 network
• Learn site specific routes directly from site
• Learn /11 summaries from DCs

46
Agent routes

• Only dual DC connected things that dont use BGP
• Many routes summarized as /19s
• I get these from MCI as OSPF externals
• Have not decided how to inject them
• They go to two data centers for redundancy
• So I need to send them via BGP
• So a router will get an OSPF external from the
  local MCI connection and the other data center
  via BGP
• eBGP lt OSPF so BOOM
• Use backdoor on core routers to set distance on
  the agent routes to gt than OSPF
• So if local MCI connection up use it, else
  transit core

47
Testing
48
Local Testing

• Use 7 routers
• 1 remote site OSPF route not shown
• Paths
• iBGP at remote
• L3
• FR to home DC
• Inter DC

49
CPOC

• Cisco Proof Of Concept
• In Raleigh and San Jose
• Lab use is free (if you are big enough)
• Send in specific test plan
• Your SE goes in a week ahead of time
• Lab is all setup when you arrive

50
Testing

• Test migrations
• Test routing
• based on our policies
• failovers
• Measure convergence
• Test a migration of a core ATM mesh to L3
• Get some data and experience on the MPLS side
• Try multicast over MPLS/VPN

51
CPOC Network Diagram
52
CPOC Learnings

• Inject all links both ATM core and L3 into BGP as
  they will source pings
• Turn sync off due to code defect
• You must explicitly code send community in iBGP
• If you reference a non-existent as-path statement
  NO ROUTES
• OSPF LSAs stay in the data base up to 90 minutes
  due to timer jitter
• This is a migration issue
• Do lots of clear routes/clear ip bgp in the
  migration
• Need to change the BGP timers as default
  convergence is 3 minutes
• iBGP only sends the best route

53
Going forward

• Already run BGP to some remote sites
• Migrate the core to bgp first
• Do a dress rehearsal
• Will be a big scary change so plan well
• Examine tools
• May not be able to assume we will get traps
• May have to watch the BGP tables for changes
• Get a test connection in place