How to Secure Your Job: Implement MPLS VPNs

1
How to Secure Your Job Implement MPLS VPNs

• roger_at_gottsponer.ch
• NANOG24 / Miami, Florida
• February 10-12, 2002

2
Agenda

• The Idea behind this Talk
• Lessons Learned
• The Service Architecture
• The Use of Route Reflectors
• Baby Jumbo Frames
• The LDP-ID
• Traceroute is no longer Your Friend
• Some Security Considerations
• no Religious Wars please -)

3
Agenda

• The Idea behind this Talk
• Lessons Learned
• The Service Architecture
• The Use of Route Reflectors
• Baby Jumbo Frames
• The LDP-ID
• Traceroute is no longer Your Friend
• Some Security Considerations
• no Religious Wars please -)

4
The Idea behind this Talk

• This talk should show some lessons we learned
  whenimplementing and running a real life MPLS
  VPN network
• Nextra Switzerland is running a national MPLS
  VPNnetwork since 2.5 years (Q3 1999)
• about 30 PoPs, Cisco based
• The Nextra Group is running a pan-European MPLS
  VPNnetwork since about one year (Q1 2001)
• present in most European countries
• provides seamless VPN services between autonomous
  Nextra companies
• I will not discuss whether MPLS is a good thing
  or not -)

5
Agenda

• The Idea behind this Talk
• Lessons Learned
• The Service Architecture
• The Use of Route Reflectors
• Baby Jumbo Frames
• The LDP-ID
• Traceroute is no longer Your Friend
• Some Security Considerations
• no Religious Wars please -)

6
Lessons Learned

• A reload a day
• keeps the TAC away
• just kidding

7
Agenda

• The Idea behind this Talk
• Lessons Learned
• The Service Architecture
• The Use of Route Reflectors
• Baby Jumbo Frames
• The LDP-ID
• Traceroute is no longer Your Friend
• Some Security Considerations
• no Religious Wars please -)

8
Service Architecture

• Is entirely based on MPLS VPN Routing Realms
• Each Added Value Service has its own Routing
  Realm/VPN
• Each Customer has several Routing Realms/VPNs
• Customers are given access to Services by
  interconnecting Customer VPNs and Service
  VPNs
• Limited Set of Building Blocks
• Full Mesh Routing Realm
• Hub/Spoke Realm
• (Point-Point Realm)
• Inter-Realm Conduits
• Inter-Realm Adapters

9
Lego-Brick 1 Full-Mesh VPN
10
Lego-Brick 2 Hub Spoke VPN
Hub
Spoke
Spoke
Spoke
11
Lego-Brick 3 Conduits
X Mbps
12
Lego-Brick 4 Adapters
13
The Four Network Building Blocks
14
Multiple VPNs per Customer
15
Managed Firewall IAS for PN
Nextra Managed Firewall
16
Standard VPN IAS
Customer Managed Firewall
17
Redundant Load Sharing PN
HSRP
RedundantAccess
POP
18
Disaster Recovery VPN
Backup Computing Center (IP 10.5.5.0/24)
Primary Computing Center (IP 10.5.5.0/24)
Fast (?) Re-routing
POP A
POP B
19
PN Dial Backup
VPDN Home Gateway
20
PartnerNet
PartnerNet
CustomerA
CustomerB
21
Agenda

• The Idea behind this Talk
• Lessons Learned
• The Service Architecture
• The Use of Route Reflectors
• Baby Jumbo Frames
• The LDP-ID
• Traceroute is no longer Your Friend
• Some Security Considerations
• no Religious Wars please -)

22
The Use of Route-Reflectors

• You do not want to maintain a full VPNv4-iBGP
  mesh
• Use at least two VPNv4-RRs for redundancy reasons
• Do not use the IPv4-RRs as VPNv4 Route-Reflectors
• IPv4 Routing instabilities could affect VPN
  routing
• If PE reloads it will get all 100000 IPv4 routes
  first. So it will take several minutes until the
  VPN routes are restored
• If RRs are separated, both feeds are done
  simultaneously
• So you need at least four RRs --gt what about core
  routers?
• Use global-local-massive-extreme unique RDs
• A different RD per VRF, not per VPN
• otherwise RR will break iBGP load-sharing
• Check next-hops of BGP routes you get from the RR
  -)

23
Agenda

• The Idea behind this Talk
• Lessons Learned
• The Service Architecture
• The Use of Route Reflectors
• Baby Jumbo Frames
• The LDP-ID
• Traceroute is no longer Your Friend
• Some Security Considerations
• no Religious Wars please -)

24
Baby Jumbo Frames

• MPLS labels are 4 Bytes long
• Available MTU on a link gets decreased by x4
  Bytes
• Due to Path-MTU-Discovery, the sender will detect
  this
• Path-MTU-Discovery seems to be broken sometimes
• Customer will blame you, not the broken PMTUD
• everything worked fine before we moved to your
  MPLS network!
• Workaround
• allow 1500(x4) Bytes packets over Ethernet
  (Baby Jumbo Frames)
• Special config command tag-switching mtu 1508
• oversized packets only allowed if oversize is
  caused by MPLS labels
• Before you migrate to MPLS, make sure all your
  routers, interfaces and switches support Baby
  Jumbo Frames!

25
Agenda

• The Idea behind this Talk
• Lessons Learned
• The Service Architecture
• The Use of Route Reflectors
• Baby Jumbo Frames
• The LDP-ID
• Traceroute is no longer Your Friend
• Some Security Considerations
• no Religious Wars please -)

26
The LDP-ID

• After a crash, a P router stopped forwarding any
  traffic...
• interface loopback99description test test
  testip address 222.222.222.222 255.255.255.255
• Like in OSPF, LDP uses highest loopback address
  as ID
• Unlike in OSPF, LDP-ID has to be reachable
• LDP-ID not reachable ? no LDP TCP-session ? no
  labels!
• So always configure the LDP-ID manually!

27
Agenda

• The Idea behind this Talk
• Lessons Learned
• The Service Architecture
• The Use of Route Reflectors
• Baby Jumbo Frames
• The LDP-ID
• Traceroute is no longer Your Friend
• Some Security Considerations
• no Religious Wars please -)

28
Traceroute is no longer Your Friend
29
A Multi ISP Environment
Trans it ISP
ISP B
ISP A
30
Customer Complains about Strange Traceroute

• CPE-Routertraceroute www.cisco.com
• 1 accesslink.company.com (10.150.73.17) 0 msec 4
  msec 0 msec lt-- PE (Europe)
• 2 r16b01-pos0-0-0.bb.nextra.net (141.22.5.2) 104
  msec 104 msec 100 msec lt-- P (Europe)
• 3 r08b02-pos1-0-0. bb.nextra.net (141.22.1.6) 104
  msec 104 msec 104 msec lt-- P (Europe)
• 4 r08b01-fe4-1-0. bb.nextra.net (141.22.5.49) 148
  msec 200 msec 200 msec lt-- P (Europe)
• 5 r10b01-pos4-0-0. bb.nextra.net (141.22.5.34)
  160 msec 200 msec 200 msec lt-- PE (USA)
• 6 Pos4-0.GW8.NYC4.ALTER.NET (157.130.21.37) 100
  msec 100 msec 100 msec
• 7 172.ATM2-0.XR2.NYC4.ALTER.NET (146.188.180.62)
  104 msec 104 msec 104 msec
• 8 188.at-1-0-0.TR2.NYC8.ALTER.NET (152.63.21.106)
  108 msec 104 msec 104 msec
• 9 124.at-6-0-0.TR2.SAC1.ALTER.NET (152.63.6.14)
  180 msec 176 msec 180 msec
• 10 190.ATM6-0.GW8.SJC2.ALTER.NET (152.63.52.181)
  184 msec 188 msec 184 msec
• 11 cisco.customer.alter.net (157.130.200.30) 188
  msec 188 msec 188 msec
• 12 pigpen.cisco.com (128.107.240.170) 188 msec
  184 msec 184 msec
• 13 www.cisco.com (198.133.219.25) 184 msec

31
What a Professional Troubleshooter Told Us -)

• Hi,I think I can explain what is happening.
  Any IP packets received with IP options that
  require processing are process switched. That
  means that the packet is sent to the Route
  Processor for forwarding, rather than switched
  directly using CEF. If the RP is busy, then this
  may add to the latency.The traceroute command
  uses IP options in the header, so these packets
  will be process switched. Any other traffic, such
  as http, ftp etc will be switched using CEF. So
  when they run the traceroute command, they are
  seeing the latency of the packet being forwarded
  through the RP, rather than the true latency of
  the packets being fast switched by
  CEF.Regards,Your professional troubleshooter

32
The Real Reason

• PE sends back ICMP Time Exceeded via appropriate
  Routing Table
• P router has no routing information!
• Only information he has is receiving LSP, but
  LSPs are unidirectional
• So P router sends ICMP Time Exceeded via
  receiving LSP
• ICMP packet travels through the MPLS backbone to
  the other PE, which then has the routing
  information to send the ICMP packet back through
  another LSP to the sender
• So each traceroute probe travels through the
  whole MPLS backbone twice
• TTL is high!
• Customer can accept this because his productive
  traffic is not affected

33
But How to Explain this to a Customer?

• CPE-Routertraceroute www.cisco.com
• 1 accesslink.company.com (10.150.73.17) 0 msec 4
  msec 0 msec lt-- PE (Europe)
• 2 r16b01-pos0-0-0.bb.nextra.net (141.22.5.2) 104
  msec 104 msec 100 msec lt-- P (Europe)
• 3 r08b02-pos1-0-0. bb.nextra.net (141.22.1.6) 15
  msec 12 msec 22 msec lt-- P (Europe)
• 4 r08b01-fe4-1-0. bb.nextra.net (141.22.5.49) 16
  msec 17 msec 14 msec lt-- P (Europe)
• 5 r10b01-pos4-0-0. bb.nextra.net (141.22.5.34)
  160 msec 200 msec 200 msec lt-- PE (USA)
• 6 Pos4-0.GW8.NYC4.ALTER.NET (157.130.21.37) 100
  msec 100 msec 100 msec
• 7 172.ATM2-0.XR2.NYC4.ALTER.NET (146.188.180.62)
  104 msec 104 msec 104 msec
• 8 188.at-1-0-0.TR2.NYC8.ALTER.NET (152.63.21.106)
  108 msec 104 msec 104 msec
• 9 124.at-6-0-0.TR2.SAC1.ALTER.NET (152.63.6.14)
  180 msec 176 msec 180 msec
• 10 190.ATM6-0.GW8.SJC2.ALTER.NET (152.63.52.181)
  184 msec 188 msec 184 msec
• 11 cisco.customer.alter.net (157.130.200.30) 188
  msec 188 msec 188 msec
• 12 pigpen.cisco.com (128.107.240.170) 188 msec
  184 msec 184 msec
• 13 www.cisco.com (198.133.219.25) 184 msec

BUG!
34
Agenda

• The Idea behind this Talk
• Lessons Learned
• The Service Architecture
• The Use of Route Reflectors
• Baby Jumbo Frames
• The LDP-ID
• Traceroute is no longer Your Friend
• Some Security Considerations
• no Religious Wars please -)

35
Configuration Integrity

• Sometimes config lines for a vrf routing
  enviromentappear on the global routing process
  level
• Some buggy software versions
• Typo in vrf name
• Copy-Paste works too fast -)
• This can be very very very dangerous
• Sometimes someone puts a interface into a wrong
  VRF -(
• The problem is not when you break connectivity
  (easy to spot) but when you add too much
  connectivity
• Sanity Check Tools!

36
Configuration Integrity

• How it should look like
• router bgp 88
• usual iBGP stuff
• no auto-summary
• !
• address-family ipv4 vrf green
• redistribute connected
• redistribute rip
• neighbor 211.27.213.9 remote-as 65500
• neighbor 211.27.213.9 activate
• default-information originate
• no auto-summary
• no synchronization
• exit-address-family

• What I found on a router
• router bgp 88
• redistribute rip
• default-information originate

37
Configuration Integrity

• How it should look like
• router rip
• version 2
• no auto-summary
• !
• address-family ipv4 vrf blue
• version 2
• redistribute bgp 88 metric 3
• network 211.27.213.0
• distribute-list 22 in
• no auto-summary
• exit-address-family

• What I found on a router
• router rip
• redistribute bgp 88 metric 3
• network 211.27.213.0

• Imagine what happens if you use addresses from
  the 62/8 block!
• Another reason for good filtering
• (Preconfigure all possible redistri-butions with
  route-map deny-all)

38
Secure Your PE VTYs!

• line vty 0 4
• access-class 99 in
• login authentication bbmethod
• access-list 99 permit 10.25.25.5 0.0.0.0 lt-- NMS
  box
• access-list 99 permit 61.8.0.0 0.0.7.255 lt--
  BB-links

• What happens, if customer sends hostroutefor
  your NMS box?
• He will be able to telnet to your PE router!

39
Secure Your PE VTYs!

• only use hostroutes for NMS
• RIP, OSPF, EBGP has better admin distance than
  IBGP-(
• put hostroutes to Null0 for NMS into all VRFs
• how to manage the CPEs?
• filter routing updates from customer
• do not accept all internal address ranges (NMS,
  BB)
• secure, but could impact customer connectivity

40
Secure Your PE VTYs!

• use better ACLs to secure the VTYs
• interface loopback0
• ip address 1.1.1.1 255.255.255.255
• access-list 199 permit 10.25.25.5 0.0.0.0 1.1.1.1
  0.0.0.0
• access-list 199 permit 61.8.0.0 0.0.7.255 1.1.1.1
  0.0.0.0
• access-list 199 permit 10.25.25.5 0.0.0.0
  61.8.0.0 0.0.7.255
• access-list 199 permit 61.8.0.0 0.0.7.255
  61.8.0.0 0.0.7.255
• needs entries for BB interface addresses as
  telnetdestination (hop-to-hop telnet)
• as BB interface addresses and loopback addresses
  arenot reachable within the vrfs, this should be
  secure
• needs a nice IP addressing scheme (or a clever
  config generation tool)

41
Thats it!

• Thank you for your attention

• Now, it is time for
• questions
• comments
• corrections
• flames

42
How to Secure Your Job Implement MPLS VPNs

• roger_at_gottsponer.ch
• NANOG24 / Miami, Florida
• February 10-12, 2002