Dynamics of Hot-Potato Routing in IP Networks

1
Dynamics of Hot-Potato Routing in IP Networks

• Jennifer Rexford
• ATT LabsResearch
• http//www.research.att.com/jrex
• Joint work with Renata Teixeira (UCSD), Aman
  Shaikh (ATT), and Timothy Griffin (Intel)

2
Outline

• Internet routing
• Interdomain and intradomain routing
• Coupling due to hot-potato routing
• Measuring hot-potato routing
• Measuring the two routing protocols
• Correlating the two data streams
• Performance evaluation
• Characterization on ATTs network
• Implications on operational practices
• Conclusions and future directions

3
Autonomous Systems
4
3
5
2
6
7
1
Web server
Client
AS path 6, 5, 4, 3, 2, 1
4
Interdomain Routing (BGP)

• Border Gateway Protocol (BGP)
• IP prefix block of destination IP addresses
• AS path sequence of ASes along the path
• Policy configuration by the operator
• Path selection which of the paths to use?
• Path export which neighbors to tell?

5
Intradomain Routing (IGP)

• Interior Gateway Protocol (OSPF and IS-IS)
• Shortest path routing based on link weights
• Routers flood link-state information to each
  other
• Routers compute next hop to reach other routers
• Link weights configured by the operator
• Simple heuristics link capacity or physical
  distance
• Traffic engineering tuning link weights to the
  traffic

2
1
3
1
3
2
1
5
4
3
Path cost 215
6
Two-Level Internet Routing

• Hierarchical routing
• Intra-domain
• Metric based
• Inter-domain
• Reachability and policy
• Design principles
• Scalability
• Isolation
• Simplicity of reasoning

Autonomous system (AS) network with unified
administrative routing policy (ex. ATT,
Sprint, UCSD)
7
Coupling Hot-Potato Routing
Routes to thousands of destinations switch exit
point!!!
Z
X
ISP network
ISP network
Y
8
BGP Decision Process

• Ignore if exit point unreachable
• Highest local preference
• Lowest AS path length
• Lowest origin type
• Lowest MED (with same next hop AS)
• Lowest IGP cost to next hop
• Lowest router ID of BGP speaker

9
Hot-Potato Routing Model

• Cost vector for Z cX10, cW8, and cY9
• Egress set for dst X, Y
• Best route for Z through Y, which is closest

Hot-potato change change in cost vector causes
change in best route
10
The Big Picture
11
Why Care about Hot Potatoes?

• Understanding of Internet routing
• Frequency of hot-potato routing changes
• Influence on end-to-end performance
• Operational practices
• Knowing when hot-potato changes happen
• Avoiding unnecessary hot-potato changes
• Analyzing externally-caused BGP updates
• Distributed root cause analysis
• Each AS can tell what BGP updates it caused
• Someone should know why each change happened

12
Our Approach

• Measure both protocols
• BGP and OSPF monitors
• Correlate the two streams
• Match BGP updates with OSPF events
• Analyze the interaction

Z
OSPF messages BGP updates
ATT backbone
X
M
Y
13
Heuristic for Matching
Stream of OSPF messages
Transform stream of OSPF messages into routing
changes
Match BGP updates with OSPF events that happen
close in time
time
Classify BGP updates by possible OSPF causes
14
Computing Cost Vectors

• Transform OSPF messages into path cost changes
  from a routers perspective

OSPF routing changes
Z
1
1
2
1
X
M
2
10
2
10
2
LSA weight change, 10
LSA weight change, 10
LSA delete
1
Y
15
Classifying BGP Updates

• Cannot have been caused by cost change
• Destination just became (un)available in BGP
• New BGP route through same egress point
• New route better/worse than old (e.g., AS path
  len)
• Can have been caused by cost change
• New route is equally good as old route (perhaps X
  got closer, or Y got further away)

M
Z
X
dst
Y
16
The Role of Time

• IGP link-state advertisements
• Multiple LSAs from a single physical event
• Group into single cost vector change
• BGP update messages
• Multiple BGP updates during convergence
• Group into single BGP routing change
• Matching IGP to BGP
• Avoid matching unrelated IGP and BGP changes
• Match related changes that are close in time

Characterize the measurement data to determine
the right windows
17
Summary Results (June 2003)

• High variability in of BGP updates
• One LSA can have a big impact

location min max days gt 10
close to peers 0 3.76 0
between peers 0 25.87 5
location no impact prefixes impacted
close to peers 97.53 less than 1
between peers 97.17 55
18
Delay for BGP Routing Change

• Router vendor scan timer
• BGP table scan every 60 seconds
• OSPF changes arrive uniformly in interval
• Internal BGP hierarchy (route reflectors)
• Wait for another router to change best route
• Introduces a wait for a second BGP scan
• Transmitting many BGP messages
• Latency for transferring the data

We have seen delays of up to 180 seconds!
19
Delay for First BGP Change
Fraction of Hot-Potato Changes
Routers in backbone (June) Routers in backbone
(September) Router in lab
time BGP update time LSA (seconds)
20
Transferring Multiple Prefixes
Cumulative Number of Hot-Potato Changes
time BGP update time LSA (seconds)
21
BGP Updates Over Prefixes
Cumulative BGP updates
Non-OSPF triggered All OSPF-triggered
prefixes
22
Operational Implications

• Forwarding plane convergence
• Accuracy of active measurements
• Router proximity to exit points
• Likelihood of hot-potato routing changes
• Cost in/out of links during maintenance
• Avoid triggering BGP routing changes

23
Forwarding Convergence
R1s scan process can take up to 60 seconds to
run
Scan process runs in R2
R2 starts using R1 to reach dst
10
R2
R1
111
10
100
dst
24
Measurement Accuracy

• Measurements of customer experience
• Probe machines have just one exit point!

loop to reach dst
10
R2
R1
111
100
dst
25
Avoid Equal-distance Exits
dst
dst
26
Careful Cost in/out Links
Z
5
100
5
X
10
10
4
10
Y
Traffic is more predictable Faster
convergence Less impact on neighbors
27
Ongoing Work

• Black-box testing of the routers
• Scan timer and its effects (forwarding loops)
• Vendor interactions (with Cisco)
• Impact of the IGP-triggered BGP updates
• Changes in the flow of traffic
• Forwarding loops during convergence
• Externally visible BGP routing changes
• Improving isolation (cooling those potatoes!)
• Operational practices preventing interactions
• Protocol design weaken the IGP/BGP coupling
• Network design internal topology/architecture

28
Thanks!

• Any questions?

29
iBGP Route Reflectors
X
Z
9
10
Y
8
11
20
W
Scalability trade-off Less BGP state vs.
Number of BGP updates from Z and longer
convergence delay
30
Exporting Routing Instability
Z
X
dst
Y
No change gt no announcement
dst