Traffic Engineering for ISP Networks

1
Traffic Engineering for ISP Networks

• Jennifer Rexford
• Computer Science Department
• Princeton University
• http//www.cs.princeton.edu/jrex

2
Outline

• Overview of Internet routing
• IP addressing and forwarding
• Interdomain and intradomain routing
• Constructing the forwarding table
• Role of optimization in traffic engineering
• Heuristics for setting the link weights
• Optimizing routing to prevailing traffic
• Local search to select the integer link weights
• Conclusion and ongoing work

3
IP Service Model Best-Effort Packet Delivery

• Packet switching
• Send data in packets
• Header with source destination address
• Best-effort delivery
• Packets may be lost
• Packets may be corrupted
• Packets may be delivered out of order

4
Packet Delivery Based on Destination IP Address

• 32-bit number in dotted-quad notation
  (12.34.158.5)
• Divided into network host portions (left and
  right)
• 12.34.158.0/24 is a 24-bit prefix with 28
  addresses

12
34
158
5
Network (24 bits)
Host (8 bits)
5
Longest-Prefix Match Forwarding

• Forwarding tables in IP routers
• Maps each IP prefix to next-hop link(s)
• Destination-based forwarding
• Packet has a destination address
• Router identifies longest-matching prefix

forwarding table
4.0.0.0/8 4.83.128.0/17 12.0.0.0/8 12.34.158.0/24
126.255.103.0/24
destination
12.34.158.5
outgoing link
Serial0/0.1
6
Where do Forwarding Tables Come From?

• Routers have forwarding tables
• Map prefix to outgoing link(s)
• Entries can be statically configured
• E.g., map 12.34.158.0/24 to Serial0/0.1
• But, this doesnt adapt
• To failures
• To new equipment
• To the need to balance load
• That is where routing protocols come in

7
Two-Tiered Internet Routing Architecture

• Goal distributed management of resources
• Internetworking of multiple networks
• Networks under separate administrative control
• Solution two-tiered routing architecture
• Intradomain inside a region of control
• Okay for routers to share topology information
• Routers configured to achieve a common goal
• Interdomain between regions of control
• Not okay to share complete information
• Networks may have different/conflicting goals

8
Autonomous Systems (ASes)

• Autonomous Systems
• Distinct regions of administrative control
• Routers and links managed by an institution
• Service provider, company, university,
• AS hierarchy
• Tier-1 provider with national or global backbone
• Regional provider with smaller backbone
• Campus or corporate network
• Interaction between ASes
• Internal topology is not shared between ASes
• but, neighboring ASes interact to coordinate
  routing

9
AS Numbers (ASNs)
Currently around 20,000 in use.

• Level 3 1
• MIT 3
• Harvard 11
• Yale 29
• Princeton 88
• ATT 7018, 6341, 5074,
• UUNET 701, 702, 284, 12199,
• Sprint 1239, 1240, 6211, 6242,

ASNs represent units of routing policy
10
Traffic Traverses Multiple ASes
Path 6, 5, 4, 3, 2, 1
4
3
5
2
6
7
1
Web server
Client
11
Interdomain Routing Border Gateway Protocol

• ASes exchange info about who they can reach
• IP prefix block of destination IP addresses
• AS path sequence of ASes along the path
• Policies configured by the ASs network operator
• Path selection which of the paths to use?
• Path export which neighbors to tell?

I can reach 12.34.158.0/24 via AS 1
I can reach 12.34.158.0/24
1
2
3
data traffic
data traffic
12.34.158.5
12
Interior Gateway Protocol (Within an AS)

• Routers flood information to learn the topology
• Routers determine next hop to reach other
  routers
• By computing shortest paths based on the link
  weights
• Link weights configured by the network operator

2
1
3
1
3
2
1
5
12.34.158.0/24
Serial0/0.1
4
3
13
Constructing the Forwarding Table

• Two routing protocols
• BGP learn the external route at some border
  router
• IGP learn outgoing link on path to other router
• Router joins the data
• Prefix 12.34.158.0/24 reached through red router
• Red router reached via link Serial0/0.1
• Forwarding entry 12.34.158.0/24 ? Serial0/0.1
• Router forwards packets
• Lookup destination 12.34.158.5 in table
• Forward packet out link Serial0/0.1

14
What if There are Multiple Choices?
12.34.158/24
egress 2
egress 1
IGP distances
56
15
This router has two BGP routes to 192.44.78.0/24.
Hot potato get traffic off of your network as
soon as possible. Go for egress 1!
15
Two Kinds of Routing Protocols
Link State
Vectoring

• Topology information is flooded within the
  routing domain
• Best end-to-end paths are computed locally at
  each router.
• Best end-to-end paths determine next-hops.
• Based on minimizing some notion of distance
• Works only if policy is shared and uniform
• Examples OSPF, IS-IS

• Each router knows little about network topology
• Only best next-hops are chosen by each router for
  each destination.
• Best end-to-end paths result from composition of
  all next-hop choices
• Does not require any notion of distance
• Does not require uniform policies at all routers
• Examples RIP, BGP

16
Role of Optimization in Traffic Engineering
17
Link Weights Control the Flow of Traffic

• Routers compute paths
• Shortest paths as sum of link weights
• Operators set the link weights
• To control where the traffic goes

2
1
3
1
3
2
3
1
5
4
3
18
Heuristics for Setting the Link Weights

• Proportional to physical distance
• Cross-country links have higher weights than
  local ones
• Minimizes end-to-end propagation delay
• Inversely proportional to link capacity
• Smaller weights for higher-bandwidth links
• Attracts more traffic to links with more capacity
• Tuned based on the offered traffic
• Network-wide optimization of weights based on
  traffic
• Directly minimizes key metrics like max link
  utilization

19
Why Are the Link Weights Static?

• Strawman alternative load-sensitive routing
• Link metrics based on traffic load
• Flood dynamic metrics as they change
• Adapt automatically to changes in offered load
• Reasons why this is typically not done
• Delay-based routing unsuccessful in the early
  days
• Oscillation as routers adapt to out-of-date
  information
• Most Internet transfers are very short-lived
• Research and standards work continues
• but operators have to do what they can today

20
Big Picture Measure, Model, and Control
Network-wide what if model
Offered traffic
Topology/ Configuration
Changes to the network
measure
control
Operational network
21
Traffic Engineering in an ISP Backbone

• Topology
• Connectivity and capacity of routers and links
• Traffic matrix
• Offered load between points in the network
• Link weights
• Configurable parameters for Interior Gateway
  Protocol
• Performance objective
• Balanced load, low latency, service level
  agreements
• Question Given the topology and traffic matrix
  in an IP network, which link weights should be
  used?

22
Key Ingredients of Our Approach

• Instrumentation
• Topology monitoring of the routing protocols
• Traffic matrix widely deployed traffic
  measurement
• Network-wide models
• Representations of topology and traffic
• What-if models of shortest-path routing
• Network optimization
• Efficient algorithms to find good configurations
• Operational experience to identify key
  constraints

23
Formalizing the Optimization Problem

• Input graph G(R,L)
• R is the set of routers
• L is the set of unidirectional links
• cl is the capacity of link l
• Input traffic matrix
• Mi,j is traffic load from router i to j
• Output setting of the link weights
• wl is weight on unidirectional link l
• Pi,j,l is fraction of traffic from i to j
  traversing link l

24
Multiple Shortest Paths With Even Splitting
Values of Pi,j,l
25
Defining the Objective Function

• Computing the link utilization
• Link load ul Si,j Mi,j Pi,j,l
• Utilization ul/cl
• Objective functions
• minl(ul/cl)
• minl(S f(ul/cl))

26
Complexity of the Optimization Problem

• NP-complete optimization problem
• No efficient algorithm to find the link weights
• Even for the simple convex objective functions
• Why cant we just do multi-commodity flow?
• E.g., solve the multi-commodity flow problem
• and the link weights pop out as the dual
• Because IP routers cannot split arbitrarily over
  ties
• What are the implications?
• Have to resort to searching through weight
  settings

27
Optimization Based on Local Search

• Start with an initial setting of the link weights
• E.g., same integer weight on every link
• E.g., weights inversely proportional to link
  capacity
• E.g., existing weights in the operational network
• Compute the objective function
• Compute the all-pairs shortest paths to get
  Pi,j,l
• Apply the traffic matrix Mi,j to get link loads
  ul
• Evaluate the objective function from the ul/cl
• Generate a new setting of the link weights

repeat
28
Making the Search Efficient

• Avoid repeating the same weight setting
• Keep track of past values of the weight setting
• or keep a small signature (e.g., a hash) of
  past values
• Do not evaluate a weight setting if signatures
  match
• Avoid computing the shortest paths from scratch
• Explore weight settings that changes just one
  weight
• Apply fast incremental shortest-path algorithms
• Limit the number of unique values of link weights
• Do not explore all 216 possible values for each
  weight
• Stop early, before exploring the whole search
  space

29
Incorporating Operational Realities

• Minimize number of changes to the network
• Changing just 1 or 2 link weights is often enough
• Tolerate failure of network equipment
• Weights settings usually remain good after
  failure
• or can be fixed by changing one or two weights
• Limit dependence on measurement accuracy
• Good weights remain good, despite random noise
• Limit frequency of changes to the weights
• Joint optimization for day and night traffic
  matrices

30
Application to ATTs Backbone Network

• Performance of the optimized weights
• Search finds a good solution within a few minutes
• Much better than link capacity or physical
  distance
• Competitive with multi-commodity flow solution
• How ATT changes the link weights
• Maintenance done every night from midnight to 6am
• Predict effects of removing link(s) from the
  network
• Reoptimize the link weights to avoid congestion
• Configure new weights before disabling equipment

31
Example from My Visit to ATTs Operations Center

• Amtrak repairing/moving part of the train track
• Need to move some of the fiber optic cables
• Or, heightened risk of the cables being cut
• Amtrak notifies us of the time the work will be
  done
• ATT engineers model the effects
• Determine which IP links go over the affected
  fiber
• Pretend the network no longer has these links
• Evaluate the new shortest paths and traffic flow
• Identify whether link loads will be too high

32
Example Continued

• If load will be too high
• Reoptimize the weights on the remaining links
• Schedule the time for the new weights to be
  configured
• Roll back to the old weight setting after Amtrak
  is done
• Same process applied to other cases
• Assessing the networks risk to possible failures
• Planning for maintenance of existing equipment
• Adapting the link weights to installation of new
  links
• Adapting the link weights in response to traffic
  shifts

33
Conclusion

• IP networks do not adapt on their own
• Routers compute shortest paths based on static
  weights
• Service providers need to adapt the weights
• Due to failures, congestion, or planned
  maintenance
• Leads to an interesting optimization problems
• Optimize link weights based on topology and
  traffic
• Optimization problem in NP-complete
• Forces the use of efficient local-search
  techniques
• Results of the local search are good
• Near-optimal solutions that minimize disruptions

34
Ongoing Work

• Robust link-weight assignments
• Link/node failures
• Range of traffic matrices
• More complex routing models
• Hot-potato routing
• BGP routing policies
• Interaction between ASes
• Inter-AS negotiation for joint optimization
• Grappling with scalability and trust issues
• Optimization as a first-class citizen
• Protocols that lead to tractable optimization
  problems
• Centralized computation of the forwarding tables

35
Extra Material Computing the Traffic Matrix
36
Computing the Traffic Matrix Mi,j

• Hard to measure the traffic matrix
• IP networks transmit data as individual packets
• Routers do not keep traffic statistics, except
  link utilization on (say) a five-minute time
  scale
• Need to infer the traffic matrix Mi,j from
• Current topology G(R,L)
• Current routing Pi,j,l
• Current link load ul
• Link capacity cl

37
Inference Network Tomography
From link counts to the traffic matrix
Sources
3Mbps
5Mbps
4Mbps
4Mbps
Destinations
38
Tomography Formalizing the Problem

• Ingress-egress pairs
• p is a ingress-egress pair of nodes (i,j)
• xp is the (unknown) traffic volume for this pair
  Mi,j
• Routing
• Plp is proportion of ps traffic that traverses l
• Links in the network
• l is a unidirectional edge
• ul is the observed traffic volume on this link
• Relationship u Px (work backwards to get x)

39
Tomography One Observation Not Enough

• Linear system of n nodes is underdetermined
• Number of links e is around O(n)
• Number of ingress-egress pairs c is O(n2)
• Dimension of solution sub-space at least c - e
• Multiple observations are needed
• k independent observations (over time)
• Stochastic model with Poisson iid counts
• Maximum likelihood estimation to infer matrix
• Doesnt work all that well in practice

40
Approach Used at ATT Tomo-gravity

• Gravitational assumption
• Ingress point a has traffic via
• Egress point b has traffic veb
• Pair (a,b) has traffic proportional to via veb

9
20
10
41
Approach Used at ATT Tomo-gravity

• Problem with gravity model
• Gravity model ignores the load on the inside
  links
• Gravity assumption isnt always 100 correct
• Resulting traffic matrix might not satisfy the
  link loads
• Combining the two techniques
• Gravity find a traffic matrix using the gravity
  model
• Tomography find the family of traffic matrices
  consistent with all link load statistics
• Tomo-gravity find the tomography solution that
  is closest to the output of the gravity model
• Works extremely well (and fast) in practice

42
Conclusions

• Managing IP networks is challenging
• Routers dont adapt on their own to congestion
• Routers dont reveal much information about
  traffic
• Measurement provides a network-wide view
• Topology
• Traffic matrix
• Optimization enables the network to adapt
• Inferring the traffic matrix from the link loads
• Optimizing the link weights based on the traffic
  matrix