Interconnection, Peering, and Settlements

1
Interconnection, Peering, and Settlements

• Geoff Huston

2
Overview

• Introduction
• Interconnection Peer or client
• Interconnection architecture
• Interconnection financials
• Settlement models
• Settlement structures
• QOS financial settlement
• Conclusions

3
Introduction

• Tens of thousands of ISPs operating in
  deregulated business space
• Complex environment
• Transaction between two ISPs involves multiple
  providers
• Basic financial cost distribution is based on
  bilateral relationships of customer/provider and
  mutual peering
• Internet industry use small set of physical
  interconnection mechanisms (LAN switches)

4
Interconnection retailing, reselling, wholesale

• Internet is an outcome of business and technology
  interaction
• ISPs do not have clear precise roles
• Retail, resel, and wholesale
• Retail ISP can easily become a wholesale provider
• Many ISPs operate as client and provider
• Hard to support stable segmentation into
  wholesale and retail market sectors

5

• Intrernet enviroment
• There is no well ordered hierarchical model of a
  set of wholesale ISPs and retail ISPs
• Diverse ISPs operating as retailers and wholesale
  providers to other retailers

6
Peer or Client

• Leads to a question of who is making subjective
  decision and on what basis.
• Traditional public solution pay a fee to a
  regulator that gives ISP a peer license
• Two problems
• Regulators are artificial in defining the market
  entities (client vs. peers)
• Discourages large-scale private investment, thus
  putting funding burden on public sector

7
Things have changed

• Regulatory environment is changing to shift
  burden of comm. infrastructure
• Public sector to private investment
• Deregulated environment
• No one can say whether two ISPs are
  client/provider or peers
• Who can say so in the industry?
• Commercial Internet eXchange (CIX)
• Based on description of infrastructure of each
  party
• Peer if you have a national transit
  infrastructure capability
• Modified later pay a fee to CIX assicaition
• Not bilateral but multilateral relationship with
  other peers !!
• Zero financial settlement (based on a fee)

8

• Other models use functional peer specification
• If ISP attaches to physical exchange entity, it
  is up to the ISP to open bilateral negotiation
  with other ISPs attached to the exchange
• True peer relationship is based on the assumption
  that either party can terminate the
  interconnection
• If one ISP relies on the interconnection more
  than the other ? provider/client relationship
• If there is a balance of mutual requirement
  between two ISPs ? peer interconnection
  relationship
• Problem no metrics to quantify requirements
  (based on perception)
• There are various levels of peering in todays
  internet due to business pressures
• Local ISPs see competing local ISPs as peers
• Local ISPs are clients of trunk ISPs

9
Interconnection Architecture

• Strict hierarchical structure
• Worst case traffic between two ISPs may traverse
  transit ISPs
• Extended paths are inefficient and costly
• To reduce costs, ISPs at different levels
  construct bilateral interconnections

10
How to connect with ISPs?

• Connect with all ISPs (full mesh, not scalableN2
  connections)
• Local exchange model
• ISP connects to a single local exchange
  (scalable N connections)
• Exchange router is active component in peering
  policy
• Each ISP must have multilateral peering with all
  other ISPs
• Router must execute its own routing policy
• When two ISPs advertise a route to same
  destination, router makes decision on behalf of
  all other connected ISPs
• Router may not be completely neutral to all ISPs

11
What do ISPs expect?

• Flexibilty of policy determination from exchange
  structure
• Bilateral interconnection at the exchange
  structure
• Make policy decisions when same destination is
  adverised by multiple providers
• Exchange must be neutral with respect to
  individual routing policies
• HOW?
• Modify exchange model to use LAN switch as
  exchange element

12

• Each ISP
• has a dedicated router at the LAN exchange
• has a bilateral peering agreement with another
  ISP by initiating router peering session with the
  others router
• If multiple peers advertise a path to same dest,
  ISP can use its own policy-based preference to
  choose route
• Exchange environment must offer high degree of
  resilience and security (costs a lot)

13
Distributed exchange model

• Exchange comes to ISP location
• Must have uniform access technology between every
  exchange participant
• Issues switching speed (contention can be a
  problem)

14
Network Access Points NAPs

• Roles of NAP
• Exchange provider between ISPs
• Transit purchase site to make agreements between
  ISPs and trunk ISPs

15
Exchange Business Models

• In the ISP industry, a common business model
  require the internet exchange to be
• Operated by neutral party that is not an ISP
• Constructed in robust and secure fashion
• Located in high density areas of internet market
• Scalable
• Operated in sound and stable fasion
• Others (Performance of the exchange, QOS)
• Common business models use flat-fee structure
• Based on the number of rack units used by an ISP
  at the exchange
• Other models are strcutured as cooperative entity
  between a number of ISPs
• Problem no ISP wants to financially take
  responsibilty for ensuring quality of the
  exchange

16
Todays Internet

• Increasing ISPs will lead to increasing
  complexity of interconnection structure
• Inability to reach stable cost distribution model
  makes each ISP optimize itself by making direct
  connections with peer ISPs (thru exchanges or
  direct 11 links)
• Curdity of inter-AS routing policy tools makes
  internet structure a source of considerable
  concern especially with
• Absence of coherent policy (or even commonly
  accepted set of practices)
• Lack of administration of the AS space

17
Interaction Financials

• Cost distribution
• Compensation of all ISPs who participate in the
  delivery of a service to a customer of a single
  ISP
• Users want comprehensive end-to-end service with
  clients being parts of different ISPs
• How do all ISPs involve in a transaction?
• Who incur the cost in supporting the transaction?
• Who receive compensation?
• What is the cost distribtion model?

18
What is the currency of interaction?

• Routing advertisements
• ISPs exchange routing entries to allow traffic
  flows
• Traffic flows in opposite direction of route
  advertisement

19
Types of Routes

• Clients routes
• Passed to ISP routing by contract with client,
  static configuration at edge of ISP, learned by
  BGP, or part of DHCP addresses
• Internal ISP routes
• DNS, SMTP, SNMP, POP..etc
• Upstream routes
• Learned from making a transit service contract
  with upstream ISP
• Peer routes
• Learned from exchange or private interconnection
• What is the route export policy ?

20
Internet settlement models

• Packet cost accounting (strawman model)
• Everytime a packet crosses network boundry, it is
  sold to next ISP
• Ultimately, the packet is sold to the receiver
  client
• Pros
• Revenue gains from packets deliverd on egress
  from network
• Economic incentives not to drop packets in
  transit ISPs
• Cons
• Packet drop is inevitable
• Mechanism is open to abuse

21
TCP session accounting

• Network boundry can
• Detect initial TCP handshake
• Count all subsequent packets with same TCP
  session
• Session initiator pays for entire traffic flow
• Such accounting allows for settlement based on
  dutration (TCP packets) or volume (TCP sessions)
• Problem very hard to do
• Router at the network boundry must do all work !!
• Real problem with any settlement models
• Todays internet have many retail pricing
  structures
• Based on received volume, sent volume, mix,
  access capacity
• There is no uniform retail pricing

22
Internet settlement structures

• Two structures of interactions between two ISPs
• Customer/provider and peering with no form of
  financial settlement
• Sender Keep All (SKA)
• Usually applies to domestic traffic between two
  ISPs
• Stable when both parties perceive equal benefit
  from interconnection
• Ex. Among local ISPs, regional ISPs, national
  ISPs
• How does it work? On each interconnection
• each ISP ONLY presents/accepts to/from other ISP
  routes associated with its customers
• Clients make contract with an ISP to present
  their routes to all other customers of the same
  ISP, to the upstream providers of the ISP, and
  all peer ISPs

23
How does all that look like ?

• Internet into two domains transit ISPs, local
  ISPs
• Transit ISPs high capacity carriage
  infrastructure
• Local ISPs retail services
• Participate at exchanges with SKA peer
  interconnection with other ISPs
• Exchange does not have full connectivity ? ISPs
  purchase transit services

24
Negotianted financial settlement

• Alternative to customer/provider and peer
  structures
• Based on both parties selling services to each
  other across the interconnection
• Simple model
• Measure volume of traffic in each direction
• Use single accounting rate for all traffic
• At the end of accounting period, two ISPs settle
  based on the agreed rate applied to the traffic
• Which way should the money flow in relation to
  traffic flow?
• One model Originating ISP should pay terminating
  ISP to deliver traffic
• Another model when traffic is generated because
  of an action of a receiver (webpage, downloads),
  terminating ISP should pay

25
Settlement Debate

• Despite great ISP attention, todays internet
  does not have sound models of financial
  settlements
• Why has the internet managed to pose hard
  challenge to the ISP industry?
• Caused by adopted retail models of ISP services
• The internet as retailed to clients is not a
  comprehensive end-to-end service
• Internet works as a result of partial path paired
  services
• Sender funds initial path component and receiver
  funds terminating path component
• Natural outcome of todays internet settlement
  environment is one of aggregation of ISPs

26
QOS and financial settlements

• The shift towards end-to-end service model and
  support of QOS are strong factors to change
  current ISP service model
• Meaningful inter-provider financial settlements
  depend highly on introducing end-to-end service
  retail models
• That in turn depends on universally shifting from
  best-effort regime to layered end-to-end service
  regimes

27
Conclusions

• 0 peering and customer/provider relationships
  are the only stable models within the internet
• As a consequence
• deployment of end-to-end QOS is highly unlikely
  in such an environment
• Inability to support highly diverse ISP env
• Aggregation within ISP industry