Massive Disruptions in the Peering Ecosystem

1
Massive Disruptions in the Peering Ecosystem

• William B. Norton
• Co-Founder Chief Technical Liaison
• Equinix, Inc.

ATUG 2005 Meeting Australia Technology
Park Sydney, Australia 3/9/2005
Slide Set 1.0
2
Part I Internet Peering Ecosystem Framework
3
Def Transit
The Internet is a Network of Networks.
An ISP sells access to the Internet, so
must itself get attached to someone who is
already attached to the Internet.
1) Transit Provider sells metered Internet access.
Transit Provider A
Transit Provider B
meter
4
Def Peering
2) Peering is a business relationship whereby two
companies RECIPRICALLY exchange access to each
others customers.

• Why Peer?
• Reduce Transit Costs
• Lower Latency
• More Control
• over Traffic

Transit Provider A
Transit Provider B
Peering
5
What is this Internet Peering Ecosystem anyway?
Global Internet Peering Ecosystem
JP Internet Region
AU Internet Region
US Internet Region
Tier 1 ISPs
Tier 2 ISPs
Content Providers
6
Ecosystem Member Tier 1 ISP
P
Def A Tier 1 ISP is an ISP that has access to
the ENTIRE Internet Region Routing Table Solely
via Peering Relationships
P



T
T
(Doesnt buy transit from anyone to reach any
destination in the Internet Region.)
Motivation Is NOT motivated to Peer in region to
reduce transit fees, Is NOT motivated to peer
with anybody else.
Behavior Restrictive Peering
Policy
7
Ecosystem Member Tier 2 ISP


T
T

P
Def A Tier 2 ISP is an ISP that has to purchase
Transit to access some part of the Internet
Region.
P



T
T
Motivation Is motivated to Peer in region to
reduce transit fees.
Behavior Open Peering or Selective Peering
Policy Active in Peering Forums
8
Content Providers


T
T

Def A Content Provider focuses on content
development and does not Sell access to the
Internet.
Motivation SLAs w/well known ISP
Behavior No Peering Policy
9
Generic Peering Ecosystem
Tier 1 ISPs
T
T
T
T
T
T
T
T
T
flow up
Tier 2 ISPs
T
T
T
Content Providers
Apply defs
10
Apply Defs Peering Dynamics Motivations
Tier 1 ISP Y
Tier 1 ISP Y
P

T
T

Tier 2 ISP
Tier 2 ISP

T
Content Provider
Synch Point You have all the defs needed to
predict behavior in the Peering Ecosystem,
and Understand Peering Ecosystem Disruptions.
11
Part II Massive Disruptions in the Peering
Ecosystem
12
Massive Disruptions

• Changes at the Core of the Ecosystem
• Example Power Structure Changes
• T1 Acquisitions
• T2 Peering Cost Dynamics
• Transit Price Trends
• T1 Regulatory Actions (or threats)
• Ex Shift in the Peering vs. Transit tradeoff
• Effect Traffic flow ? Dollar Flow ? End User
  Performance ? End User Strategy

13
Disruptions in Australia
14
Massive Disruption in AU Peering Ecosystem
Transit Prices Drop
What is happening here?
Source APRICOT 2005 Peering BOF Discussions,
Kyoto
15
AU Disruption Peering Costs Drop
In 2004, Telstra 3500USD for 2M Local
Loop Competitors - 2500USD for 100M local
loop In 2005, PIPE 800 USD for a 100M to IX!
(1)
IX
T
ISP
TeleGlobe
T
Intl
Peering Fabric
MCI
Peer traffic here (costlooprackport)
Intl
/Mbps
P
P
P
Qwest
Intl
ISP
ISP
CP
Cost of Transit
Buy Intl Transit Here for 300/Mbps (estimate)
(2)
Cost of Peering
(3) Blended Peering and Cheap Transit Circumvents
Tier1s
Mbps
16
Implications for Australia

• For more Tier 2 ISPs - Peering reduces cost
• Cost per Mbps can approach 30/Mbps for 100M of
  peered domestic traffic
• Peering becomes compelling for Large Scale
  Network Savvy Content Providers (Yahoo!)
• Leverage Intl ISPs (MCI, Qwest, TeleGlobe, etc.)
  there already 300/Mbps Intl transit
• Cheap T2s run pipes hot, can cool down
  w/aggregated transport into IX for PT

17
Disruptions in New Zealand
18
New Zealand Peering Ecosystem Last Week
Tier 1 ISPs
Stratification going on now. TelstraClear Is
depeering
Tier 1 peered w/T2
Wellington Internet Exchange (WIX)
Auckland Peering Exchange (APE)
CityLink Fabric in Auckland
CityLink Fabric in Wellington (300 buildings!)
Govt
Tier 2 ISPs
Pressuring ISPs to sell transit peer at WIX
Network Savvy Large Scale CP
Effect of depeering?
19
TelstraClear Announced Depeering

• At around 1100 today. TelstraClear (AS7714 and
  AS4768) will disconnect
• from the WIX route servers.
• Technical queries should be directed at the
  TelstraClear Customer Help
• team (0508 555 500) or to your normal fault
  escalation point.
• Business queries should be directed to your
  TelstraClear account
• manager.
• Thanks
• Dean Pemberton

Source http//www.crc.net.nz/mglb1/tc-depeer/
20
and then De-peered

• 64 bytes from trademe.co.nz (202.21.128.2)
  icmp_seq816 ttl56 time39.3 ms
• 64 bytes from trademe.co.nz (202.21.128.2)
  icmp_seq817 ttl56 time26.7 ms
• 64 bytes from trademe.co.nz (202.21.128.2)
  icmp_seq818 ttl56 time32.5 ms
• 64 bytes from trademe.co.nz (202.21.128.2)
  icmp_seq819 ttl56 time32.6 ms
• 64 bytes from trademe.co.nz (202.21.128.2)
  icmp_seq820 ttl56 time34.4 ms
• 64 bytes from trademe.co.nz (202.21.128.2)
  icmp_seq821 ttl56 time36.7 ms
• 64 bytes from trademe.co.nz (202.21.128.2)
  icmp_seq822 ttl56 time39.8 ms
• 64 bytes from trademe.co.nz (202.21.128.2)
  icmp_seq823 ttl56 time36.7 ms
• 64 bytes from trademe.co.nz (202.21.128.2)
  icmp_seq824 ttl56 time35.8 ms
• 64 bytes from trademe.co.nz (202.21.128.2)
  icmp_seq825 ttl56 time29.5 ms
• 64 bytes from trademe.co.nz (202.21.128.2)
  icmp_seq826 ttl56 time191 ms
• 64 bytes from trademe.co.nz (202.21.128.2)
  icmp_seq827 ttl56 time179 ms
• 64 bytes from trademe.co.nz (202.21.128.2)
  icmp_seq828 ttl54 time184 ms
• 64 bytes from trademe.co.nz (202.21.128.2)
  icmp_seq829 ttl54 time183 ms
• 64 bytes from trademe.co.nz (202.21.128.2)
  icmp_seq830 ttl54 time191 ms
• 64 bytes from trademe.co.nz (202.21.128.2)
  icmp_seq831 ttl54 time189 ms
• 64 bytes from trademe.co.nz (202.21.128.2)
  icmp_seq832 ttl54 time188 ms

Source http//www.crc.net.nz/mglb1/tc-depeer/
21
Circuitous Path

• Packets Pings
• Hostname Loss
  Rcv Snt Last Best Avg Worst
• 1. gtw-svr.crc.net.nz 0
  4 4 0 0 0 0
• 2. 192-107-171-49.network.waikato.ac.n 0
  4 4 0 0 0 0
• 3. pegasus.network.waikato.ac.nz 0
  4 4 1 0 1 1
• 4. 203.167.234.85 0
  4 4 21 20 21 23
• 5. jcore2-ge-0-2-0-927-acld.auckland.c 0
  4 4 21 13 19 23
• 6. ge-0-2-0-1.xcore1.acld.telstraclear 0
  4 4 16 16 19 20
• 7. xcore2.tspn.telstraclear.net 34
  2 3 151 145 148 151
• 8. GigabitEthernet3-0.GW1.LAX1.ALTER.N 0
  3 3 153 153 158 162
• 9. 0.so-0-0-0.XL1.LAX1.ALTER.NET 0
  3 3 137 137 141 148
• 10. 0.so-1-1-0.XL1.LAX7.ALTER.NET 0
  3 3 131 131 139 146
• 11. 0.so-6-0-0.BR1.LAX7.ALTER.NET 0
  3 3 136 134 138 145
• 12. 204.255.168.30 0
  3 3 146 146 147 149
• 13. tbr1-p014001.la2ca.ip.att.net 0
  3 3 154 146 151 154
• 14. tbr2-cl3.sffca.ip.att.net 0
  3 3 160 151 155 160
• 15. gbr5-p20.sffca.ip.att.net 34
  2 3 161 149 155 161
• 16. gar2-p360.sffca.ip.att.net 34
  2 3 165 165 166 166
• 17. ???

Better in picture form
22
25 hops, Avg. 174ms
ATT Tokyo
ATT Chicago
ATT San Fran
PACIFIC
ATT Los Angeles
OCEAN
ATT Sydney
TradeMe NZ
UUNet Los Angeles
Waikato Hamilton
TelstraClear Auckland
Average Was 17ms before de-peering!
Source http//www.crc.net.nz/mglb1/tc-depeer/
23
TelstraClear now has a Paid Peering Product

• Possible Motivations?
• Leveraging/Exploiting market power?
• Each Tier 1 has 30 Mkt share
• Or
• High-Performance New Networking Service?
• Or
• Following the rest of the world (Tier 1 ISPs
  dont generally peer with anyone else!)
• Many views/spins/opinions here
• Many people upset in NZ about this

24
Implications for New Zealand

• Stratified Tier 1 / Tier 2 / Content Provider
  Ecosystem
• Tier 2s and Content continue to grow peering
  alliances
• Regulators watch this very closely

25
Summary

• Framework needed for meaningful discussion
• Transit and Peering relationships
• Tier 1 ISP, Tier 2 ISP, Content Provider Players
• Players and relationship dynamics active in the
  Peering Ecosystem
• Australia
• Transit Costs dropped
• Cheap Transport
• Peering lowering cost, latency, increasing
  control
• New Zealand
• Stratification of Tier 1 Tier 2
• Effects of de-peering

We can talk about Japan and US Disruptions
off-line
26
(No Transcript)
27
Disruptions to US Peering Ecosystem
28
1998
29
P?
YES!
30
4) 2005 RBOCs buy T1s
1) Cable Cos Peer
2) LS Content Peer
3) LS peer Cable
31
4) 2005 RBOCs buy T1s
1) Cable Cos Peer
2) LS Content Peer
3) LS peer Cable
32
Implication for US

• SBCATT will continue Selective Restrictive
  Peering for Retail Wholesale
• VerizonMCI will aggregate networks
• VMCI will be restrictive, global, w/ffth
• Broadband Players become power players
• Tier 1s become Intl Tier 1s sell Intl
  routes competively