Traditionally, a settlement free peering arrangement was entered into between "peers," in other words, equal networks. Peering policies articulated the benchmarks used for determining if networks were "peers," that "peers" were roughly equal in size and exchanged roughly balanced traffic and therefore the increased value from interconnection would be mutually beneficial. In the nascent commercial Internet ecosystem of the 1990s, there were Tier 1 backbone networks, Tier 2 regional networks, Tier 3 access networks, end users, and edge providers. The network was hierarchical. Each participant in the ecosystem had a relatively discreet business plan. Thus "peers" were (somewhat) easy to identify.
The Internet ecosystem evolved tremendously in the last three decades. It is much more difficult to identify a "peer." The network is no longer hierarchical. Providers' business plans have expanded, offering access, regional, and backbone service. A provider may be a mix of hosting, cloud, CDN, and network services. Large access networks no longer are reliant upon third party backbone service and no longer pay for transit service; they charge content networks to access the access network's subscribers. Content networks are also no longer as reliant on third party backbone services, instead using CDN's to move content as close to end-users as possible. Instead of content moving up the Internet hierarchy until it reaches Tier 1 and an interconnection point, it is moved to the gateway of the access network where the CDN attempts to negotiate interconnection with the access network. This evolution has changed the inclination of networks to enter into settlement free peering arrangements.
This paper seeks to resurvey peering policies in light of the evolution of the Internet ecosystem. What has this evolution meant for the inclination of network providers to peer? Is this evolution reflected in peering policies? Is there a difference in peering policies between backbone service providers, large access providers, and content networks? Are there clauses that remain common across all policies? Can there be "peering" if there are no longer "peers"?
Methodology
The scope of the survey is peering policies for networks providing service in the domestic United States market. Peering policies were found using Google, reviewing CAIDA and DYN's list of top transit providers, and reviewing PeeringDB.
Peering policies were broken down into common clauses. The breakdown of those clauses sometimes did and sometimes did not track the breakdown used by the providers.
Providers could have multiple peering policies. Some providers have distinct business units that have distinct inclinations. For example, Verizon's CDN Edgecast has a different peering policy than Verizon broadband Internet. Other companies have multiple policies due to mergers and acquisitions. For example, Charter Communications acquired Time Warner Cable; both networks continue to have posted peering policies. Verizon also has distinct posted peering policies for its acquired assets AOL, Yahoo!, and XO. Legacy peering policies of acquired networks were reviewed if available online even though it cannot be determined whether they remain in effect.
In total, 60 networks were surveyed. While the sample size of networks surveyed is relatively small, this sample represents the top backbone providers, BIAS providers, CDNs, and edge providers, representing a strong majority of network traffic. This methodology, however, biased the results towards the behavior of the largest providers versus small providers; markets have a long tail of hundreds of small providers who may produce different results.[1]
Providers were classified according to primary Internet business: edge provider, broadband Internet access service, content delivery network (both private and commercial grouped together), educational, and infrastructure (i.e., ARIN, ICANN, ISC). This proved difficult. One of the premises of this survey is that providers no longer fit within discrete business plans where peers are easily discerned. Providers have diversified their business plans, with incentives to move towards more profitable services. Therefore, providers were classified according to their dominant characteristic at the point of interconnection. For instance, Netflix is a video delivery service. At the point of interconnection, Netflix could be appropriately characterized as a private CDN. Many of the edge providers and content providers have constructed their own private CDNs and at the point of interconnection are attempting to deliver data to end users. It is a subjective classification but probably sufficient to observe market dynamics.
Findings
Peering Policies
Sixty five percent of surveyed networks had posted peering policies. One hundred percent had PeeringDB entries (in other words, all networks without posted peering policies nonetheless had PeeringDB entries). [2] Two networks had no peering policy or PeeringDB peering information (they had PeeringDB entries but the peering fields were blank). It appears that for many networks PeeringDB has taken on the role of peering policies as a means of expressing peering inclination and establishing arrangements.
Fifty eight percent of backbone providers have peering policies compared to 79% of BIAS providers, 66% of CDNs, and 83% of edge providers. Sorting the survey for the largest backbone transit providers as identified on Dyn's Baker's Dozen 2016 produced a consistent result of 55% of backbone providers with posted peering policies. These numbers are a little muddled because a number of the BIAS providers are also backbone networks and reportedly charge paid peering, such as AT&T, Centurylink, Time Warner Cable (Charter), Comcast and Verizon; these networks were classified as BIAS providers and all of them had published peering policies.
These findings reflect that all participants in the Internet network ecosystem, not just backbones, are actively seeking peering arrangements. Curiously, backbone networks had the lowest percentage of posted peering policies and yet backbone interconnection conflicts were the original impetus for the calls for published peering policies. Then again, as peering policies are statements of inclination to peer, and backbone networks are inclined to sell transit, not peer, the lack of posted peering policy could be consistent with their lower interest in peering. [3] A backbone network does not enter an IXP with the anticipation of peering with the other firms at the IXP; the backbone enters the IXP to sell transit.
Generally, networks inclination to peer is divided into four: open, selective, restrictive, and no peering. "Open" is the most liberal inclination where a network is generally willing to peer with anyone who can meet basic requirements (such as meeting the network at an interconnection point). "Selective" is the similar to open, but the selective networks is looking for some basic requirements before they enter into such relationships (for example, minimum traffic utilization). "Restrictive" is the more traditional inclination where a provider is only willing to entering into peering with "peers," networks that meet a relatively high bar of criteria. These inclinations can be regionally specific. A network may have an open policy in one geographic regional and a selective policy in another.
According to PeeringDB,[4] 35% of networks surveyed had open peering policies, 43% had selective, and 18% has restrictive.[5] Combining "open" and "selective," 78% of networks are interested and inclined to enter into a peering arrangement. This is consistent with what one might anticipate that peering is a beneficial arrangement, but the largest group of networks has at least some criteria for when the expense and logistics justifies the arrangement.
CDN's surveyed had 50% open policies and 50% selective policies. CDNs are in the business of delivering data; they get paid by edge providers for that service. Interconnecting with backbone and access networks facilitates the delivery of traffic and thus furthers their business model.
BIAS providers had 14% open policies, 64% selective, and 14% restrictive. BIAS providers are a mixed bag. BIAS providers that do not have significant backbones and are reliant on transit service to bring the full Internet to their customers have an incentive to avoid transit costs by entering into peering arrangements. Historically, access providers would have welcomed offers of settlement free peering as a means of defraying transit costs. However, large BIAS providers that charge access paid peering to CDNs have no incentive to agree to settlement free peering, increasing the settlement free capacity into their networks that edge providers could use instead of paid peering routes. AT&T, Centurylink, and Comcast have selective peering policies where Time Warner Cable (now Charter) and Verizon have restrictive peering policies.
A provider can have different peering policies based on the service it is providing. Verizon's general peering policy is restrictive whereas Verizon CDN Edgecast's peering policy is open.
Not surprising, educational and infrastructure networks generally have open peering policies. They are not in the business of selling network services; the goal is connectivity.
Words Per Policy
On average peering policies contained 679 words and 8 provisions. As would be expected, the open policies were shorter than the restrictive policies. Open policies had 451 words on average and 5 provisions; selective policies had 665 words and 9 provisions; and restrictive policies had 1183 words and 11 provisions. The shortest peering policy was Limelight's selective policy with 87 words. The longest policy was Time Warner Cable's (Charter) restrictive policy with 1701 words. The third longest peering policy was Charter's at 1404 words, even though it is a selective policy.
Green = Open. Yellow = Selective. Red = Restrictive.
The most common peering policy clause was the requirement for a 24/7 Network Operations Center. Eighty two percent of surveyed peering policies required this. It appears to be the highest priority in interconnection relationship that partners are able to promptly respond to and resolve networking issues. The next highest clause was the "no abuse" provision (no gateway of last resort, no default routes, no transit traffic in a peering relationship, no static routes),[7] which, likewise, is an assurance that the partner will behave in terms of routing.
| Clause | All Policies | BIAS | Backbone | CDN | All Policies Norton 2010 |
Percent Change 2017 versus Norton |
| 24/7 NOC | 82% | 100% | 86% | 63% | 89% | -7% |
| No Abuse | 74% | 91% | 43% | 63% | 64% | +10% |
| Minimum Required interconnection points |
67% | 91% | 71% | 63% | 46% | +21% |
| Consistent Routing Announcements |
64% | 82% | 86% | 50% | 75% | -11% |
| Minimum Traffic | 54% | 91% | 57% | 38% | 71% | -17% |
| Geographically Diverse Network |
41% | 73% | 43% | 38% | 46% | -5% |
| Filter Routes | 36% | 73% | 14% | 0% | 29% | +7% |
| Customers are not eligible to become peers |
36% | 45% | 43% | 0% | 71% | -35% |
| Potato Routing | 33% | 45% | 29% | 38% | 29% | +4% |
| PeeringDB | 31% | 9% | 0% | 63% | 7% | +24% |
| Traffic Ratio | 28% | 64% | 57% | 0% | 32% | -4% |
Minimum Interconnection Points
Twenty-six policies had minimum required points of interconnection. Three of the highest minimums were for peering on a global basis and included international POPs. The average for the rest was a minimum of three points of interconnection. Some networks were willing to peer even if the networks meet at only one point of interconnection. Of the domestic networks, Charter had the highest minimum required points of interconnection with nine for content providers.
There were a number of networks that required that a potential partner interconnect at "all" points of interconnection where both networks are present. These networks may be willing to interconnect with only a few POPs, but they want redundancy and Cold Potato delivery and thus want to interconnect anywhere where they both co-exist. While one might expect this to be a clause of CDNs, it was found in one backbone's policy and three edge providers' policies.
Traffic Ratios
Twenty eight percent of peering policies contained balanced traffic ratio requirements. These clauses were found in backbone and BIAS network policies. All of the BIAS providers that are known to charge paid peering had balanced traffic ratios as a condition for interconnection. No CDN, edge provider, infrastructure network, or educational network included a traffic ratio clause. The average requirement was 2:1 and the range was from 1.5:1 to 3:1. Three networks simply indicated that a balance traffic ratio was required without specifying a number.
In 2012, the OECD published a report that found that 99.5% of peering arrangements were without written contracts. This finding was on a global basis and was based on a per network measurement, not based on the amount of traffic carried by networks (in other words, the hypothesis is that larger networks, networks that carry more traffic, are more likely to require contracts, and represent a larger share of exchanged traffic).
According to PeeringDB, 26% of networks require contracts. None of the networks with open peering policies required contracts; 23% of selective networks require contracts; and 82% of restrictive networks require contracts. Consistently, none of the CDNs or edge providers require contracts where 75% of backbones require contracts. Of the top transit providers listed in Dyn's Baker's Dozen for 2016, every provider except Hurricane Electric requires a contract. Four of the BIAS providers known to charge access paid peering require a contract (TWC, acquired by Charter, indicates that a contract is not necessary).
Hot or Cold Potato Routing
Hot Potato routing is a traditional arrangement where traffic from the sending network is handed off to the receiving network at the closest interconnection point to the origin of the traffic. Hot Potato routing was common between peering Tier 1 backbones.
Cold Potato is the opposite, where the originating network carries the traffic as far as possible and hands it off as close to the receiving end user as possible. The sending network is responsible for long haul transport while the receiving network is responsible for local access transport. CDNs delivering traffic to access networks employ Cold Potato routing.
Thirty three percent of peering policies surveyed contained a Potato clause. Sixty nine percent called for Hot Potato routing; 31% called for Cold Potato routing. The Potato clauses did not fall out as one might expect; one might expect that BIAS providers want CDNs and backbones to carry heavy traffic as far as possible and deliver it as close to end users as possible in order to avoid network costs and improve end user experience. Yet sixty percent of BIAS providers that contained this clause called for Hot Potato routing; this reflects in part BIAS providers that own their own backbone networks. Charter, which owns the TWC backbone, called for Cold Potato routing in both of its peering policies. In the CDN group, Highwinds called for Hot Potato routing; the other two CDNs that had a Potato clause called for Cold Potato routing; most CDNs did not have a Potato clause (CDNs are in the business of quality delivery of content; one might expect them to have an incentive of avoiding congested backbones and improving performance by locating CDN servers as close to end users as possible). Only the backbone providers fell out as expected, with all networks that had this clause calling for Hot Potato routing.
Congestion
The recent interconnection disputes have left their mark on peering policies. Historically, peering disputes were between backbones (generally as backbones grew and tried to join the Tier 1 club) and lasted a few days (the value of network effect and providing service to customers exceeded the value of the dispute, creating an incentive for rapid resolution and getting back to business before customers switch to a backbone service that isn’t affected). Recent interconnection disputes have been between content networks and access networks and have been protracted.
Several peering policies indicate that partners must agree to avoid congestion (Fastly: "Maintain congestion free interconnection"; Yahoo!: "Maintain congestion free interconnection"; Blizzard: "maintain sufficient capacity to exchange traffic without congestion"; Microsoft: "sufficient capacity to exchange traffic without congestion", Frontier: "Peers must agree to resolve congestion issues"). Other networks include in the policy that capacity will be augmented at a specific benchmark. An industry norm is that capacity is augmented when utilization reaches 70%. Google and Microsoft include a clause that capacity will be augmented when utilization reaches 50%. Finally, Comcast includes a clause that partners will participate in capacity planning "and work towards timely augments as identified." Twenty three percent of networks surveyed that had peering policies included some clause to address congestion.
Divergence from Norton 2010 Survey
PeeringDB
A requirement that a potential interconnection partner use PeeringDB is the most significant increase from the Norton 2010 survey. Norton found that only 7% of policies required the use of PeeringDB. Norton observed "nLayer does not even suggest using it - surprising, since Richard Steenbergen from nLayer leads the peeringDB project." This survey found that 31% of peering policies require the use of PeeringDB (and 100% of surveyed networks have PeeringDB entries). The type of network requiring a PeeringDB entry showed interesting variation. Only 1 BIAS provider included this clause and no backbone providers. Conversely, 63% of CDNs and 60% of edge providers included this clause.
No Customers
The clause that had the most significant decrease and largest divergence from the Norton Survey was "no transit customers may become peers." Norton found that 71% of peering policies contained this clause while this survey found only 36%. It seems like this clause would only be of concern to those networks in the business of selling transit. This would be less of an issue for those providers with incentives to avoid transit costs or improve connectivity.
Observations: Peering Policies Dont Measure "Peers" Anymore
A "peering policy" is no longer a measuring stick by which "peers" are determined. By definition, "peering" is an interconnection arrangement between partners who exchange traffic from their own networks. Historically, "peering" was an arrangement between "peers" (equals). A "peer" was a network of roughly equal size that exchanged roughly balanced traffic. A "peering policy" articulated how a "peer" would be measured and determined.
There are no more peers.[8] Providers have diversified their business plans vertically and horizontally, and have merged and consolidated businesses. At the time of the Commercial Internet eXchange, Verizon, Comcast, AT&T, Charter (Time Warner Cable), and Centurylink did not have Internet backbones; now they are the top Internet network providers. Almost none of the original members of the Commercial Internet eXchange exist in their 1992 form. The big question of peering has evolved from an interconnection arrangement in the core between backbones to an interconnection arrangement on the edge between access networks and CDNs.
Since there are no longer "peers," a "peering policy" is no longer a mechanism for measuring "peers." Since there are not networks of "roughly equal size," the first of two necessary benchmarks, then the second benchmark "balanced traffic ratio" likewise no longer helps to measures equals. Instead, what was once a benchmark has become an interconnection condition. "Balanced traffic ratio" is now a condition that limits the amount of traffic that can be exchanged (in order for traffic exchange to be balanced, one network can send the other network no more traffic than it receives).
Peering continues to be an attractive interconnection arrangement. Smaller access networks that are reliant on transit have an incentive to avoid transit costs by entering into peering arrangements with each other, content providers, and edge providers. Content and edge providers have an incentive to directly interconnect with access networks, bypassing congested third-party transit providers. Large networks have an incentive to peer with other large networks, avoiding transit costs and providing quality routes for traffic. Direct interconnection through peering can result in efficiencies, quality delivery of traffic, and better management of traffic flows.
Peering policies reflect the evolution of the Internet ecosystem. Formerly a statement of peering inclination by backbone providers, now peering policies are posted by access networks, CDNs, and edge providers. The more peering furthers a business plan, the more inclined a network will be to enter into a peering arrangement. An access provider indicates that it will enter into a peering arrangement but only on the condition of cold potato routing and that traffic is delivered to specified interconnection points; a CDN indicates that, in order to further its business plan of high quality traffic delivery, it is willing to carry traffic as close end users as possible and openly enter into interconnection arrangements. Peering Policies have become statements of how providers will establish interconnection arrangements with non-equal providers.
Endnotes
[1] Contrast Aemen Lodhi, Natalie Larson, Amogh Dhamdhere, Constantine Dovrolis, kc claffy, Using PeeringDB to Understand the Peering Ecosystem, ACM SIGCOMM Computer Communication Review, 44(2), 20-27 (2014) (reviewing peering characteristics based on the full PeeringDB database. When Lodhi et al.'s results were sorted for the largest providers, their results were consistent with the findings of this survey.).
[2] Compare Lodhi at 22 ("We find that 93% of the top-100, 80% of the top-200 and 74% of the top-300 ASes from [CAIDA's] AS-rank were present in the [August 2013 PeeringDB] dataset, including all known Tier-1 and major Tier-2 ASes").
[3] See Jayne Miller, IP Transit vs. Peering: How a Network of Networks is Built, Telegeography Blog (Sept. 20, 2016) ("Large networks that are the major providers of IP transit have no incentive to peer with potential customers.").
[4] I did not contrast self reported peering inclinations in PeeringDB with inclinations articulated in posted peering policies in order to confirm consistency. However, the Using PeeringDB study did. Aemen Lodhi, Natalie Larson, Amogh Dhamdhere, Constantine Dovrolis, kc claffy, Using PeeringDB to Understand the Peering Ecosystem, ACM SIGCOMM Computer Communication Review, 44(2), 20-27, 23 (2014), http://www.sigcomm.org/sites/default/files/ccr/papers/2014/April/0000000-0000002.pdf ("we obtained the peering policy URLs of 50 networks in PeeringDB, and compared the policy seen on their URL with the policy mentioned in the PeeringDB record. In each case, the peering policy listed on PeeringDB (Open, Selective or Restrictive) matched the peering policy at that network’s policy URL.").
[5] Compare Aemen Lodhi, Natalie Larson, Amogh Dhamdhere, Constantine Dovrolis, kc claffy, Using PeeringDB to Understand the Peering Ecosystem, ACM SIGCOMM Computer Communication Review, 44(2), 20-27, 23 (2014), http://www.sigcomm.org/sites/default/files/ccr/papers/2014/April/0000000-0000002.pdf ("Of the 3392 ASes in the [August 2013 PeeringDB] dataset, 76% use Open peering, 21% use Selective, and 3% use Restrictive."). The Using PeeringDB study found that inclination to peer was inversely correlated to network size; larger networks were more likely to have restrictive peering policies. Id. at 25. This survey focused on the largest backbones, content providers, CDNs and BIAS providers and therefore the results reflect that methodology.
[6] Compare Aemen Lodhi, Natalie Larson, Amogh Dhamdhere, Constantine Dovrolis, kc claffy, Using PeeringDB to Understand the Peering Ecosystem, ACM SIGCOMM Computer Communication Review, 44(2), 20-27, 21 (2014), http://www.sigcomm.org/sites/default/files/ccr/papers/2014/April/0000000-0000002.pdf ("we find widespread adoption of Open peering among transit providers, which is counterintuitive given that transit providers prefer other ASes as their customers instead of peers. ").
[7] [William Norton, Don’t Abuse Peering, DrPeering] [Understanding Default Routes, IS-IS Feature Guide, Juniper Networks (May 20, 2010), ("A default route is the route that takes effect when no other route is available for an IP destination address.")] [Configuring Static Routes, CISCO Nexus 7000 Series NX-OS Unicast Routing Configuration Guide (June 14, 2017)] [Luc De Ghein, Specify a Next Hop Address for Static Routes, CISCO Technical Support, Sept. 2, 2014] [Configuring a Gateway of Last Resort Using IP Commands, CISCO Technical Support, Aug. 10, 2005 ("Default routes are used to direct packets addressed to networks not explicitly listed in the routing table. Default routes are invaluable in topologies where learning all the more specific networks is not desirable, as in case of stub networks, or not feasible due to limited system resources such as memory and processing power.")]
[8] There are Peering Fundamentalists who adhere to the tenant that "peering" is an arrangement between "peers." An interconnection arrangement that is not between peers, therefore, cannot be "peering," regardless of the routes exchanged. To them, the term "paid peering" amounts to heresy as it is implicitly a relationship between non-equals. As a member of the Reformed Peering Denomination, I view such talk as semantics. There is no platonic form to the word "peering." It means just what we chose it to mean – neither more nor less. It could be called "On-Net traffic exchange." It could be called "Pineapples." As long as we adhere to a term and a common understanding, we are good.
No comments:
Post a Comment