PCH
Home Resources Oldwiki Internet Exchange Point Policy Documents

Internet Exchange Point Policy Documents

Nearly all Internet exchange points define their participation, operational, and governance policies in an "IXP policy document." This document is typically relatively short, one or two pages in length, and written in plain language, not legal boilerplate. It's important to distinguish IXP policy documents, which define the relationship between the IXP and its participants (and concern mostly the data-link layer 2), from ISP peering agreements, which are between the participants, and do not involve the IXP (and concern mostly the network layer 3).

Note that in this document, we will use the word "participants" to mean the parties who are BGP peering at an IXP. The word "members," though widely used, does not adequately encompass the participants at the many IXPs that are not membership associations (such as commercially-operated, academic, and governmental exchanges), nor does "ISPs" sufficiently cover the diverse other sorts of participants (content delivery networks, research & education networks, DNS and other stand-alone servers, et cetera) that participate at most exchanges.

We will describe each point of policy that's either commonly found in IXP policy documents, or is commonly discussed and rejected in the drafting of policy documents, along with PCH's recommendation if we have one. Where wording is widely divergent between different policies that attempt to express the same intent, we will try to link to multiple examples.

Participation requirements

Organization type

Sometimes nascent IXPs attempt to constrain participation, in order to allow a specific constituency to uniquely capture the benefits of the exchange. Typically this plays out as ISPs attempting to preclude potential customers from connecting directly to the exchange, in the mistaken belief that this will keep them from peering, and cause them to purchase more transit. This is a fallacy, since the barrier to entry in setting up an IXP is very low, while the long-term attention required to operate a network that includes peering as well as transit is somewhat higher. Thus anyone who really wants to peer will do so, constructing their own IXP if necessary, but only people who really want to peer will bother; the vast majority of transit-purchasing networks will continue to only purchase transit, and not peer, because it is simpler for them to do so. Other examples of restrictive policies are ones which allow only corporations to connect (excluding private individuals, sole proprietorships, and partnerships), allow only domestic corporations or citizens to connect, allow only academic degree-granting institutions to connect, et cetera. PCH's strong recommendation is to NOT attempt to restrict the types of organizations that may participate in an IXP. Doing so greatly reduces the value that the exchange will create, and the degree to which it will attract broad participation and grow. Essentially all IXPs that initially attempted to impose organization type limitations on participation have since either removed the restriction, or experienced severely limited growth.

Licenses

A more subtle way of restricting participation in an IXP, or putting up barriers to innovation and new market entrants, is to require that IXP participants be holders of some local license. While this may seem like a tricky work-around that the regulator will not object to on anti-competition grounds, it fails to benefit participants for the same reason that any other restriction on participation does: it does not dissuade smaller domestic competitors, for whom peering is a business requirement, but it does dissuade international content networks, whose participation would greatly benefit all domestic eyeball networks. No major international network will bother to get a local license, so that they can peer at your IXP. Moreover, any such requirement runs parallel to local law in an unintended way; if the regulator wants to require IXP participants to hold a license, the law will say so. The IXP policy document need not second-guess the law. PCH's strong recommendation is that IXP policy documents NOT reference licenses or any other law. This avoids synchronization problems as laws are updated, as well as generally encouraging participation and growth.

Right reserved to disconnect

An important feature of any IXP policy document, though it may be implicit rather than stated up-front, is the right of the IXP operators to disconnect, and revoke the participatory rights, of any network that fails to follow the policies stated in the rest of the document. The degree to which an explicit clause stating this right is needed is typically dependent upon the number of participants, and how litigious the society the IXP is located in is. Examples PCH's mild recommendation is that this right be stated explicitly, so as to avoid later legal challenges.

Accession to the Policy Document

Some IXPs are governed by an /incorporated/ association, while others are governed by an /unincorporated/ association. An unincorporated association is a much more lightweight structure, which confers a number of benefits: no annual corporation and tax filings, no need to apply for not-for-profit status, et cetera. In the case of an unincorporated association, the policy document becomes the legal instrument that ties the association together. A participant in the IXP, from a legal perspective, is an organization that has /both/ physically connected to the IXP /and/ acceded to the terms of the policy document by affixing its signature to the end. The policy document is thus a binding legal contract between the full set of participants, and only parties who have both physically connected and signed the document may partake of the benefits of the exchange, i.e. route traffic across it. PCH's strong recommendation is that IXPs structure their policy document as a contract that may be executed in counterparts, such that any organization that wishes to participate in the exchange must agree to the terms of the policy document in a legally binding way before they may utilize the functions of the exchange, vote, et cetera.

Governance process

Meeting Freqency and Attendance

Most IXPs have one statutory annual meeting, at which the board of directors is elected and any major or long-time-frame issues are discussed. It is typically also possible for the board to call for an interim meeting, or for some significant majority of participants to call for an interim meeting. All participants are welcome at any meeting, and typically visitors or observers are welcomed as well. Some IXPs also invite speakers to their annual meeting, and have a keynote or panel or agenda of talks. PCH's recommendation is that IXPs hold one meeting each year, scheduled well in advance, defaulting to a regular and predictable day (i.e. the first Monday of March, or something similar), avoiding days used by other IXPs that have a significantly overlapping participant community. Meetings should include a summary of significant events of the past year, discussion of any upcoming events, an election, and generally a social event to encourage participants to get to know each other and develop peering relationships.

Governance Board

Most IXPs have a governance board, elected by the participants, and responsible for effecting the operation and business of the exchange. In essentially all cases, the board is uncompensated, and is usually required to do the work themselves, rather than delegating it to unelected parties. This structure is used to avoid "vanity" boards by ensuring that only people who have a strong interest in the success of the exchange will run, and only those deemed competent and trustworthy by their peers and competitors will win election. In most IXPs, the board seats are undifferentiated, and wind up mostly being filled by the CTOs and senior technical management of the larger participants. In a few IXPs, there are specific board seats designated for business or public relations roles; in these cases, each candidate runs for a specific seat. PCH's recommendation is that IXPs utilize an uncompensated (volunteer) board of three to five elected representatives, charged with the direct, undelegated, maintenance of the exchange.

Election Process

IXPs essentially all allocate one vote to each participant organization, regardless of how many ports, or how much capacity, each is using. Where multiple peering ASes are ultimately owned by the same controlling entity, that entity still generally only receives a single vote. Only L3 BGP peers are generally counted, which typically leaves parties like the owner of the facility in which the IXP is located without a vote, but without a conflict-of-interest, as well. Votes are held at (and remotely in conjunction with) the annual meeting, and votes of no-confidence may be called by a significant majority of the participants. PCH recommends one vote per participating organization.

Donor Recognition

It is common practice for an IXP to acknowledge donations received through a listing on a "donors" page, or in the case of significant donations, a small news item. When an IXP does so, it's also common practice for the listing or announcement to link back to the web page of the donor. However, this practice is generally true of non-profits the world over, and search engine optimizers (SEO) are aware of the practice and seek to exploit it. PCH's recommendation is that the IXP policy document specify that only donations amounting to a significant percentage of the IXP's annual budget have a link associated with the listing. So, a small donation can still be recognized on the donor web site, but the text would not have an underlying link that could be abused for SEO purposes.

Layer 2 participant technical requirements

Ethernet frame format

In the early days of Ethernet, there were several different versions of the framing, the formatting of the Ethernet packet headers, that were not entirely interoperable. The Ethernet frame format that has predominated is called "Ethernet Version 2", "DIX Ethernet" or "802.3", which are not entirely identical, but are interoperable. Glossing over ten years of vigorous debate in the 1980s, other frame formats that are no longer common are "802.2", "SNAP", "LLC", and "Novell 802.3 raw". Further exposition in section 4.1 of this FAQ. Although they are not a default, many of these are supported for backwards-compatibility on products that have long development histories, such as the enterprise builds of Classic IOS. Therefore it is PCH's mild recommendation that IXPs explicitly require Ethernet Version 2, DIX Ethernet, or 802.3 framing.

Ethernet data types

The Ethernet header contains a field that specifies what type of Layer-3 data is enclosed within the Ethernet header. Conservative IXPs tend to explicitly limit the types of Ethernet data to these three:

MAC address limitation

Media Access Control, or MAC, addresses are the unique identifiers of Layer-2 nodes on an IEE 802 network like Ethernet or WiFi, in the same way that IP addresses are unique identifiers at Layer-3. A Layer-2 end-node (that is not hosting virtualized elements) like a host or a router typically has a single MAC address, while a Layer-2 active device like a switch may possess, and also answer on behalf of, many different MAC addresses. Because the health of an IXP peering network is vital to many different organizations, yet also requires the active cooperation of many different organizations to maintain, distinctions between roles and responsibilities are critical. Specifically, a clear understanding is needed of who's operating routers, and who's operating switches, on the peering network. Because routers are Layer-3 (IP) devices, they typically cause Layer-3 (IP routing) problems; switches, which are Layer-2 (Ethernet) devices, typically cause Layer-2 (Ethernet switching) problems. When these expectations are muddied or confounded, enormous amounts of effort can be wasted in debugging and correcting problems. Conservative IXPs allow exactly one MAC address to be used by participants' equipment connected to a single IXP switch port. Less-conservative IXPs employ an IXP switch fabric extension policy (see below) to allow participants to connect an Ethernet switch to their port, thereby extending (and complicating) the IXP switch fabric, but allowing more participants to connect in more diverse ways. PCH's recommendation is that MAC addresses be limited to one per port except on switch fabric extension ports, and that IXPs employ a cautious switch fabric extension policy to allow more rapid and diverse growth.

Maximum Transmission Unit

The Maximum Transmission Unit, or MTU, is the maximum size of an Ethernet packet that can be recognized by a device. All devices within a single Ethernet network must share the same MTU, or large packets will be discarded by devices with smaller MTUs. 1500 bytes was a standard MTU for Ethernet for a long time, but poses significant performance limitations on modern gigabit-and-faster networks. Because an IX switch fabric has routers connected to it under many different administrative authorities, it's very difficult to change the MTU of an IXP peering subnet once it's been established. A higher MTU has very mild disadvantages to flows of small packets, while giving large TCP flows as much as a 20% throughput advantage, and increasing the maximum efficiency of the network. Where parallel 1500-byte and 9000-byte VLANs exist at the same IXP, relatively complex configuration may be necessary on participants' routers to ensure that the BGP session over the 9000-byte VLAN takes routing priority over that on the 1500-byte VLAN. Examples.: PCH's recommendation to newly-formed IXPs is to require all IX participants to explicitly configure a 9000-byte MTU on their router interfaces, facing the IX switch fabric. PCH's recommendation to preexisting IXPs is to establish a new VLAN with a 9000-byte MTU, and begin promoting its use to participants.

Switch fabric extension policy

A "switch fabric extension" or "switch interconnection" policy is a set of rules that govern the extension of an IXP's switch fabric to include switches under the control of different managerial authorities. In many cases, an existing IXP fails to serve some constituency (for example, wealthy networks that demand high uptime, or content networks that demand large and diverse colocation). This can be addressed in one of three ways: a new IXP can be formed to serve the needs of the unaddressed constituency; the existing IXP can be reformed to meet their needs; or the existing IXP can be extended to also serve their needs. Creating a new, unconnected IXP in the same geography is wasteful (it drives up the average cost per route learned), so this is not a desirable outcome. Reform of an existing IXP is often politically or economically infeasible, since the reason it fails to serve some constituency is often inflexibly enshrined in policy or cost models. Therefore the most palatable way of serving the diverse needs of radically differing participants is often through the establishment of additional peering locations, served by new switches, in new facilities, with different management. The key, however, is to ensure that the new group of peers are able to peer transparently with the existing peers, across a single unified switch fabric. This ensures that both new and old groups capture maximum value. The danger of a too-open switch fabric extension policy is that poorly-managed switches will accumulate, and layer-2 problems will propagate across the whole exchange, without any clear responsible and enabled party for their resolution. Examples.: PCH's recommendation is that IXPs have a clearly-stated and nondiscriminatory process by which anyone may apply to extend the IXP switch fabric to a new location, and that the proposal be judged solely on its technical merits and the apparent competence-to-execute of the proposer.

Anti-Congestion

Some IXPs have a policy aimed at eliminating routine congestion and queue-overflow packet loss, by mandating that congested ports be upgraded by their users. Typically, the policy will require that the participant on any port which reaches a certain level of peak traffic, often 85% or 90%, some number of days in a row, often three, upgrade their port to the next-higher available speed. For example, a participant on a 1gbps port would be required to upgrade to a 10gbps port on the third day that their peak traffic exceeded an average of 85% utilization in any five-minute interval. Examples.: PCH's strong recommendation is that IXPs require mandatory upgrades of congested ports. 85% and three days are perfectly reasonable parameters for this requirement.

Privacy

Many IXPs have a policy prohibiting the "sniffing" of network traffic that crosses the IXP switch fabric, and prohibiting the connection of devices with Ethernet interfaces in "promiscuous mode." These policies are intended to preserve the privacy of network traffic, and discourage spying, espionage, et cetera. While the policy is admirable, violations are undetectable, which makes it an unenforceable policy, and intelligence services and law enforcement agencies routinely require IXPs within their jurisdictions to provide interception access, which would violate this policy if the agencies were subject to it. PCH's mild recommendation is to not include this policy, since it's essentially unenforceable.

Broadcast traffic

Examples.:

Static configuration

A common requirement is that the speed and duplex settings of any 10base-T and 100base-T connections be statically configured, rather than left on "auto-detect." Auto-detect is not infallible, and can sometimes create problems which unnecessarily take IXP operator time to resolve. examples: PCH's recommendation is to include this requirement, as it makes the origin and resolution of a common class of problem more explicit and obvious.

Layer 3 participant technical requirements

Mandatory multi-lateral peering

A "Mandatory Multi-Lateral Peering Agreement" or MMLPA, is a requirement that all IXP participants also peer with each other, either in a full mesh, or using a route-server. ISPs large enough to have legal departments will essentially never allow themselves to become party to an agreement with unlimited future signatories. Large ISPs also tend to choose to use route-servers less frequently than do small ones. The net effect of making multi-lateral peering mandatory, rather than optional, is to dissuade large ISPs from participating at your IXP. This greatly reduces the amount of bandwidth produced by the exchange, and thus reduces the value of the exchange to all participants. PCH's strong recommendation is to NOT require (make mandatory) multilateral peering. In other words, it is our recommendation that you allow each participant to decide individually whether they wish to peer with all, or only some, of the other participants, and whether they wish to do so using bilateral or multilateral agreements.

BGP4 protocol requirement

Many exchanges include a simple one-line requirement that all participants utilize BGP4 or any future successor protocols for their peering route-exchange. This requirement does no harm, but has not served any real function since the very early 1990s, when ISPs still considered using other routing protocols across IXPs. PCH's mild recommendation is to NOT bother including this requirement.

IPv4 and IPv6 transport for BGP peering sessions

IPv4 routes and IPv6 routes may be exchanged over IPv4 transport, IPv6 transport, or both. For historical reasons having to do with the very gradual introduction of IPv6 by router vendors, IPv6 routes have often been carried over separate IPv6 transport, between separate BGP sessions (previously running on separate physical routers). In the present, we have three choices: carry both IPv4 and IPv6 routes over IPv4 transport; carry IPv4 routes over IPv4 transport and IPv6 routes over IPv6 transport; or carry both IPv4 and IPv6 routes over IPv6 transport. In the very long term, IXPs will no longer have IPv4 peering subnets. Consequently, in the future, there will come a time when all routes (whether or not that includes any IPv4 routes) will be carried over IPv6 transport. Therefore, from a complexity and management point of view, it would make sense to begin carrying both IPv4 and IPv6 routes over IPv6 transport with each peer that can accommodate that, as soon as it does not require any significant additional work. In this way, we will progress towards the future that we all agree is necessary. PCH's mild recommendation is to suggest that participants carry both IPv4 and IPv6 routes over a single IPv6 transport session, and to configure any route-server that way.

Pointing default

Most exchanges include a simple prohibition against "pointing default" routes at other participants, or otherwise sending traffic to them that was not solicited via a routing announcement on the recipient's part. Although participants should protect themselves by using appropriate IP address prefix filters on inbound traffic, this prohibition against abuse is reasonable, and gives IXP management a policy basis upon which to disconnect miscreants. PCH's strong recommendation is to include a prohibition against pointing default: "Participants may not direct traffic to others unless it has been solicited via the announcement directly to the participant of a matching prefix."

Next-Hop-Self (IXP side)

Note that there are two different policies related to Next-Hop-Self. First, a policy related to pointing default requires that if any participant provides transit between two other participants across the exchange, that they set the BGP "next-hop-self" parameter in their routing announcements. As an example, if participant A advertises a peering route to 1.0.0.0/8 to participant B, and participant B then provides that route to participant C (//transiting// participant B's network) this policy would require that participant B actually receive said traffic before retransmitting it to A. Because this causes the same traffic to cross the switch fabric twice, it is inefficient, and is thus not the default behavior of the BGP protocol, which would ordinarily cause B's advertisement to C to show A as the "next hop," directing C to send traffic directly to A, without B ever having to carry it. Use of this default property of BGP has been thought to be abusive by many IXP participants in position A. PCH's mild recommendation is to not worry about this policy, since it's punitive against party B, for behavior that's under the control of party A.

Next-Hop-Self (Participant side) and the redistribution of the IXP peering prefixes

Separately, IXPs need to consider the propagation of the IXP peering prefixes. Since anyone who has a route to the peering prefix can directly address any device attached to the IXP, regardless of where they are, routing redistribution of the IXP subnets makes all routers at the IXP vulnerable to attack. In a default BGP configuration, it's necessary to carry routes for the next-hop addresses of all of your peer routers throughout your AS. If participants set Next-Hop-Self on their inward-facing iBGP sessions, they do not need to carry the IXP peering subnets in their internal routing. PCH's strong recommendation is to disallow the propagation of IXP peering subnets in any routing protocol, and to suggest setting Next-Hop-Self towards internal peers as a mechanism for achieving this: "Participants may not propagate the IXP peering prefixes in any routing protocol. It is suggested that participants set Next-Hop-Self towards peers within their AS, in order to avoid having to carry the IXP prefixes as next-hops for their BGP peers."

Published under the Creative Commons Attribution-ShareAlike (CC BY-SA) License, which is included by reference.