« BACK  |  PRINT

RS

FRONT PAGE CONTRIBUTOR

Comcast did the right and fair thing cutting off Level 3

There’s a lot of heat and unfortunately not enough light being produced online with respect to the recent decision by Comcast to cut off its free peering deal with Level 3, thanks to the throwing around of the name Netflix. People like Netflix and want to know what’s going on. I even have a Netflix streaming-only account. So I’ll try to explain here what’s going on, why it matters, and why we need to keep the government from intervening in favor of Level 3 and Netflix.

Firstly, I know what the question is that many will need answered: What is peering? To understand what peering is, we must first remember that the Internet is a series of networks. We all have our networks at our homes and businesses. Those networks connect to ISPs. Those ISPs then connect their networks to each other and to ‘backbone’ providers. Those backbone providers also connect to each other. The further along the chain we go, the more of the Internet’s functioning depends on those connections. Backbone providers would not have a useful service unless their networks gave access to the entire Internet.

So what the big boys do is create peering agreements with each other, where they agree to connect to each other for free, on the grounds that the data will travel both ways, benefit both sides, and create value for all involved. Firms have to be careful though to ensure that their peers are actually trading equally and fairly, creating benefit for all instead of just leeching. That is why ISPs make contracts that specify how the peerings work, and document their policies and practices of peering. Level 3 spells it out and so does Comcast.

So what happened that was the big deal?

Comcast ended its peering agreement with Level 3. So much of the coverage online is breathlessly biased against Comcast on this (see MSNBC for an example), but it really is a simple business decision, fully in line with the principles and policies they published all along and that were surely documented in any formal deal between Comcast and Level 3. Comcast directly says more on the matter.

Note that this termination is not unprecedented. Back in 2005 Level 3 did the same thing to Cogent that Comcast did to Level 3, and for the exact same reasons. Sureel Choksi, then an executive VP at Level 3, said at that time:

…In order for free peering to be fair to both parties, the cost and benefit that parties contribute and receive should be roughly the same. The previous arrangement with Cogent was a contractual agreement that, when entered into, met that criteria.

“Over the last six months, our operating subsidiary has assessed all of our relationships to determine whether or not settlement-free peering is still appropriate. We determined that the agreement that we had with Cogent was not equitable to Level 3. There are a number of factors that determine whether a peering relationship is mutually beneficial. For example, Cogent was sending far more traffic to the Level 3 network than Level 3 was sending to Cogent’s network. It is important to keep in mind that traffic received by Level 3 in a peering relationship must be moved across Level 3′s network at considerable expense. Simply put, this means that, without paying, Cogent was using far more of Level 3′s network, far more of the time, than the reverse. Following our review, we decided that it was unfair for us to be subsidizing Cogent’s business.”

All of the above is precisely what Comcast would and does say about its peering arrangement with Level 3, only it’s Level 3 that’s the leech. So, Comcast did what Level 3 has proven it would do in the reverse situation, and Comcast terminated the deal.

So the next question is, what does Netflix have to do with it? Well, that’s easy: the reason Comcast saw an imbalance in the traffic with Level 3 was that Comcast users were streaming Netflix movies via Level 3, who had just made a new deal with Netflix to handle that traffic. Essentially, Netflix and Level 3 made their deal in the hopes that they could max out on Level 3′s peering agreements and ship off all their traffic without having to pay for it. Comcast, having a bottom line to watch out for, saw this and pulled the plug. Note that Comcast isn’t charging its home users extra to see Netflix. There is no toll booth. Comcast is just saying that if Level 3 wants to have an imbalanced traffic relationship with Comcast that they’re going to have to pay.

Level 3 got greedy. Level 3 went after Netflix without looking to keep its traffic balanced, and expected to spread those costs out to its peers. That’s not fair to anyone, and if they got away with it the result would truly mean higher Internet costs for all of us at home, including all Comcast home and business users.

Investors are high on Netflix and there’s no reason to think they shouldn’t have to carry their own weight. Ultimately I believe the burden is on them to spread out their traffic among a number of providers to ensure that the peering arrangements don’t continue to break down in this way. Or alternately they could bypass the middlemen and deal directly with the Comcasts of the Internet themselves and get dedicated deals going to ensure that paying Netflix users with paid Internet connections can get good video streams going.

Unfortunately instead of letting the market sort this out, forcing Netflix to innovate and allowing Level 3′s business partners to demand fairness, the FCC might act to impose broad bans on new business models and instead attempt to socialize the Internet backbone. Note that this is different from the possibility in December of broad action against ISPs under the banner of Net Neutrality. This a separate set of regulations, though rhetorically the proponents may cite the same ‘principles.’

It turns out Digital Society has been covering this for a while. Here’s a post from almost one year ago which, in the models George Ou discusses, is directly on point to the issues Netflix faces today. Ou returned to the topic last month and while it discusses Google, insert Netflix where it says Google or Content Provider to follow along.

The Internet is a marvel of engineering and overall does a remarkably great job at shipping vital communications around the world in a timely manner. Seriously: at no other point in history has data gone back and forth from virtually any two populated points on the globe with such little cost, redundant reliability, and outstanding speed. However new services like Netflix streaming push the limits of the network.

In the past we started out sending around text-only data. Then we started shipping images. Later we started including short video clips and animations. Over time the size and quality of those videos have gotten bigger and better. Netflix is at the higher end of the size and quality scales, but is pushed way outside the envelope in the sheer volume of video it puts out. They’re approaching one quarter of the Internet traffic at night in America.

Netflix must innovate to ensure they can continue to do this and keep their business model alive while being a good Internet citizen. Their business partners online must be able to do the same as well in order to deal with the flood of movies. FCC must not intervene and interfere with the necessary engineering and business innovation needed here.

COMMENTS

  • http://impudent.edublogs.org/ kyle8

    thank you for helping us understand this issue.

  • america1st

    Thanks, Neil. You have edjumacated me on some of the electronic back office stuff and provided the probable cause of the recent 10% increase in my Netflix charges.

  • http://www.criterionchemical.com Chemical Sam

    and a good explanation is worth a year’s worth of figuring it out sometimes.

  • scipio62

    Thanks for the real explanation of this issue.

  • http://lazarusreport.blogtownhall.com/ Tom Lesser

    This is the 21st century equivalent of the railroad arrangements in the 19th and 20th century (and still). Railroads own some of their freight cars. So let?s say a Union Pacific box car is filled at a company along the UP?s line, but its final destination is on a Norfolk Southern railroad line. The box car travels as part of a train along the UP line until it must be switched to NS train to reach its destination. NS keeps track of the number of day the UP box car is on its line (until it returns to the UP or some other railroad). NS must pay UP for the use of the box car. At the end of the year when the railroads settle up these fees, for many railroads it is a wash; using the example above NS and UP owed each other about the same amount. For a smaller railroad, if these fees go too high (if a car is on its line too long), they must either raise the rates they charge customers, renegotiate with the larger railroads, or go out of business. If they raise their rate, their customers may switch over to trucks.

    Comcast can: raise its rates to its customers, renegotiate with Level 3, or lose money.

  • http://www.hakubi.us/ Neil Stevens

    Nice pull from history.

  • http://perlstalker.blogspot.com/ perlstalker

    Don’t forget that Comcast’s customers are paying for so much bandwidth to access the Internet. All of the Internet. The bandwidth that Level 3 and Netflix are using on Comcast’s network has been paid for once by Comcast’s customers. Now Comcast wants to charge Level 3 for those same bits.

    If Comcast isn’t making enough from it’s customers to pay for the bandwidth they are using, then perhaps it’s time they another look at their pricing model.

    Netflix did innovate by offering a service that had not been offered before. ISPs like Comcast want to shut it down because it breaks the pricing model they’ve used since the dial-up days. In Comcast’s case, it directly competes with their cable TV service so they have even more reason to try to kill off Netflix.

    http://perlstalker.blogspot.com/2010/11/comcast-works-close-internet.html

  • peg_c

    Having worked in IT and with bandwidth and connectivity issues, I understand this perennial problem now exacerbated by streaming movies and heavy multimedia, but your explanation is excellent and really something everyone who uses the Internet should read. I have friends who brag that they cancelled cable and get all their TV shows and movies free or virtually free, and they don’t get that they are hogging the resources without paying much for them. There is no FREE. Libs especially don’t get this (besides not getting the business/profits angle of it). I also don’t mean this in any collectivist sense but we do share bandwidth and hogs ruin it for everybody.

    I use Amazon On Demand once in a while (and pay for it) but don’t stream much other than occasional Youtube use. Streaming heavy multimedia, if we all did it, would result in the world’s biggest traffic jam and no watchable media for anyone. As it is, many of us live with data networks that are chronically slow and overloaded. Where I live, I can’t even watch a Youtube video in the evening – after 5 minutes it’s still buffrering. The network is clogged (my neighbors are all probably streaming Netflix!).

    Stuff will always expand to fill the space allotted and networks certainly adhere to this rule. And the heaviest users should pay the most, to fund fatter pipes and better connections. If we all cancelled our cable and satellite services to get “free” TV and movies, believe me in no time we’d be paying as much if not more than we are now. The companies are not stupid and they know what’s going on. No one is in this out of charity. And Level 3 proves that what goes around comes around. Taxing a peer network to the max regularly without paying for it seems just plain evil.

    Just keep government OUT of it…

  • http://www.hakubi.us/ Neil Stevens

    So what do you propose?

    Should Comcast be required to allow any ISP to free ride on the Comcast network and get free peering?

    You’re dreaming here. You seem to have no grasp on how the Internet actually works. Level 3 charges its customers. Comcast charges its customers. The only fair solution to peering between Level 3 and Comcast, is that if their interchange isn’t fair, that the excess be paid for.

    That’s all Comcast asks.

    But you have no counterproposal. You just whine based on lies told by the radical left that Comcast is blocking traffic from Netflix. That’s not true and you supply no evidence to back up your implication.

  • congressworksforus

    Why should a non-Netflix-using Comcast customer be subsidizing a Netflix-using customer?

    I’m all for the consumer, but the actual customer needs to be paying the right amount, and not relying on others to pick up a portion of the tab.

  • Lina Inverse

    I’m working up a more detailed diary entry, but for now this posting on the North American Network Operators’ Group (NANOG, i.e. the people who do this for a living) covers part of what’s wrong with Comcast’s action in trying to create a new business model.

    Although I think we’re all agreed (including the author of the above) that going to the government for resolution of this conflict is the worst possible path.

  • Lina Inverse

    The previous link was to the beginning of the discussion (which is quite worthwhile and was started by an employee of Akamai, one of the companies that was previously delivering Netflix content to Comcast). Here’s the correct posting.

  • Lina Inverse

    This is all a question of who pays where. It’s impractical right now for the Netflix customer to pay Comcast a distinct charge for each high bandwidth content provider he is using (because that would require Comcast to make an arrangement with each high bandwidth content provider, of which there are many (in the long term, middlemen could be set up like ASCAP or the Copyright Clearance Center)).

    Instead he pays Netflix, who pays Level 3 (L3) for the bandwidth part (which is much less than what Netflix has to pay the studios).

    The question then becomes how do these costs get split between Content Delivery Networks (CDN) and/or backbones like L3 and residential broadband ISPs like Comcast. Comcast handsomely charges their residential customers for “Internet service” (with new price increases just announced) and caps their total use per month.

    Comcast is trying to create a new business model for companies like Netflix who not coincidentally compete with Comcast’s Video On Demand (VOD) service, the bandwidth of which is not charged against a customer’s monthly cap.

    If Comcast wasn’t a monopoly granted by local governments and its only broadband competitor wasn’t a telephone company that was granted a monopoly by the Federal government along with the states, we could trust the free market to work this out.

    It all comes down to who has the most raw power; for a mirror of this new Comcast business model, look at ESPN360 (now ESPN3). Individuals can’t buy subscriptions to it, the ISP has to pay ESPN. This works because ESPN’s content is in such high demand.

  • Lina Inverse

    Add to A and B a third railroad C (Cogent), which is using B (Level 3) to get a boxcar to a customer of A. This third railroad is using B purely for transit, i.e. A<-B<-C and B must get paid by A and/or C since it’s not going to do that for free.

    In the ’90s, Internet traffic ratios between backbones were ideally close to 1:1and was a basis for “free” (no settlement) peering. If a backbone like C was using B to get to A (and the reverse), this would be detected by the ratio diverging from 1:1 and B would demand that C pays money for transit to A or sets up peering directly with A so that B is no longer transiting packets for C a no cost.

    That gets a bit confusing so let me try this:

    AC, each has a mutually satisfactory arrangement with the other (nowadays if A is a residential broadband ISP and C is a backbone or CDN, the ratio is not going be even since by definition the ISP’s customers are more consumers of bandwidth than creators).

    ABC. B must get payment from A and/or C, otherwise it’s transiting packets between them for free, which it can’t afford. That’s what I heard happened with the Level 3 and Cogent dispute.

  • Read Chesterton

    Especially considering that railroad right-of-ways are common paths for today’s fiber optic communications infrastructure (The SPR in Sprint Network stand for Southern Pacific Railroad.).

  • Read Chesterton

    had the industry fully embraced the telecom standard ATM in the 1990′s rather than high speed ethernet. The network bottle-necking due to video bandwidth demand would have been mostly avoided.

  • Lina Inverse

    And paying for it a la carte.

    The economic battle between circuit switching and packet switching, between reliable “smart” networks and “dumb” endpoints vs. best effort dumb networks and smart endpoints has been decisively concluded. Best effort dumb networks are much less expensive and if you make them fast enough they by and large address the quality of service issues that the smart network people brought up (witness the success of Skype).

    It’s also worth pointing out that ATM is a horrible standard:

    The computer centric US wanted 64 byte packets, the telephone centric Europeans wanted 32 byte packets. A political compromise of 48 was reached, with 5 byte headers (10% overhead!!!). (Wikipedia appears to have good coverage of this.)

    The real test of a protocol like this is what happens in the face of congestion. If you want to carry TCP traffic over ATM, dropping a small 53 byte cell requires retransmitting an entire 1500 or so byte IP packet.

    Of course, if we’d adopted your prefered ATM over TCP/IP model, that wouldn’t be an issue … but we’d be paying for every connection to every data source we were using. And we’d be doing a lot less of it, people and companies prefer constant level billing, not variable charges and especially not surprises.

    That, and all the friction it requires, is the other reason why we’ve adopted this best effort, dumb network, packet switching solution. It’s less than perfect, but it’s cheap.

  • Lina Inverse

    [Internet peering] Ratios are an artifact of the ?90s

  • Lina Inverse

    Let’s try again, or just go to my diary: Internet peering Ratios are an artifact of the ?90s

    (I do wish commenting had a preview button….)

  • The_Gadfly

    ISPs and backbones to start charging according to usage instead of on an unlimited basis.

    I understand how the argument works when you look only at the Comcast-Level 3 interaction and your analysis works on that level. But there is also the level of the Customer-Comcast-Netflix relationship. The problem here is that although it is unfair to Comcast to subsidize Level-3, it is also unfair to the consumer who purchased their internet service through Comcast and expects to be able to use that connection to stream Netflix to their home because Comcast promised UNLIMITED access to the internet. What is needed is a system that is fair to everyone involved in the chain.

    Looking at the railroad example from a slightly different angle, you can make the argument that what is happening is more akin to Railroads charging high fares on routes where there was no competition to subsidize low fares on routes where there was competition in order to undercut the competition. In this case Comcast is undercutting the competition by preferring their On-Demand system to the Netflix system.

    I think part of what needs to go away are the free peering arrangements. The backbone companies need to ink deals with explicit charge rates for traffic on their networks. Whether the companies actually exchange money in any given period based on actual usage or whether they simply work out the difference between the two and then settle is up to them. I think the other part that needs to go away is unlimited internet service. The replacement might be something akin to a cell phone plan with a certain base amount of usage included in the price and charges if you exceed that. Most cell phone users don’t exceed their base plans, or select base plans so they don’t exceed it. In fact, this might even make a good basis for the backbone peering arrangements.

    “Free” always causes disruptions in the market even when its the vendors who are promising the “Free” stuff.

  • http://www.hakubi.us/ Neil Stevens

    To suggest it go away is… not in line with how the Internet has always worked.

  • The_Gadfly

    I believe we are very near the point at which the choice is between giving up the way the Internet has always worked in favor of real pricing to let the markets decide, or having the net neutrality freetards land the heavy hand of government control on the internet. Trying to keep free peering and unlimited home use tied to the markets may work out as well for us today as keeping slave ownership tied to States Rights worked out for the South.

  • Lina Inverse

    One of the big ones is that at this level the costs for companies like Comcast and Level 3 (L3) are not based on the traffic that passes between them. It’s based on the size of the pipes and interconnections between those (routers), those are always on 24×7. Intense metering of all these interconnections and resulting settlements billing is expensive, is thoroughly detested by customers who want simple and predictable bills and in general just isn’t done unless available bandwidth is severely constrained (i.e. in the wireless world).

    It made sense in the bad old days when there were a limited number of long distance interconnections between cities and metered pricing helped to match demand to availability, but we’re way past that sort of thing in the wired world.

    So, as traffic increases, all these companies have to build out their networks and make various parts of them bigger (conveniently you can generally change the connections to fiber and coax without changing the cables and when backbone fiber is laid many “dark” fibers are included for future growth, since the fiber itself isn’t all that expensive compared to everything else).

    So Comcast and Level 3 have been beefing up the pipes and routers between themselves, Comcast has been upgrading its residential customers to DOCSIS 3.0 modems. And unfortunately for them they also have to run new upstream coax: due to the head end noise issue the architecture requires more upstream coax even though it provides less upstream bandwidth to customers. (DSL has a similar but much less severe head end noise problem.)

    So the question is, who pays for what and where? In this diary posting I address the ratios red herring (e.g. why the ’90s business model of roughly equal traffic going both ways no longer applies). If you accept that, Comcast’s explanation doesn’t hold water and something else is going on.

  • Lina Inverse

    I might agree with you if this was truly a free market situation, but when residential broadband (the “last mile”) is at best a choice between two government granted monopolies then the heavy hand of government control is already in play.

    Which is not to say te control can’t and won’t get worse even with the best of intentions. But to dismiss all proponents of something that’s called “network neutrality” as freetards is unproductive and does not match history.

    This wasn’t something ginned up by the Left or any political entities as such. It was started by a fanatically obtuse quote in late 2005 by AT&T CEO Ed Whitacre about extracting monopoly rents from companies like Google, Yahoo and Vonage for access to AT&T’s government monopoly residential broadband customers. And at the same time some of these companies are offering services that compete with AT&T’s.

    In this case, it’s Netflix vs. Comcast’s Video On Demand (VOD) offerings … with the latter not counting against customer monthly usage caps, strangely enough.

  • http://www.hakubi.us/ Neil Stevens

    Well they put it in their policy. They’ve left it in their policy.

    So it’s your assertion that their policy doesn’t make sense and that there’s some secret, shady reason that they’re *not* ignoring their written policy.

  • Lina Inverse

    I find it … awkward to use the word “policy” to describe the relationship between the countries’ largest residential broadband ISP and largest backbone; they’ve got a relationship that’s more complicated than the posted policy.

    Per my posting I find it unbelievable to interpret:

    Applicant must maintain a traffic scale between its network and Comcast that enables a general balance of inbound versus outbound traffic.

    As meaning equal traffic between Comcast and classical backbone providers, because for technical and customer requirements Comcast has engineered its network for asymmetrical traffic and having backbone providers dump a lot more bits into its network than Comcast sends to them is the only thing that makes sense for a residential broadband ISP.

    The NANOG discussion I’ve previously referenced suggests it simply comes down to ego and power, i.e. Comcast is doing this because they can. Others have suggested that Level 3 rolled over because the amount of money demanded wasn’t large.

    Comcast does have a point that previously Netflix was using Akamai and to a lessor extent LimeLight Networks as Content Distribution Networks (CDN) and they pay Comcast to put their endpoints connecting to Comcast in mutually convenient locations. Comcast is of course not happy to lose a (we assume significant) fraction of those payments and said as much in their blog.

    A counter-argument is that major amounts of streaming video delivered to broadband residential customers has become a “new normal” in what we call “Internet service” and that Comcast may be trying to create a new business model, a mirror of the EPSN3 one (where ISPs and not customers pay the content provider). If so, a lot of people don’t think that’s a good thing; if it’s a government granted monopoly rent then I would hope none of us think it’s a particularly good thing.

  • http://www.hakubi.us/ Neil Stevens

    So your claim is that Netflix’s previous partners were entering into paid peering with Comcast, then Netflix’s new partners did NOT enter into paid peering with Comcast, but it’s not as simple as that?

    Come on, get serious.

    Level 3 was leeching off of Comcast, and Comcast wouldn’t have any of it.

    It’s that simple.

    Let’s be frank: You’re a Democrat who’s here to shill for greater regulation, aren’t you?

  • Lina Inverse

    You?re a Democrat who?s here to shill for greater regulation, aren?t you?

    I was “rooting for Nixon in short pants” as I like to put it. My first memory of an election was 1968; I knew from my parents it was very important but don’t remember why they said so. A bit of a nail biter with Wallace in the ring; I don’t think I watched the ’90s election on TV so that’s the only time I remember a 3 way, 3 color split. And even after knowing all of Watergate I would still have chosen him over McGovern, not that he was particularly conservative….

    Anyway, seeing as it’s obvious that disagreement with the party line on Red State is taken as bad faith trolling, I will stop wasting everyone’s time.

  • Read Chesterton

    but here we are looking at the equivalent of border skirmishes between backbone providers as they try to keep up with ever increasing demand for high volume video based bandwidth, which ATM will always be better at sorting out.

    It’s all academic anyway…. you are right that IP switching ain’t goin’ anywhere. But I always believed that the expectation that global IP bandwidth capacity would keep up with demand ad infinitum would get us exactly where we are today… on the verge of paralyzing regulation as big business throws in with the government to avoid competitive disadvantage.

  • rtg74

    Normally I would say let the market work its magic. If Comcast terminates the peering agreement, that means that all data from Level 3 cannot be accessed by Comcast users. In a free market world, if an end user wants access to this Level 3 data, he could find a new ISP. What does a user do when Comcast has a monopoly in the user’s area? He is essentially stuck. Some communities will not allow new ISPs to install service and Comcast is not required to allow sharing of their physical plant. I know… until recently I was one of those people. When the monopoly restriction was lifted (after being fought by Comcast) magically customer service went up and prices went down.

    The problem is that these ISPs want it both ways. They want regulation to “protect their investments” and they want the free market to decide peering agreements.

    The best solution could be to completely open it up to the free market and let the market decide what is best.

  • http://www.flaliberty.org scorpio0679

    But what exactly is the problem with what actually, in reality is going on? based on what Neil reported, L3′s arrangement with netflix is using unbalanced amount of Comcast’s bandwidth, so Comcast pulls the plug and demands to renegotiate their agreement.

    Reminds me of the whole Fox/Cablevision and Fox/Brighthouse dispute over how much Fox was charging for access. Yeah, it caused some hand wringing when people couldn’t watch their baseball games or whatever it was they were trying to watch, but ultimately they worked it out without bureaucrats stepping in.

    It seems to me that what you are stealthily suggesting is that the government regulate away the current ‘peering’ arrangements and require a new way of doing business. Why not just leave it alone and if it gets to the point they need a new arrangement, well then, won’t it happen freely and voluntarily?

  • http://www.flaliberty.org scorpio0679

    Right on the money.

  • http://www.flaliberty.org scorpio0679

    I pointed out elsewhere that Verizon is trying to expand its FiOS network but it being checkmated everywhere. Why not let other companies lay their own lines and compete? For one, it would alleviate traffic since now you have the same customer base competing over two or more ISP networks . . . and two, you inject market based competition into the mix, which will inevitably make things more efficient.

    I am very suspicious of these people who go on and on about the need for the government to come in and regulate more because its a monopoly, blah blah blah . . . . fix the monopoly problem, don’t build upon in.

  • http://www.800cart.com Ron Robinson

    pls contact me – rsr(at)cartsupport.com – interested in determining what happened with your precinct committeeman filing in NY in July – party.procinct.net for discussion, OK?