Archive for the ‘technology’ Category

Deploying Rack Apps With Cherokee and RVM

I recently spent three days trying to get cherokee-1.0.19 (the version in FreeBSD ports as of this writing) to serve up a variety of custom web apps (mostly rails-2.3.8) partitioned and RVM. I had pretty much zero success Googling for help during this exercise so I thought I might compile my experience here for future Googlers.

Before this I had them deployed using apache22 and Phusion Passenger (without RVM). This set up wasn’t working well for me. I’m not saying it’s the wrong way to go (I’m also not going to say a zillion rails users can’t be wrong) but it wasn’t performing well for me and it wasn’t really my style. I’ve been using apache forever, and piling more and more crap on top of it was just getting frustrating. It really just wasn’t built for this.

So I start to hear about all these young httpds like lighttpd, nginx, and cherokee. I poked around a bit, and I decided cherokee’s graphical configuration interface sounded nice for a change. At this point I should inform you that I did not at all download or look much into lighttpd or nginx. They could be just as good for this, but that’s another blog entry on another blog.

Read more

Why Comcast is Lying

There’s currently a big debate going on about Comcast, Netflix, and Level 3. Level 3 is saying that Comcast is forcing them to pay them money that really serves Comcast, and Comcast is saying this is a simple peer dispute and not only is Level 3 lying, but Comcast has “bent over backwards” to accommodate Level 3.

Comcast’s argument goes along the lines of a peering dispute. Level 3 is sending much more traffic to Comcast’s network than Comcast is sending to Level 3′s network, thus Level 3 must pay (let’s ignore that this is Comcast’s very business model: their customers are allotted much more download bandwidth than upload bandwidth, but Comcast feigns surprise). This happened just a few years ago when Cogent was passing much more traffic onto Level 3′s network than Level 3 was onto Cogent’s network. Cogent refused to pay up, and Level 3 terminated their peering arrangement. The case of Level 3 vs Cogent was right, and the case of Comcast vs Level 3 is wrong, and here’s why.

The Simple Explanation

Comcast is wrong, because Level 3 isn’t sending traffic over Comcast’s network in order to reach a destination on the other side of Comcast’s network (called transmit traffic). Level 3 is delivering Netflix content from its network directly to Comcast customers, because Comcast customers have asked for this content (“I want to watch this movie, stream it to my face please”). Comcast’s subscribers are paying Comcast for this. What Comcast is doing is turning around and making Level 3 pay for the same thing again. They are double-dipping, just as they have in the past. When it was Level 3 vs Cogent, Cogent was transmitting traffic over Level 3′s network to another end point. They were using Level 3 to cut their own costs because, at the time, Level 3 was directly peering with Cogent’s end destination, and it was saving Cogent money while costing Level 3 money. This is not the same thing. Level 3 is delivering traffic directly to Comcast, not using Comcast’s network to transmit traffic to a different end point. Comcast’s customers are paying Comcast to deliver this content to them, and the people that are doing the actual delivering shouldn’t only not have to pay Comcast, they should actually be getting paid by Comcast. You pay the UPS driver, not the other way around.

The Not-As-Simple Explanation

It’s a little more complicated than that.

The way the Internet works is that you buy access from your ISP (e.g. Comcast), they in turn (perhaps through some number of additional intermediaries) buy access to the Internet’s core backbone from a Tier 1 provider (like Level 3). If you were a big enough network, you could “peer” without paying anything (since there was no particular reason for one network to pay the other or vice versa so long as they exchange roughly the same amount of traffic in both directions). The way it worked in the Old Days was that content providers (like Netflix) would always be uploading more than they download, so the rule of thumb became that if you upload too much data without downloading, you can’t peer for free. This rule of thumb existed to distinguish content providers from ISPs and make them pay. It is unheard of for a backbone provider to pay an ISP. ISPs like Comcast always used to have approximately symmetrical load to their uplink providers: their customers would download more than they upload, which means they would have “surplus” upload bandwidth “for free” which they could sell to local content providers. Selling to content providers like this is efficient because otherwise the surplus upload bandwidth is lost (“use it or lose it”).

Some of these content providers in modern times turned into content delivery networks (CDNs) like Akamai. The CDNs kind of messed with the model. Instead of buying Comcast’s surplus upload bandwidth to the backbone, they connected to Comcast’s network directly to serve content only to Comcast’s customers. In other words, they evened out Comcast’s uplink to Level 3 not by increasing upload traffic as was the traditional model, but by decreasing download traffic from the backbone. This still works.

So now what’s happening with Level 3? Netflix is moving from Akamai to a data center operated by Level 3 which is only connected to Comcast through Level 3′s backbone. What this does from Comcast’s perspective is both deprive them of the money Akamai was paying them and give them a huge amount of “free” upload bandwidth. What Comcast is then supposed to do is shop this surplus bandwidth around to content providers; or, sell Comcast Business customers higher upload rates (or just about anything else) They’re supposed to sell it, because they can sell it most efficiently; there is a pipe going into Comcast which is only full in one direction and the only way to fill it is to put something which uploads a lot on the Comcast side.

Comcast apparently doesn’t want to do that. I can really only think of two possible reasons for this. First, there is so much upload bandwidth that no one would possibly want to buy it. I find this to be pretty unfathomable. Supply and demand says that if the price is right, someone will pay. Second, Comcast is double-dipping and trying to sell access to their customers as a scarce resource; in other words they’re trying to force Level 3 to pay more for the bandwidth than it would sell for on the open market, because buying it from Comcast is the only way to get access to Comcast’s customers. Comcast has a monopoly on its customers (who already pay them, mind you).

As for Cogent, they were sending traffic over Level 3′s network to third party networks. That is, they weren’t delivering traffic to Level 3 customers, they were delivering content to people Level 3 isn’t getting paid by, while not paying Level 3 themselves. So Cogent was making money, and Level 3 was losing money. In Comcast’s case, Level 3 is delivering content to Comcast customers, who are paying Comcast for that content.

Level 3 has no monopoly on access to third party networks; there are other backbone providers. Comcast has a monopoly over access to Comcast customers. This is the main difference.

Why You Should Care

The people that are calling this a net neutrality issue aren’t far off: Comcast isn’t filtering based on content; it’s actually much worse than that: Comcast is charging content providers in order to allow them to send their content to Comcast’s customers. This is absurd. If this is allowed to continue (which all of the ISPs are hoping for judging by Time Warner’s recent endorsement of the Comcast/Level 3 deal) then you’ll only be able to get Netflix from Sprint, and you’ll only be able to get Hulu on Comcast, and you’ll only be able to get to YouTube from, well, whomever YouTube decides to pay. Because ISPs are so large they have the only pipe leading to tens of millions of people and they’ve decided they should be able to charge the rest of the world to send those people the content those people want to see, which is in fact the very reason those people pay those ISPs to begin with.

So Why Pay Comcast?

Level 3 is alleging they agreed to the deal under duress, and are complaining about it now because now is the only time they reasonably could. If they had said “suck it” and disconnected from Comcast, any traffic that Level 3 needs to send to Comcast customers (namely Netflix, which they have a contracted obligation to do) would have to be transmitted over one of Level 3′s peers, and then routed to Comcast.

Ignore that Comcast would march over to the new peer and demand the same thing they’ve demanded from Level 3. Ignore that Level 3 would undoubtedly have to pay quite a bit of money to this peer to transmit their Netflix traffic to Comcast’s customers. Doing this would seriously degrade the quality of the video Level 3 is transmitting because now it has to take (at minimum) one more step between Level 3 and the customers that are paying both Comcast and Netflix for high quality video. You can bet that Comcast knew they couldn’t say “blow us” without violating their contract with Netflix.

Not-A-Car Analogy

Say you rent a DSL line from the telephone company, but (because you’re currently free to do so) you use an ISP that is not the telephone company. You’re assigned a bandwidth of 1000Kbps downstream and 256Kbps upstream. You download 200GiB per month, and you upload 50GiB per month from the ISP (which doesn’t own your line). In this case, your ISP is sending your line far more traffic than you’re sending back to their lines. Under Comcast’s new logic, the ISP should be paying you in this situation. You own the line, it costs you money for them to send data to you (never mind you’ve requested it, just like Comcast’s customers), and it’s not balanced because you’re not costing them the same amount.

Does this make any sense at all to any sane person? Of course it doesn’t. The ISP would laugh your invoice all the way to the trash.

Return top

To Teach Pain

This is a blog by a guy that lives an ordinary life, except for living every single moment in severe pain. Chronic pain is something most people cannot understand. Pain changes everything.

This is life, in pain.