Before this I had them deployed using apache22 and Phusion Passenger (without RVM). This set up wasn’t working well for me. I’m not saying it’s the wrong way to go (I’m also not going to say a zillion rails users can’t be wrong) but it wasn’t performing well for me and it wasn’t really my style. I’ve been using apache forever, and piling more and more crap on top of it was just getting frustrating. It really just wasn’t built for this.

So I start to hear about all these young httpds like lighttpd, nginx, and cherokee. I poked around a bit, and I decided cherokee’s graphical configuration interface sounded nice for a change. At this point I should inform you that I did not at all download or look much into lighttpd or nginx. They could be just as good for this, but that’s another blog entry on another blog.

Mongrel or Unicorn or Thin or WEBrick or…

One of the first things you need to decide is which back end you want to run. Cherokee needs to know about these processes as this is how it’ll speak to your app (see the cherokee section below). By default, rails (and almost any rack app) uses WEBrick. WEBrick is great for development, but it’s pure ruby and pretty slow. I don’t think you want it serving up your production sites. Historically, a mongrel cluster has been the method of choice for deploying rails. However, mongrel doesn’t run on ruby-1.9 which is what I use to deploy. Thin is a great little server that combines mongrel’s parser with EventMachine (a fast event-based code system) with rack. Thin can run just about any kind of rack app (rack, rails2, rails3, ramaze, probably more) and is really easy to use. However, cherokee isn’t great at maintaining a thin cluster for you. If one of your instances dies, you’re going to end up handing someone a 502 error at least once. I personally decided to go with unicorn. Unicorn can’t really handle being a front-facing server because it has no KeepAlive support. It acknowledges this, and expects you to use nginx (or in our case, cherokee) in front of it. The great thing about unicorn is that it spawns a management process and then a cluster of workers. The management process does the listening, and decides which worker is best suited to handle your request. If any of the workers die, or need to be killed, the management process will restart them. This means you only need one information source in cherokee (since it manages its own workers it only needs one port, whereas something like thin will need a separate port and thus a separate information source per server in a cluster). I ended up going with unicorn for my big production rails apps with 2-4 workers, and a single thin instance for small sinatra apps that don’t get a lot of load.

When it comes down to it, this decision is up to you. If you like thin be prepared to juggle as many information sources in cherokee as you have thin servers per cluster. Also be prepared for 502 errors while cherokee spawns thin if it’s not running. Once spawned, thin will stay running even if cherokee exits (not the case with unicorn, which will exit if cherokee exits or restarts). I don’t have personal experience with mongrel, but it probably goes something along the same path as thin. WEBrick… well… I guess if that’s how you roll.

Cherokee and RVM

Cherokee was the relatively easy part. Aside from the fact that the neat wizards it uses to add virtual servers don’t usually work right (it seems to assume you’re using rails-2.2 from the ancient times), the process is pretty simple. You can use the rails wizard to get started. After that, go to the information sources it creates. Here is where you need to tweak. It creates three by default. Depending on which back end you decided on earlier, you’ll have to tweak these to your pleasure. Most important is the spawning command. This is the command cherokee executes if it can’t find the interpreter running where it’s supposed to be. First, however, each information source has to have an environment. You can inherit the environment from cherokee (probably not what you want) or you can set up a fresh one. Since we’re using RVM we’re going to need bash, because RVM is written in bash shell script. One of the maddening things that happened to me was that I was executing the right commands without realizing there was no PATH and so RVM couldn’t work because bash couldn’t be found. So if nothing else, create an environment variable named PATH with some value along the lines of:

/bin:/usr/bin:/usr/local/bin:/usr/local/www/.rvm/bin

You should change that last bit there to the location of the RVM installation you’re using. Other pertinent environment variables might be things such as RAILS_ENV and RACK_ENV (for “production”, “development”, etc).

The starting point and probably most intensive part of this is RVM. RVM is a set of shell scripts and environment trickery that allows you to install and manage multiple versions of ruby. In addition to this it allows you to separate gems into “gemsets” which are unique to each version of ruby you have managed. The gemsets allow you to install a fresh set of gems for each gemset. This was why I was determined to use RVM with my deployment. I don’t really need multiple versions of ruby itself, but doing that would work just fine with how I did things as well. I installed a limited set of gems that appear in all gemsets (the “global” gemset) such as rake and bundler and thin and unicorn. After that, each app has its own gemset with its own local version of rails, rack, and whatever else it needs. This keeps things nicely partitioned and even acts as a sandbox to some degree (to really get sandboxing, bundler offers just superb possibilities).

However, RVM also proved to be the most difficult part of this setup to get working properly. When you’re working with RVM in your shell you have environment variables and your shell loads RVM’s magic into itself when you first start the shell. Of course, cherokee isn’t going to execute your ~/.profile when it runs the interpreter for requests. So how do you get RVM’s magic in there? RVM provides a set of “wrappers” that set up all of its environment and magic. These are in ~/.rvm/wrappers/ by default. So in cherokee’s information source you need to set up PATH as described above. After that, it’s really just a matter of perfecting your spawn command. I use unicorn, which only requires a single information source, and my RVM is installed in /usr/local/www/.rvm/, so my spawn command looks something like:

/usr/local/www/.rvm/wrappers/ruby-1.9.2-p180@NameOfApp/ruby /usr/local/www/.rvm/gems/ruby-1.9.2-p180@global/bin/unicorn_rails -c /usr/local/www/NameOfApp/unicorn.conf -E production

Wow, that’s a mess. Unfortunately they’re probably all going to be pretty messy like this. So let’s break that down:

/path/to/rvm/wrapper/ruby-version@gemset/ruby /path/to/backend --args

The first command is your ruby executable. Since cherokee doesn’t know about RVM, in order to allow RVM to set up its magic you have to execute a bash script in the wrappers/ directory. This script sets up the necessary magic for the gemset you selected (in this case, NameOfApp) and executes the ruby binary with those gems. The second part is the back end you’re using. In my case, I execute unicorn_rails which I’ve installed as a gem in the global gemset (which appears in all gemsets for that ruby version) and pass it a configuration file that tells it what app it’s serving and how to serve it. Let’s say you’re using the default rails server:

/path/to/rvm/wrapper/ruby-version@gemset/ruby /path/to/rails_app/script/server -p port

This is a little easier, right? In rails-2.3.x (use rails server or script/rails server for rails-3) this would probably launch WEBrick listening on the port that your cherokee information source is expecting (cherokee asks us not to pass any flags to make the server daemonize as cherokee takes care to route the input/output/error streams to useful places and manages other aspects that forking would break). The bit that makes others installed as a gem ugly looking is that you have to find the binary they installed and you have to figure out how to tell them “go in /path/to/rails_app and run that app in there.” With unicorn, you can tell it that (and much more) with a unicorn.conf (that doesn’t need to be in the same directory as the app). With thin, you can tell it that with a configuration file that thin can output using thin config or you can simply run thin start -c /path/to/rails_app. While mine in particular is kind of ugly, it isn’t as complex as it appears.

Once you figure out the right spawning command, and the RVM scripts can figure out everything they need to run from PATH, you should be good to go. Depending on what back end you’re using, the first time you try to visit the app’s address you might get a 502 error from cherokee while it launches the spawning process (typically happens with thin but not with unicorn, and sometimes with script/server) so if you get a 502 try refreshing the page immediately.

Troubleshooting

A lot can go wrong. I think probably just about all of it went wrong for me at some point.

502 Bad Gateway

This is the error cherokee tosses out if it can’t get one of your information sources on the phone. This can be because they’re all busy, but more likely if you’re reading this it’s because your spawn command is broken. The best way to investigate this is to set up an error log in the virtual server using the information source that isn’t working. Cherokee will log the output from the spawn command here. If you see something like ruby not found it’s because you’ve gotten the first part of your spawn command wrong. First off, use full paths. Don’t assume ruby-1.9.2-p180@NameOfApp is in the PATH you set. Sometimes even if it is, it still won’t run right without a full path. If you see something like env: bash not found it’s because something or another wasn’t found in your PATH. Make sure the PATH you set includes anything ruby, RVM, or your back end needs to run.

Another chance is that your spawn command is right but the app you’re trying to run is having an error. Maybe you’re missing a gem, or there’s a syntax error, or any other ruby-level error that’d cause a backtrace in ruby. You can examine your back end or app logs to find this. Another good way to make sure that your app runs fine is running an equivalent of the spawn command in your shell. Try thin start in an app you’re running with thin. If it’s a ruby backtrace you’ll see it here. Same goes for unicorn_rails or script/server. Generally, if your app runs in your development environment and it doesn’t run here, it’s because you forgot something like a gem or other dependency. Also make sure you’re using the right ruby/gemset in RVM. Running the back end by hand before allowing cherokee to spawn it automatically is always a good idea.

504 Gateway Timeout

This usually happens because, while cherokee can get one of your information sources on the phone, it doesn’t answer in a timely fashion. This can be because you’re getting slammed and they’re just running slowly, or more likely because one of the back end processes is attempting to process a request that’s taking a while. This can be anything from something CPU-intensive to something database-intensive. Examine your SQL queries. check your back end or app logs. For me, this is most often a network or database issue.

The End

I hope this helps. If not, and you’re using cherokee, RVM, and rack apps, my contact information is all over this blog. Feel free to get in touch.

Comcast’s argument goes along the lines of a peering dispute. Level 3 is sending much more traffic to Comcast’s network than Comcast is sending to Level 3′s network, thus Level 3 must pay (let’s ignore that this is Comcast’s very business model: their customers are allotted much more download bandwidth than upload bandwidth, but Comcast feigns surprise). This happened just a few years ago when Cogent was passing much more traffic onto Level 3′s network than Level 3 was onto Cogent’s network. Cogent refused to pay up, and Level 3 terminated their peering arrangement. The case of Level 3 vs Cogent was right, and the case of Comcast vs Level 3 is wrong, and here’s why.

The Simple Explanation

Comcast is wrong, because Level 3 isn’t sending traffic over Comcast’s network in order to reach a destination on the other side of Comcast’s network (called transmit traffic). Level 3 is delivering Netflix content from its network directly to Comcast customers, because Comcast customers have asked for this content (“I want to watch this movie, stream it to my face please”). Comcast’s subscribers are paying Comcast for this. What Comcast is doing is turning around and making Level 3 pay for the same thing again. They are double-dipping, just as they have in the past. When it was Level 3 vs Cogent, Cogent was transmitting traffic over Level 3′s network to another end point. They were using Level 3 to cut their own costs because, at the time, Level 3 was directly peering with Cogent’s end destination, and it was saving Cogent money while costing Level 3 money. This is not the same thing. Level 3 is delivering traffic directly to Comcast, not using Comcast’s network to transmit traffic to a different end point. Comcast’s customers are paying Comcast to deliver this content to them, and the people that are doing the actual delivering shouldn’t only not have to pay Comcast, they should actually be getting paid by Comcast. You pay the UPS driver, not the other way around.

The Not-As-Simple Explanation

It’s a little more complicated than that.

The way the Internet works is that you buy access from your ISP (e.g. Comcast), they in turn (perhaps through some number of additional intermediaries) buy access to the Internet’s core backbone from a Tier 1 provider (like Level 3). If you were a big enough network, you could “peer” without paying anything (since there was no particular reason for one network to pay the other or vice versa so long as they exchange roughly the same amount of traffic in both directions). The way it worked in the Old Days was that content providers (like Netflix) would always be uploading more than they download, so the rule of thumb became that if you upload too much data without downloading, you can’t peer for free. This rule of thumb existed to distinguish content providers from ISPs and make them pay. It is unheard of for a backbone provider to pay an ISP. ISPs like Comcast always used to have approximately symmetrical load to their uplink providers: their customers would download more than they upload, which means they would have “surplus” upload bandwidth “for free” which they could sell to local content providers. Selling to content providers like this is efficient because otherwise the surplus upload bandwidth is lost (“use it or lose it”).

Some of these content providers in modern times turned into content delivery networks (CDNs) like Akamai. The CDNs kind of messed with the model. Instead of buying Comcast’s surplus upload bandwidth to the backbone, they connected to Comcast’s network directly to serve content only to Comcast’s customers. In other words, they evened out Comcast’s uplink to Level 3 not by increasing upload traffic as was the traditional model, but by decreasing download traffic from the backbone. This still works.

So now what’s happening with Level 3? Netflix is moving from Akamai to a data center operated by Level 3 which is only connected to Comcast through Level 3′s backbone. What this does from Comcast’s perspective is both deprive them of the money Akamai was paying them and give them a huge amount of “free” upload bandwidth. What Comcast is then supposed to do is shop this surplus bandwidth around to content providers; or, sell Comcast Business customers higher upload rates (or just about anything else) They’re supposed to sell it, because they can sell it most efficiently; there is a pipe going into Comcast which is only full in one direction and the only way to fill it is to put something which uploads a lot on the Comcast side.

Comcast apparently doesn’t want to do that. I can really only think of two possible reasons for this. First, there is so much upload bandwidth that no one would possibly want to buy it. I find this to be pretty unfathomable. Supply and demand says that if the price is right, someone will pay. Second, Comcast is double-dipping and trying to sell access to their customers as a scarce resource; in other words they’re trying to force Level 3 to pay more for the bandwidth than it would sell for on the open market, because buying it from Comcast is the only way to get access to Comcast’s customers. Comcast has a monopoly on its customers (who already pay them, mind you).

As for Cogent, they were sending traffic over Level 3′s network to third party networks. That is, they weren’t delivering traffic to Level 3 customers, they were delivering content to people Level 3 isn’t getting paid by, while not paying Level 3 themselves. So Cogent was making money, and Level 3 was losing money. In Comcast’s case, Level 3 is delivering content to Comcast customers, who are paying Comcast for that content.

Level 3 has no monopoly on access to third party networks; there are other backbone providers. Comcast has a monopoly over access to Comcast customers. This is the main difference.

Why You Should Care

The people that are calling this a net neutrality issue aren’t far off: Comcast isn’t filtering based on content; it’s actually much worse than that: Comcast is charging content providers in order to allow them to send their content to Comcast’s customers. This is absurd. If this is allowed to continue (which all of the ISPs are hoping for judging by Time Warner’s recent endorsement of the Comcast/Level 3 deal) then you’ll only be able to get Netflix from Sprint, and you’ll only be able to get Hulu on Comcast, and you’ll only be able to get to YouTube from, well, whomever YouTube decides to pay. Because ISPs are so large they have the only pipe leading to tens of millions of people and they’ve decided they should be able to charge the rest of the world to send those people the content those people want to see, which is in fact the very reason those people pay those ISPs to begin with.

So Why Pay Comcast?

Level 3 is alleging they agreed to the deal under duress, and are complaining about it now because now is the only time they reasonably could. If they had said “suck it” and disconnected from Comcast, any traffic that Level 3 needs to send to Comcast customers (namely Netflix, which they have a contracted obligation to do) would have to be transmitted over one of Level 3′s peers, and then routed to Comcast.

Ignore that Comcast would march over to the new peer and demand the same thing they’ve demanded from Level 3. Ignore that Level 3 would undoubtedly have to pay quite a bit of money to this peer to transmit their Netflix traffic to Comcast’s customers. Doing this would seriously degrade the quality of the video Level 3 is transmitting because now it has to take (at minimum) one more step between Level 3 and the customers that are paying both Comcast and Netflix for high quality video. You can bet that Comcast knew they couldn’t say “blow us” without violating their contract with Netflix.

Not-A-Car Analogy

Say you rent a DSL line from the telephone company, but (because you’re currently free to do so) you use an ISP that is not the telephone company. You’re assigned a bandwidth of 1000Kbps downstream and 256Kbps upstream. You download 200GiB per month, and you upload 50GiB per month from the ISP (which doesn’t own your line). In this case, your ISP is sending your line far more traffic than you’re sending back to their lines. Under Comcast’s new logic, the ISP should be paying you in this situation. You own the line, it costs you money for them to send data to you (never mind you’ve requested it, just like Comcast’s customers), and it’s not balanced because you’re not costing them the same amount.

Does this make any sense at all to any sane person? Of course it doesn’t. The ISP would laugh your invoice all the way to the trash.

The spinal action of opioids is an excellent example of how basic research in animals can lead to improvements in the clinical relief of pain. The knowledge gained from basic animal studies showing an opioid inhibition of nociceptive spinal neurons and the direct analgesia following epidural and intrathecal opioids was soon applied to humans. Importantly, the use of various different models of clinical pain states has led to animal studies addressing the extent and mechanisms of plasticity in opioid spinal function, since pathological and physiological and pharmacological events can alter the degree of opioid antinociception. It is noteworthy that opioid receptors originally cloned from rats and mice allowed much in vivo research, and ultimately it was discovered that the animal opioid receptors are identical biochemically, and pharmacologically, to human opioid receptors.

These animals were used to develop pain and suffering relief in humans. If a couple of massively overpopulated rodent had to die for me to be pain free, I have to tell you that I’m not that upset by it.

At first, I thought it was just a bout of homesickness. I don’t particularly care for my life at the moment. So far I’ve managed to convince myself it’s merely a transition–I must simply “hang in there” as it were, while my beloved girlfriend completes nursing school, which will allow us to move anywhere we please–or so the theory goes. The problem with this is I can only convince myself for so long. The cold hard truth is that there is a big difference between plans and reality, and reality has never seemed to have much of a predisposition for conforming to my plans–in fact, it seems to almost have its own plan for my life. I would prefer it left it to me. I find that reality is almost always wrong.

My friends from back home ask me what I’m doing in Baltimore, and when I tell them I work for an engineering company that has me write computer software for the government they’re pretty impressed–if only I were also. I’ve been doing this for a long time, and I’m no longer particularly interested in it. In fact, when I coded as a hobby I vowed I would never rely on it for a paycheck, simply because I cannot maintain an interest in some arbitrary project. I have the “open source itch,” as it’s come to be known. That is, I have an urge to code particular things I’m interested in, and not something that a project manager tells me to code. Fortunately I’m not in that situation–I get to code however I want, with whatever method, style, or language I prefer–but I still don’t get to pick what I code. When I first started this job I ran at full speed; I completed two major projects in the first three months, one of which had been in limbo for nearly three years. I am good at what I do, I just don’t like what I do.

So what do I want to do? Due to various encounters–positive and negative–with the healthcare profession, I would very much like to attend medical school and work as a pain management physician. I don’t know if I’m smart enough, and I don’t think I’m young enough. If I wait until my girlfriend has finished nursing school I will be–at the minimum–three years shy of thirty before I can even start pre-med. I just can’t convince myself this is a realistic thing to do, especially seeing as how we have plans to start a family around that time.

I dreadfully find that I have fallen into the exact same predicament most members of our society do–one that I promised myself I would never be trapped into: not achieving my dreams. Most of us have something happen in their early twenties, and you think “oh, I’ll just wait a year until I can figure this out,” or “I’ll just take a few years to get this right,” and then you turn around and it’s been five, seven, ten years. I’ve spent a mere twenty-three years on this planet–four of them taken away by intractable pain that isn’t stopping anytime soon–and already I feel the pressure of time slowly pushing me farther and farther from my dreams, and closer and closer to mediocrity.

I see my friends saving money, buying houses, having families; I see me stuck, as if running in a dream: never quite able to catch what you’re chasing (or running from). No matter how hard and fast you run, it only gets farther anyway. I’m very afraid that I’ll become like so many blue-collar families–like my own parents: spending their entire lives attempting to ensure a better one for their children. That’s simply not what my life was made for. It was made for greater things than these.