<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Life in Pain &#187; Eric Will</title>
	<atom:link href="http://blog.ericw.org/author/admin/feed/" rel="self" type="application/rss+xml" />
	<link>http://blog.ericw.org</link>
	<description>live, love, hurt</description>
	<lastBuildDate>Wed, 18 May 2011 02:37:26 +0000</lastBuildDate>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
	<generator>http://wordpress.org/?v=3.2.1</generator>
		<item>
		<title>Deploying Rack Apps With Cherokee and RVM</title>
		<link>http://blog.ericw.org/2011/02/rack-cherokee-rvm/</link>
		<comments>http://blog.ericw.org/2011/02/rack-cherokee-rvm/#comments</comments>
		<pubDate>Wed, 23 Feb 2011 15:22:06 +0000</pubDate>
		<dc:creator>Eric Will</dc:creator>
				<category><![CDATA[technology]]></category>
		<category><![CDATA[cherokee]]></category>
		<category><![CDATA[rack]]></category>
		<category><![CDATA[rails]]></category>
		<category><![CDATA[ruby]]></category>
		<category><![CDATA[rvm]]></category>
		<category><![CDATA[technical]]></category>
		<category><![CDATA[thin]]></category>
		<category><![CDATA[unicorn]]></category>

		<guid isPermaLink="false">http://blog.ericw.org/?p=140</guid>
		<description><![CDATA[A very long effort to document my rails/rack deployment using cherokee and RVM.</p]]></description>
			<content:encoded><![CDATA[<p>I recently spent three days trying to get <a href="http://www.cherokee-project.com/">cherokee</a>-1.0.19 (the version in <a href="http://www.freebsd.org/">FreeBSD</a> ports as of this writing) to serve up a variety of custom web apps (mostly <a href="http://www.rubyonrails.org/">rails</a>-2.3.8) partitioned and <a href="http://rvm.beginrescueend.com/">RVM</a>. I had pretty much zero success Googling for help during this exercise so I thought I might compile my experience here for future Googlers.</p>

<p>Before this I had them deployed using <a href="http://httpd.apache.org/">apache</a>22 and <a href="http://www.modrails.com/">Phusion Passenger</a> (without <a href="http://rvm.beginrescueend.com/">RVM</a>). This set up wasn&#8217;t working well for me. I&#8217;m not saying it&#8217;s the wrong way to go (I&#8217;m also not going to say a zillion rails users can&#8217;t be wrong) but it wasn&#8217;t performing well for me and it wasn&#8217;t really my style. I&#8217;ve been using apache forever, and piling more and more crap on top of it was just getting frustrating. It really just wasn&#8217;t built for this.</p>

<p>So I start to hear about all these young httpds like <a href="http://www.lighttpd.net/">lighttpd</a>, <a href="http://nginx.org/en/">nginx</a>, and <a href="http://www.cherokee-project.com/">cherokee</a>. I poked around a bit, and I decided cherokee&#8217;s graphical configuration interface sounded nice for a change. At this point I should inform you that I did not at all download or look much into <a href="http://www.lighttpd.net/">lighttpd</a> or <a href="http://nginx.org/en/">nginx</a>. They could be just as good for this, but that&#8217;s another blog entry on another blog.</p>

<p><span id="more-140"></span></p>

<h3>Mongrel or Unicorn or Thin or WEBrick or&#8230;</h3>

<p>One of the first things you need to decide is which back end you want to run. Cherokee needs to know about these processes as this is how it&#8217;ll speak to your app (see the cherokee section below). By default, rails (and almost any rack app) uses WEBrick. WEBrick is great for development, but it&#8217;s pure ruby and pretty slow. I don&#8217;t think you want it serving up your production sites. Historically, a mongrel cluster has been the method of choice for deploying rails. However, mongrel doesn&#8217;t run on ruby-1.9 which is what I use to deploy. <a href="http://code.macournoyer.com/thin/">Thin</a> is a great little server that combines mongrel&#8217;s parser with EventMachine (a fast event-based code system) with rack. Thin can run just about any kind of rack app (rack, rails2, rails3, <a href="http://ramaze.net/">ramaze</a>, probably more) and is really easy to use. However, cherokee isn&#8217;t great at maintaining a thin cluster for you. If one of your instances dies, you&#8217;re going to end up handing someone a 502 error at least once. I personally decided to go with unicorn. <a href="http://unicorn.bogomips.org/">Unicorn</a> can&#8217;t really handle being a front-facing server because it has no KeepAlive support. It acknowledges this, and expects you to use nginx (or in our case, cherokee) in front of it. The great thing about unicorn is that it spawns a management process and then a cluster of workers. The management process does the listening, and decides which worker is best suited to handle your request. If any of the workers die, or need to be killed, the management process will restart them. This means you only need one information source in cherokee (since it manages its own workers it only needs one port, whereas something like thin will need a separate port and thus a separate information source per server in a cluster). I ended up going with unicorn for my big production rails apps with 2-4 workers, and a single thin instance for small <a href="http://www.sinatrarb.com/">sinatra</a> apps that don&#8217;t get a lot of load.</p>

<p>When it comes down to it, this decision is up to you. If you like thin be prepared to juggle as many information sources in cherokee as you have thin servers per cluster. Also be prepared for 502 errors while cherokee spawns thin if it&#8217;s not running. Once spawned, thin will stay running even if cherokee exits (not the case with unicorn, which will exit if cherokee exits or restarts). I don&#8217;t have personal experience with mongrel, but it probably goes something along the same path as thin. WEBrick&#8230; well&#8230; I guess if that&#8217;s how you roll.</p>

<h3>Cherokee and RVM</h3>

<p>Cherokee was the relatively easy part. Aside from the fact that the neat wizards it uses to add virtual servers don&#8217;t usually work right (it seems to assume you&#8217;re using rails-2.2 from the ancient times), the process is pretty simple. You can use the rails wizard to get started. After that, go to the information sources it creates. Here is where you need to tweak. It creates three by default. Depending on which back end you decided on earlier, you&#8217;ll have to tweak these to your pleasure. Most important is the spawning command. This is the command cherokee executes if it can&#8217;t find the interpreter running where it&#8217;s supposed to be. First, however, each information source has to have an environment. You can inherit the environment from cherokee (probably not what you want) or you can set up a fresh one. Since we&#8217;re using RVM we&#8217;re going to need bash, because RVM is written in bash shell script. One of the maddening things that happened to me was that I was executing the right commands without realizing there was no <code>PATH</code> and so RVM couldn&#8217;t work because bash couldn&#8217;t be found. So if nothing else, create an environment variable named <code>PATH</code> with some value along the lines of:</p>

<pre><code>/bin:/usr/bin:/usr/local/bin:/usr/local/www/.rvm/bin
</code></pre>

<p>You should change that last bit there to the location of the RVM installation you&#8217;re using. Other pertinent environment variables might be things such as <code>RAILS_ENV</code> and <code>RACK_ENV</code> (for &#8220;production&#8221;, &#8220;development&#8221;, etc).</p>

<p>The starting point and probably most intensive part of this is <a href="http://rvm.beginrescueend.com/">RVM</a>. RVM is a set of shell scripts and environment trickery that allows you to install and manage multiple versions of ruby. In addition to this it allows you to separate gems into &#8220;gemsets&#8221; which are unique to each version of ruby you have managed. The gemsets allow you to install a fresh set of gems for each gemset. This was why I was determined to use RVM with my deployment. I don&#8217;t really need multiple versions of ruby itself, but doing that would work just fine with how I did things as well. I installed a limited set of gems that appear in all gemsets (the &#8220;global&#8221; gemset) such as <code>rake</code> and <code>bundler</code> and <code>thin</code> and <code>unicorn</code>. After that, each app has its own gemset with its own local version of rails, rack, and whatever else it needs. This keeps things nicely partitioned and even acts as a sandbox to some degree (to really get sandboxing, <code>bundler</code> offers just superb possibilities).</p>

<p>However, RVM also proved to be the most difficult part of this setup to get working properly. When you&#8217;re working with RVM in your shell you have environment variables and your shell loads RVM&#8217;s magic into itself when you first start the shell. Of course, cherokee isn&#8217;t going to execute your <code>~/.profile</code> when it runs the interpreter for requests. So how do you get RVM&#8217;s magic in there? RVM provides a set of &#8220;wrappers&#8221; that set up all of its environment and magic. These are in <code>~/.rvm/wrappers/</code> by default. So in cherokee&#8217;s information source you need to set up <code>PATH</code> as described above. After that, it&#8217;s really just a matter of perfecting your spawn command. I use unicorn, which only requires a single information source, and my RVM is installed in <code>/usr/local/www/.rvm/</code>, so my spawn command looks something like:</p>

<pre><code>/usr/local/www/.rvm/wrappers/ruby-1.9.2-p180@NameOfApp/ruby /usr/local/www/.rvm/gems/ruby-1.9.2-p180@global/bin/unicorn_rails -c /usr/local/www/NameOfApp/unicorn.conf -E production
</code></pre>

<p>Wow, that&#8217;s a mess. Unfortunately they&#8217;re probably all going to be pretty messy like this. So let&#8217;s break that down:</p>

<pre><code>/path/to/rvm/wrapper/ruby-version@gemset/ruby /path/to/backend --args
</code></pre>

<p>The first command is your ruby executable. Since cherokee doesn&#8217;t know about RVM, in order to allow RVM to set up its magic you have to execute a bash script in the <code>wrappers/</code> directory. This script sets up the necessary magic for the gemset you selected (in this case, <code>NameOfApp</code>) and executes the ruby binary with those gems. The second part is the back end you&#8217;re using. In my case, I execute <code>unicorn_rails</code> which I&#8217;ve installed as a gem in the <code>global</code> gemset (which appears in all gemsets for that ruby version) and pass it a configuration file that tells it what app it&#8217;s serving and how to serve it. Let&#8217;s say you&#8217;re using the default rails server:</p>

<pre><code>/path/to/rvm/wrapper/ruby-version@gemset/ruby /path/to/rails_app/script/server -p port
</code></pre>

<p>This is a little easier, right? In rails-2.3.x (use <code>rails server</code> or <code>script/rails server</code> for rails-3) this would probably launch WEBrick listening on the port that your cherokee information source is expecting (cherokee asks us not to pass any flags to make the server daemonize as cherokee takes care to route the input/output/error streams to useful places and manages other aspects that forking would break). The bit that makes others installed as a gem ugly looking is that you have to find the binary they installed and you have to figure out how to tell them &#8220;go in /path/to/rails_app and run that app in there.&#8221; With unicorn, you can tell it that (and much more) with a unicorn.conf (that doesn&#8217;t need to be in the same directory as the app). With thin, you can tell it that with a configuration file that thin can output using <code>thin config</code> or you can simply run <code>thin start -c /path/to/rails_app</code>. While mine in particular is kind of ugly, it isn&#8217;t as complex as it appears.</p>

<p>Once you figure out the right spawning command, and the RVM scripts can figure out everything they need to run from <code>PATH</code>, you should be good to go. Depending on what back end you&#8217;re using, the first time you try to visit the app&#8217;s address you might get a 502 error from cherokee while it launches the spawning process (typically happens with thin but not with unicorn, and sometimes with <code>script/server</code>) so if you get a 502 try refreshing the page immediately.</p>

<h3>Troubleshooting</h3>

<p>A lot can go wrong. I think probably just about all of it went wrong for me at some point.</p>

<h4>502 Bad Gateway</h4>

<p>This is the error cherokee tosses out if it can&#8217;t get one of your information sources on the phone. This can be because they&#8217;re all busy, but more likely if you&#8217;re reading this it&#8217;s because your spawn command is broken. The best way to investigate this is to set up an error log in the virtual server using the information source that isn&#8217;t working. Cherokee will log the output from the spawn command here. If you see something like <code>ruby not found</code> it&#8217;s because you&#8217;ve gotten the first part of your spawn command wrong. First off, use full paths. Don&#8217;t assume <code>ruby-1.9.2-p180@NameOfApp</code> is in the PATH you set. Sometimes even if it is, it still won&#8217;t run right without a full path. If you see something like <code>env: bash not found</code> it&#8217;s because something or another wasn&#8217;t found in your <code>PATH</code>. Make sure the <code>PATH</code> you set includes anything ruby, RVM, or your back end needs to run.</p>

<p>Another chance is that your spawn command is right but the app you&#8217;re trying to run is having an error. Maybe you&#8217;re missing a gem, or there&#8217;s a syntax error, or any other ruby-level error that&#8217;d cause a backtrace in ruby. You can examine your back end or app logs to find this. Another good way to make sure that your app runs fine is running an equivalent of the spawn command in your shell. Try <code>thin start</code> in an app you&#8217;re running with thin. If it&#8217;s a ruby backtrace you&#8217;ll see it here. Same goes for <code>unicorn_rails</code> or <code>script/server</code>. Generally, if your app runs in your development environment and it doesn&#8217;t run here, it&#8217;s because you forgot something like a gem or other dependency. Also make sure you&#8217;re using the right ruby/gemset in RVM. Running the back end by hand before allowing cherokee to spawn it automatically is always a good idea.</p>

<h4>504 Gateway Timeout</h4>

<p>This usually happens because, while cherokee can get one of your information sources on the phone, it doesn&#8217;t answer in a timely fashion. This can be because you&#8217;re getting slammed and they&#8217;re just running slowly, or more likely because one of the back end processes is attempting to process a request that&#8217;s taking a while. This can be anything from something CPU-intensive to something database-intensive. Examine your SQL queries. check your back end or app logs. For me, this is most often a network or database issue.</p>

<h3>The End</h3>

<p>I hope this helps. If not, and you&#8217;re using cherokee, RVM, and rack apps, my contact information is all over this blog. Feel free to get in touch.</p>
]]></content:encoded>
			<wfw:commentRss>http://blog.ericw.org/2011/02/rack-cherokee-rvm/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Why Comcast is Lying</title>
		<link>http://blog.ericw.org/2010/12/why-comcast-is-lying/</link>
		<comments>http://blog.ericw.org/2010/12/why-comcast-is-lying/#comments</comments>
		<pubDate>Thu, 09 Dec 2010 02:27:44 +0000</pubDate>
		<dc:creator>Eric Will</dc:creator>
				<category><![CDATA[technology]]></category>
		<category><![CDATA[comcast]]></category>
		<category><![CDATA[internet]]></category>
		<category><![CDATA[net neutrality]]></category>
		<category><![CDATA[technical]]></category>

		<guid isPermaLink="false">http://pain.ericw.org/?p=122</guid>
		<description><![CDATA[A brief essay on how Comcast is breaking all the rules in order get paid twice for the same thing.</p]]></description>
			<content:encoded><![CDATA[<p>There&#8217;s currently a big debate going on about Comcast, Netflix, and Level 3. Level 3 is saying that Comcast is forcing them to pay them money that really serves Comcast, and Comcast is saying this is a simple peer dispute and not only is Level 3 lying, but Comcast has &#8220;bent over backwards&#8221; to accommodate Level 3.</p>

<p>Comcast&#8217;s argument goes along the lines of a peering dispute. Level 3 is sending much more traffic to Comcast&#8217;s network than Comcast is sending to Level 3&#8242;s network, thus Level 3 must pay (let&#8217;s ignore that this is Comcast&#8217;s very business model: their customers are allotted much more download bandwidth than upload bandwidth, but Comcast feigns surprise). This happened just a few years ago when Cogent was passing much more traffic onto Level 3&#8242;s network than Level 3 was onto Cogent&#8217;s network. Cogent refused to pay up, and Level 3 terminated their peering arrangement. The case of Level 3 vs Cogent was right, and the case of Comcast vs Level 3 is wrong, and here&#8217;s why.</p>

<h3>The Simple Explanation</h3>

<p>Comcast is wrong, because Level 3 isn&#8217;t sending traffic over Comcast&#8217;s network in order to reach a destination on the other side of Comcast&#8217;s network (called <strong>transmit</strong> traffic). Level 3 is delivering Netflix content from its network directly to Comcast customers, because Comcast customers have asked for this content (&#8220;I want to watch this movie, stream it to my face please&#8221;). Comcast&#8217;s subscribers are paying Comcast for this. What Comcast is doing is turning around and making Level 3 pay for the same thing again. They are double-dipping, just as they have in the past. When it was Level 3 vs Cogent, Cogent was <strong>transmitting</strong> traffic <em>over</em> Level 3&#8242;s network to another end point. They were using Level 3 to cut their own costs because, at the time, Level 3 was directly peering with Cogent&#8217;s end destination, and it was saving Cogent money while costing Level 3 money. This is not the same thing. Level 3 is <strong>delivering</strong> traffic directly to Comcast, not using Comcast&#8217;s network to <strong>transmit</strong> traffic to a different end point. Comcast&#8217;s customers are paying Comcast to deliver this content to them, and the people that are doing the actual delivering shouldn&#8217;t only <em>not</em> have to pay Comcast, they should actually be <em>getting paid by</em> Comcast. You pay the UPS driver, not the other way around.</p>

<h3>The Not-As-Simple Explanation</h3>

<p>It&#8217;s a little more complicated than that.</p>

<p>The way the Internet works is that you buy access from your ISP (e.g. Comcast), they in turn (perhaps through some number of additional intermediaries) <strong>buy</strong> access to the Internet&#8217;s core backbone from a Tier 1 provider (like Level 3). If you were a big enough network, you could &#8220;peer&#8221; without paying anything (since there was no particular reason for one network to pay the other or vice versa so long as they exchange roughly the same amount of traffic in both directions). The way it worked in the Old Days was that content providers (like Netflix) would always be uploading more than they download, so the rule of thumb became that if you upload too much data without downloading, you can&#8217;t peer for free. This rule of thumb existed to distinguish content providers from ISPs and make them pay. It is <strong>unheard of</strong> for a backbone provider to pay an ISP. ISPs like Comcast always used to have approximately symmetrical load to their uplink providers: their customers would download more than they upload, which means they would have &#8220;surplus&#8221; upload bandwidth &#8220;for free&#8221; which they could sell to local content providers. Selling to content providers like this is efficient because otherwise the surplus upload bandwidth is lost (&#8220;use it or lose it&#8221;).</p>

<p>Some of these content providers in modern times turned into content delivery networks (CDNs) like Akamai. The CDNs kind of messed with the model. Instead of buying Comcast&#8217;s surplus upload bandwidth to the backbone, they connected to Comcast&#8217;s network directly to serve content only to Comcast&#8217;s customers. In other words, they evened out Comcast&#8217;s uplink to Level 3 not by increasing upload traffic as was the traditional model, but by decreasing download traffic from the backbone. This still works.</p>

<p>So now what&#8217;s happening with Level 3? Netflix is moving from Akamai to a data center operated by Level 3 which is only connected to Comcast through Level 3&#8242;s backbone. What this does from Comcast&#8217;s perspective is both deprive them of the money Akamai was paying them and give them a huge amount of &#8220;free&#8221; upload bandwidth. What Comcast is then supposed to do is shop this surplus bandwidth around to content providers; or, sell Comcast Business customers higher upload rates (or just about anything else) They&#8217;re supposed to sell it, because they can sell it most efficiently; there is a pipe going into Comcast which is only full in one direction and the only way to fill it is to put something which uploads a lot on the Comcast side.</p>

<p>Comcast apparently doesn&#8217;t want to do that. I can really only think of two possible reasons for this. First, there is so much upload bandwidth that no one would possibly want to buy it. I find this to be pretty unfathomable. Supply and demand says that if the price is right, someone will pay. Second, Comcast is double-dipping and trying to sell access to their customers as a scarce resource; in other words they&#8217;re trying to force Level 3 to pay more for the bandwidth than it would sell for on the open market, because buying it from Comcast is the only way to get access to Comcast&#8217;s customers. Comcast has a monopoly on its customers (who already pay them, mind you).</p>

<p>As for Cogent, they were sending traffic over Level 3&#8242;s network to third party networks. That is, they weren&#8217;t delivering traffic to Level 3 customers, they were delivering content to people Level 3 isn&#8217;t getting paid by, while not paying Level 3 themselves. So Cogent was <strong>making</strong> money, and Level 3 was <strong>losing</strong> money. In Comcast&#8217;s case, Level 3 is delivering content to Comcast customers, who <strong>are</strong> paying Comcast for that content.</p>

<p>Level 3 has no monopoly on access to third party networks; there are other backbone providers. Comcast has a monopoly over access to Comcast customers. This is the main difference.</p>

<h3>Why You Should Care</h3>

<p>The people that are calling this a net neutrality issue aren&#8217;t far off: Comcast isn&#8217;t filtering based on content; it&#8217;s actually much worse than that: Comcast is <em>charging content providers</em> in order to allow them to send their content to Comcast&#8217;s customers. This is absurd. If this is allowed to continue (which all of the ISPs are hoping for judging by Time Warner&#8217;s recent endorsement of the Comcast/Level 3 deal) then you&#8217;ll only be able to get Netflix from Sprint, and you&#8217;ll only be able to get Hulu on Comcast, and you&#8217;ll only be able to get to YouTube from, well, whomever YouTube decides to pay. Because ISPs are so large they have the only pipe leading to tens of millions of people and they&#8217;ve decided they should be able to charge the rest of the world to send those people the content those people want to see, which is in fact the very reason those people pay those ISPs to begin with.</p>

<h3>So Why Pay Comcast?</h3>

<p>Level 3 is alleging they agreed to the deal under duress, and are complaining about it now because now is the only time they reasonably could. If they had said &#8220;suck it&#8221; and disconnected from Comcast, any traffic that Level 3 needs to send to Comcast customers (namely Netflix, which they have a contracted obligation to do) would have to be <em>transmitted</em> over one of Level 3&#8242;s peers, and then routed to Comcast.</p>

<p>Ignore that Comcast would march over to the new peer and demand the same thing they&#8217;ve demanded from Level 3. Ignore that Level 3 would undoubtedly have to pay quite a bit of money to this peer to <em>transmit</em> their Netflix traffic to Comcast&#8217;s customers. Doing this would seriously degrade the quality of the video Level 3 is transmitting because now it has to take (at minimum) one more step between Level 3 and the customers that are paying both Comcast <strong>and</strong> Netflix for high quality video. You can bet that Comcast knew they couldn&#8217;t say &#8220;blow us&#8221; without violating their contract with Netflix.</p>

<h3>Not-A-Car Analogy</h3>

<p>Say you rent a DSL line from the telephone company, but (because you&#8217;re currently free to do so) you use an ISP that is not the telephone company. You&#8217;re assigned a bandwidth of 1000Kbps downstream and 256Kbps upstream. You download 200GiB per month, and you upload 50GiB per month from the ISP (which doesn&#8217;t own your line). In this case, your ISP is sending your line far more traffic than you&#8217;re sending back to their lines. Under Comcast&#8217;s new logic, the ISP should be paying you in this situation. You own the line, it costs you money for them to send data to you (never mind you&#8217;ve requested it, just like Comcast&#8217;s customers), and it&#8217;s not balanced because you&#8217;re not costing them the same amount.</p>

<p>Does this make any sense at all to any sane person? Of course it doesn&#8217;t. The ISP would laugh your invoice all the way to the trash.</p>
]]></content:encoded>
			<wfw:commentRss>http://blog.ericw.org/2010/12/why-comcast-is-lying/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Winter&#8217;s Heart</title>
		<link>http://blog.ericw.org/2010/10/winters-heart/</link>
		<comments>http://blog.ericw.org/2010/10/winters-heart/#comments</comments>
		<pubDate>Tue, 05 Oct 2010 21:39:59 +0000</pubDate>
		<dc:creator>Eric Will</dc:creator>
				<category><![CDATA[life]]></category>
		<category><![CDATA[coping]]></category>

		<guid isPermaLink="false">http://pain.ericw.org/?p=110</guid>
		<description><![CDATA[Winter is coming, and so is pain.</p]]></description>
			<content:encoded><![CDATA[<p>I hate winter.</p>

<p>Autumn is my favorite season back home in St. Louis. Out there the blasting heat slacks off to reveal a serene climate with sunny blue skies and a constant cool breeze. The leaves spend a leisurely month or two shifting between various hues of orange and red before finally releasing their grip and slipping softly to the ground where they inevitably cover the yard and require raking and burning more than a few times (for the sorts of people that enjoy lawn work). Autumn is more like a soft summer, where you can relax outside without sweating or shivering. Baseball winds down and we enjoy the post-season. Long drives down the river with the windows (or top) down, taking in the crisp air the water seems to breathe. This lasts through October (it&#8217;s never too cold to go trick-or-treating) and well into November. Back home in St. Louis, winter&#8217;s heart rarely arrives before January.</p>

<p>Here in Maryland there doesn&#8217;t seem to be an autumn so much as a brief transitory period between temperatures in the 90s and temperatures in the 40s. The leaves briefly begin to redden and suddenly skip the other colors of the spectrum and drop to the ground, dead. It&#8217;s almost as if winter is in a hurry to reestablish it&#8217;s firm grip over the land and air, and freeze my poor joints to bits. I haven&#8217;t met anyone from Maryland who lists autumn as their favorite season. I can&#8217;t say that I blame them. For autumn is so bleakly chilling here, and with cold comes pain.</p>

<p>I used to think it was an urban legend; why would the cold make things like arthritis hurt more? The temperature in your body doesn&#8217;t change, so why would it? The most common explanation&#8211;changes in barometric pressure&#8211;don&#8217;t even hold much sense either. Barometric pressure doesn&#8217;t change much more than 1mm/Hg during the seasons, and it changes a lot more than that during thunderstorms. Thunderstorms certainly don&#8217;t make my pain worse. It doesn&#8217;t make much sense. That, of course, doesn&#8217;t matter to pain. Pain doesn&#8217;t care about urban legends or even much about making sense, and so pain does hurt more in the winter. I&#8217;ve been living with significant pain for a number of years now, and I have come to realize that my pain at least doubles&#8211;if not triples&#8211;in the winter months. They are miserable, and it seems every single day is a struggle to make it through and try to find a few hours of sleep somewhere in-between the hurt. Increasing your medication for this period of course means you&#8217;ll have to go through the struggle of decreasing it when the weather warms, if you do indeed decrease it at all. Decreasing pain medication if your pain hasn&#8217;t lessened is a daunting task. One must usually simply grin and bear it, and hope for the warmth of the sun.</p>

<p>This bleak week of weather we&#8217;re experiencing here in Maryland now is but a grim warning of things to come. Last year I thought this week was a fluke in the weather and that at least a little bit of autumn would return to embrace me for a few weeks yet, but this year I know better. I know winter&#8217;s heart is coming fast, and with it, my pain is coming on strong. I keep reminding myself I need to move somewhere quiet and gently warm. I love the weather, but I&#8217;d rather have boring weather than furious hurt.</p>

<p>I never quite understood exactly why I hated winter. Now I know.</p>
]]></content:encoded>
			<wfw:commentRss>http://blog.ericw.org/2010/10/winters-heart/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>New Hope for Novel Analgesics</title>
		<link>http://blog.ericw.org/2010/09/new-hope-for-novel-analgesics/</link>
		<comments>http://blog.ericw.org/2010/09/new-hope-for-novel-analgesics/#comments</comments>
		<pubDate>Fri, 03 Sep 2010 21:38:44 +0000</pubDate>
		<dc:creator>Eric Will</dc:creator>
				<category><![CDATA[medication]]></category>
		<category><![CDATA[pharmacology]]></category>
		<category><![CDATA[research]]></category>
		<category><![CDATA[technical]]></category>

		<guid isPermaLink="false">http://pain.ericw.org/?p=92</guid>
		<description><![CDATA[A lengthy piece examining acetaminophen's metabolic path into AM404 and the activation of the endogenous cannabinoid system.</p]]></description>
			<content:encoded><![CDATA[<p>Pretty much everyone knows what acetaminophen is. If you don&#8217;t, acetaminophen is the active ingredient in the brand names Panadol and Tylenol. Acetaminophen is known by different names&#8211;especially outside the United States&#8211;and is most commonly called paracetamol and often abbreviated APAP (from here on out). All of these names come from the chemical name, n-acetyl-para-aminophenol. APAP is notable as one of the first non-opioid (non-narcotic) analgesics without anti-inflammatory properties (this honor actually goes to the drug phenacetin, which was widely used but taken off the market in 1983 due to carcinogenicity concerns. APAP is a metabolite of phenacetin). It is a pain-relieving (analgesic), fever-reducing (antipyretic) drug in the aniline class, of which itself is the only remaining member.</p>

<p>Until recently, pharmacologists did not fully understand APAP&#8217;s mechanism. That is, exactly how does it relieve pain and reduce fevers? Considering the only other non-opioid analgesics consist entirely of the non-steriodal anti-inflammatory drugs (NSAIDs) such as ibuprofen and naproxen, APAP&#8217;s mechanism was assumed to be a similar one. NSAIDs work by inhibiting enzymes called cyclooxygenase (COX) which produce chemical messengers called prostiglandins which set off inflammation and pain. While NSAIDs markedly reduce inflammation, there is almost little to no inflammation reduction with APAP usage. Why is this?</p>

<p>There are two varieties of cyclooxygenase: COX-1 and COX-2. Most NSAIDs inhibit both of these types equally. COX-1 inhibition has the unwanted side-effect of reducing protective liners in the stomach which can lead to gastric bleeding (indeed, the number one problem with NSAID use). However, inhibition of COX-2 does not produce this effect. Due to this, a number of drugs were developed that selectively inhibit COX-2 while leaving COX-1 alone, and these drugs were called &#8220;COX-2 inhibitors&#8221; with drug name suffixes of &#8220;coxib,&#8221; for &#8220;COX inhibitor.&#8221; Examples of such drugs include valdecoxib, rofecoxib, and celecoxib. A number of these drugs were developed and were very well-regarded by pain management physicians and chronic pain patients alike for their excellent ability to lower pain and inflammation without marked side-effects and even alleviated the need for opioid use (or at least reduced it). Unfortunately some of these drugs were abruptly removed from the United States market and, aside from celecoxib, no new COX-2 inhibitors have been approved or remain on the US market.</p>

<p>So where am I going with this? As APAP&#8217;s mechanism becomes more clear, recent findings have suggested that APAP is strongly selective of COX-2 (so much for the need to remove them from the market). So while APAP does indeed inhibit COX like the NSAIDs, there is strong evidence that APAP works through at least two pathways. The first one is well-researched and well-understood (COX inhibition), and the second pathway is what we&#8217;re interested in. So what exactly is going on here?</p>

<p>Recent research suggests that APAP may earn its analgesic and antipyretic properties by indirectly activating the endogenous cannabinoid system. The same way that opioids activate our own natural pain-relief system that endorphins and other natural ligands use, the body also has a natural cannaboinoid system which is responsible for the effects of tetrahydrocannabinol, or THC, which is the main active ingredient found in marijuana. Just like morphine binds to opioid receptors (mu, kappa, delta, and others), drugs like marijuana bind to the cannabinoid receptors <strong>CB1</strong> and <strong>CB2</strong>. A well-known natural opioid is endorphin. There are also natural cannabinoids, and the one floating around our brains is called <strong>anandamide</strong>. The entire purpose of the endogenous cannabinoid system has yet to be fully elucidated, but we will explore some of the regulatory functions they serve below.</p>

<p>When you take APAP, it is metabolized by the body into a number of different chemicals. Some are active, some are inactive. One particular metabolite is taken in by an enzyme in the body called <strong>fatty acid amide hydrolase</strong> (or FAAH), which converts it into a metabolite called <strong>AM404</strong>. <a href="http://en.wikipedia.org/wiki/AM404">AM404</a> is versatile. It&#8217;s effect is as an analgesic and an antipyretic (sound familiar?). AM404 inhibits FAAH, which also metabolizes <a href="http://en.wikipedia.org/wiki/Anandamide">anandamide</a> (the natural cannabinoid). The net effect is that anandamide uptake is inhibited, and levels of anandamide in the brain increase. AM404 also directly inhibits the formation of COX-1, COX-2, and prostaglandins (sound even more familiar?). AM404 also activates a receptor called <strong>TRPV1</strong>, which is also where the substance capsaicin (the substance that makes hot peppers hot) binds. <a href="http://en.wikipedia.org/wiki/Vanilloid_receptors">TRPV1</a> is responsible for pain transmission and thermoregulatory actions. When activated, TRPV1 enhances and modulates pain transmission, and also tells the body to cool itself down. However, when TRPV1 is bound to for long periods of time it &#8220;shuts down,&#8221; preventing it from functioning, thus reducing pain.</p>

<p>So let&#8217;s take a step back. We&#8217;ve got a lot of things going on. Thanks to AM404&#8211;which is introduced by acetaminophen&#8211;we have the following things going on:</p>

<ol>
<li>AM404 inhibits FAAH&#8211;which <a href="http://en.wikipedia.org/wiki/Anandamide#Synthesis_and_degradation">metabolizes anandamide</a>&#8211;resulting in an increase of anadamide.</li>
<li>Anadamide binds to cannabinoid CB1 and CB2, and also activates the TRPV1 receptor. Each of these actions are known to inhibit pain on their own.</li>
<li>AM404 also activates the TRPV1 receptor.</li>
<li>AM404 also inhibits cyclooxygenase and prostagladins.</li>
</ol>

<p>All of these processes are working to reduce pain (and fever). So, what does this really matter? By investigating these processes we can create novel analgesic drugs that aim to inhibit FAAH in the same way AM404 does (APAP&#8217;s use itself is limited due to its toxicity at higher doses) and giving rise to this exact process. We can also make drugs to target TRPV1, and in fact there are <a href="http://en.wikipedia.org/wiki/Vanilloid_receptors#Clinical_significance">already several</a> in advanced testing phases (both agonists and antagonists are being explored, but I&#8217;d personally be interested in a partial agonist&#8211;can we activate and overload it without causing the burning sensations?).</p>

<p>Let&#8217;s remember, this started by looking closely at the metabolism and mechanism of a drug almost everyone worldwide knows of and has made use of: acetaminophen. First we found out that APAP is most likely a highly selective COX-2 inhibitor, and so that trash about taking Bextra and Vioxx off the market was just that: trash. More importantly&#8211;if you&#8217;ve managed to follow along&#8211;you&#8217;ve almost certainly deduced that because acetaminophen introduces AM404, and AM404 causes activations in the endocannabinoid system, and in this fashion acetaminophen acts as a pro-drug for a cannabimimetic metabolite (AM404 itself), this means that Tylenol and Panadol and hugely popular drugs containing acetaminophen are activating the endocannabinoid system&#8211;like marijuana&#8211;in order to produce it&#8217;s primary effect of analgesia. Tylenol&#8217;s pain-relieving action involves activation of the endogenous cannabinoid system.</p>

<p>And marijuana is illegal?</p>
]]></content:encoded>
			<wfw:commentRss>http://blog.ericw.org/2010/09/new-hope-for-novel-analgesics/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>An Unfortunate Series of Events</title>
		<link>http://blog.ericw.org/2010/06/unfortunate-events/</link>
		<comments>http://blog.ericw.org/2010/06/unfortunate-events/#comments</comments>
		<pubDate>Fri, 11 Jun 2010 22:16:13 +0000</pubDate>
		<dc:creator>Eric Will</dc:creator>
				<category><![CDATA[life]]></category>
		<category><![CDATA[medication]]></category>
		<category><![CDATA[addiction]]></category>
		<category><![CDATA[adverse effects]]></category>
		<category><![CDATA[prejudice]]></category>

		<guid isPermaLink="false">http://pain.ericw.org/?p=84</guid>
		<description><![CDATA[I suddenly find myself without a pain physician or anyone willing to continue my current treatment. It's not a fun story.</p]]></description>
			<content:encoded><![CDATA[<p>In March, I was told by my pain management physician that he&#8217;d be leaving the office and indeed Hopkins all together. Long story short, Hopkins wanted him to do a bunch of things to patients purely for the sake of bringing in revenue, and he refused to do so and instead quit, taking several other physicians with him. I was told, at the time, that he&#8217;d be there until May and in May I&#8217;d get several post-dated prescriptions to cover me until he had his private practice set up in June. That plan&#8211;like so many plans&#8211;didn&#8217;t quite turn out that way.</p>

<p>In May I met&#8211;for the last time&#8211;with my pain management physician. He refilled all my medications and told me which physician I&#8217;d be seeing until he gets set up somewhere. This was now not a certainty as I&#8217;d been led to believe before. Now he may or may not continue his practice elsewhere, and I&#8217;m somewhat concerned. I take the scripts and bid him farewell.</p>

<p>Thirty days later it&#8217;s time for a refill. I don&#8217;t have a treating physician anymore. I call the office and they arrange for a one-time 30 day refill of my medications in exchange for a urine sample. Apparently it was horrible of my previous physician to simply trust me and not require urine analyses (UAs) in the past, and some more stuff about why he&#8217;s a bad doctor and I should stay there. I&#8217;m told to make an appointment with the new physician. The appointment was 52 days away, and I had scripts for 30 days.</p>

<p>Thirty days later it&#8217;s time for a refill. I call up and this time I&#8217;m told that I&#8217;m not allowed a refill. Why not? Apparently it was &#8220;made clear to me&#8221; that the previous script was a &#8220;one-time thing&#8221; (which it wasn&#8217;t [made clear to me]). I was supposed to get an appointment before those scripts ran out! Wait, I <em>did</em> make an appointment. It was 52 days away. How am I supposed to get one 30 day script when my appointment was more than 30 days away? Well, according to &#8220;the system&#8221; my appointment was &#8220;just created&#8221; (likely because someone modified it or something and it changed the date it was edited) and that means I didn&#8217;t make an appointment and it&#8217;s my fault. I go in and they decide that, for yet another urine sample, I may obtain 14 days worth of my medication. Except that you can&#8217;t fill 14 days of fentanyl patches because they come in boxes of 5 and 14 days would be 7 patches and they can only fill them per box, so they only give me 5. I tell the office this. When my urine comes back &#8220;clean&#8221; they give me another 30 days.</p>

<p>Not quite thirty days later is finally time for my appointment. I come in for my new physician, the one that&#8217;s been writing these scripts, and the first words out of his mouth consist of some FUD about my current treatment being all wrong, and it&#8217;s very wrong for someone as young as me to be on &#8220;two hardcore narcotics&#8221; (he specifically and repeatedly used the word &#8220;narcotic&#8221; over &#8220;opiate&#8221; or &#8220;opioid&#8221;&#8211;both of which are more appropriate for medical professionals&#8211;because &#8220;narcotic&#8221; is a scary-sounding word) and instead I should look into permanent back surgery. I argue that I have no surgical options given my condition and that I&#8217;ve already had every interventional procedure that I <em>could</em> have. He tells me of an orthopedic surgeon I should immediately go see for a second opinion. As it turns out I know this orthopedic surgeon. He referred me to pain management&#8211;to the very office I am currently sitting in. When I tell him this I&#8217;m still instructed to &#8220;go back&#8221; to get new radiographs and with new radiographs, a new opinion on surgery. The surgeon had before told me the only thing he could offer me were spinal fusions. I turned him down given my age, which he agreed with, and he sent me to pain management which he deemed, in his own words, &#8220;the only reasonable next step.&#8221; Two years later my only reasonable next step is backwards, it seems. The new physician continues to try to alarm me about my current treatment, telling me that my medications are &#8220;dangerous and addictive.&#8221; I tell him I&#8217;ve been on them off and on for four or five years and have yet to have any issues with abuse or addiction, and that every UA he&#8217;s ordered on me has come back pristine. He tells me that <strong>narcotics</strong> build up in the body and <em>always</em> eventually cause addiction and that they&#8217;ll make me sleepy and surely I must be intolerably constipated and throws in every other classic opioid side effect, trying to get me to admit that there&#8217;s a reason to discontinue the therapy. I tell him no, the only side effect I even notice is constipation and I simply have a fiber bar or two every day and don&#8217;t even notice the difference. He&#8217;s angry, realizing that I&#8217;m actually educated about my condition and the treatment and ties up the appointment by telling me I have until August to find &#8220;other options&#8221; before he discontinues my medications.</p>

<p>One of the best parts of this whole thing is that when I first got there a resident finishing up her fellowship came in and talked to me first. After she went through everything she agreed with my current treatment and, after asking how it was working, thought I should increase my fentanyl dose (which I&#8217;ve been wanting to do for months). When the attending came in and shot down all that with his bullshit, she looked rather put off and disappointed. I don&#8217;t know if she was disappointed because he didn&#8217;t agree with her and thus she must be wrong, or because he was being so obviously obtuse about my treatment and there was nothing she could do to help me. The best part of this was that his own Fellow disagreed with him and thought my current treatment should stand. I liked her. She was a very nice lady. I&#8217;m sad I&#8217;ll never see her again.</p>

<p>Currently I&#8217;m waiting for some results from various tests I had with my rheumatologist. He&#8217;s a very smart physician. While he doesn&#8217;t know exactly what&#8217;s wrong with me (he suspects an extremely mild non-specific &#8220;type&#8221; of <a href="http://en.wikipedia.org/wiki/Ehlers-Danlos_syndrome">Ehlers-Danlos Syndrome</a>, but it&#8217;s a wild shot) he told me he was positive it wasn&#8217;t progressive osteoarthritis, which is what my pain clinic has been telling me for two years. They also tell me it &#8220;just happens,&#8221; even to males in their early 20s. I&#8217;m also expected to see the orthopedic surgeon that originally referred me to the pain clinic in the first place. I&#8217;m not totally sure I&#8217;m going to do this, because I don&#8217;t see the point. I&#8217;m not going to have surgery, and the last time I saw him my options were &#8220;surgery or pain management.&#8221; Guess which one I chose.</p>

<p>Monday I&#8217;m going to talk to my primary care physician about what she could do for me if I lose my medications. The Monday after that I&#8217;m going to get the results of all my rheumatology tests, and maybe have an answer and even treatment. If all of this fails, come August I&#8217;ll have no answers and no pain management. I will have to quit my job, move home to St. Louis, and try desperately not to curl up into a miserable ball of existence, wondering why I should continue to bother to live.</p>
]]></content:encoded>
			<wfw:commentRss>http://blog.ericw.org/2010/06/unfortunate-events/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
	</channel>
</rss>

