Fixing slow HTTP PUTs with Rails and Typhoeus

I’m working on a project that involves communication between two Rails apps, one as an API and the other as a backend for web app. The web app communicates with the API via the excellent Typhoeus ruby gem, and everything was going swell until I got to implementing updates via HTTP PUTs for a couple resources.

The problem was that they were way slower than any of the other requests, regardless of payload size–slower than GETs, DELETEs, even POSTs. I tried a PATCH and that was as fast as the rest.

Since at the time I was originally debugging this I had no Internet access, I began throwing logging into the internals of Typhoeus everywhere, and eventually narrowed it down to the Ethon gem, which wasn’t too much help since it is basically just an interface to the CURL library. But when I did a plain ol’ CURL PUT to the same endpoint, everything worked just fine.

I tried using tcpump to diagnose the problem, suspecting that it lay in a difference in the way commandline CURL and the Typhoeus use were encoding the request, but my tcpdump-fu was weak: I figured the traffic I was interested would not be going over any of the usual interfaces listed by `ifconfig -a` but didn’t know which one it was, and it was around this time that I temporarily gave up, figuring some quick Googling would solve these problems when I got back online.

It turns out that this traffic goes over the “loopback” interface, which is in fact listed in the output of `ifconfig -a` as the first one on OSX: `lo0`, so I could have run `tcpdump -i lo0`. But now that I had the Internets back, I installed Wireshark, which for some reason I didn’t already have, and fired it up, setting a display filter for http traffic only.

After making a command line CURL PUT request, which was speedy, and a slow ruby Typhoeus request from the rails console, I had a look at the relevant captures in Wireshark and quickly noticed this difference:

Wireshark CURL request

 

Wireshark Typhoeus request

You’ll notice that the second, slow one is composed of two separate TCP frames.

Examining the headers further, I noticed this:

Expect: 100-continue header
Expect: 100-continue header

Turns out that despite all the times I’ve looked at the list of HTTP codes I’ve always just skipped over the 1XXs. Wikipedia has a more detailed breakdown, but the gist is that clients may send an `Expect: Continue-100` header so that the server can check if a request is appropriate based solely on the headers, which can be useful for large requests. It turns out that the libcurl will send this header by default, though you can disable it with the `-H` command line flag, which is what my CURL commands were doing by specifying various content-type and authorization headers.

It also turns out that CURL will just send the second request anyway after one second if it doesn’t receive a 100 code, which explains why my API client was consistently slow but not broken.

Now, how to work around this. Some Googling revealed some people monkeypatching the Rails default Webrick server, which I didn’t want to do. I could also have hacked Typhoeus or Ethon, but that didn’t seem like the way to go either. The best way, it seemed, was to take advantage of this feature and have my application return a 100 code if the user was indeed authorized to make the request. (Alternatively, you can just specify an Expect header with an empty value, rather than “100-continue”.)

The sensible place to put this code, it seemed to me, was in Rack middleware. (It should be noted at this point that this does not seem to work with Webrick or Thin.) A barebones implementation could look like so:

class Preauthenticator
  def initialize app
    @app = app
  end

  def call env
    if env["HTTP_EXPECT"] =~ /100-continue/
      return [100, {}, [""]]
    else
      @app.call(env)
    end
  end
end

 

(For an example from the Unicorn source code, see here.)

The only tricky part here was figuring out how exactly to get Rails to use it. Middleware generally goes in your `config/application.rb` file (make sure to require your file with the middleware class in it at the top of that file), but including this like so: `config.middleware.use Preauthenticator` resulted in rack complaining about a “app error: deadlock; recursive locking (ThreadError)”. Running `rake middleware` will list all the middleware used by your Rails app; it seemed to me that my Preauthenticator should run before any of this, so I tried `config.middleware.insert_before Rack::Sendfile, Preauthenticator`, Rack::Sendfile being the first middleware listed, and that worked like a charm.

Now my HTTP PUTs were as speedy as the rest of my requests.

As of Dec 14, 2014, the middleware here will not work with Webrick 1.3.1, Thin 1.6.3, or Puma 2.10.0 ruby servers.

TL;DR

Typhoeus and CURL will make some PUTs and POSTs slow by sending an “Expect: 100-Continue” header. Add rack middleware that returns this header to make your PUTs and POSTs faster.

Exploring Chrome’s timeline panel

Subsequent to some performance issues with a few Backbone apps, I spent some time digging further into Google Chrome’s timeline panel, and specifically the memory view that shows allocated memory, and breaks down the number of DOM nodes held in memory.

Some tests/demos along with provisional findings are provided here: chrome timeline exploration. The official documentation for Chrome’s dev tools is getting better, but could still use improvement. Hopefully this will go some way to providing a bit more insight into what’s going on, what different numbers mean, and what sort of behaviour you can expect from common scenarios.

As I’m nowhere near an expert of Chrome’s internals nor on memory profiling in general, any suggestions or corrections are more than welcome (pull requests or whatever).

Marxist or Marxian

Is there a difference?

Can someone tell me what the difference is between being a Marxist vs. being a Marxian, or whether there even is a difference? For a long time, I assumed there was no difference, and no distinction was needed. But the more Marx I read, the more value I see in having terminology to differentiate positions that are  “actually” Marx’s (i.e. for which there is relatively agreed-upon textual evidence) from those to which “Marxists” subsequently adhered, or promulgated.

Continue reading “Marxist or Marxian”

How not to do a call to action

As any email marketer knows, your call to action is a crucial element of your campaign. When tweaking your text can double your conversion rates, you can’t afford to ignore it. Even a 5% increase in conversion rates can translate into huge sales if you’re talking about thousands of customers.

So why PayPal, a company who you might think would know better, would use this confusing and non-clickthrough-inspiring call to action text is beyond me.

Continue reading “How not to do a call to action”

Google’s confusing and questionable advice on URL rewriting

Kudos to the Google Webmaster Central Blog for what seems like a conscious effort to try to address head-on the concerns of the community over common problem areas like duplicate content and 404s. But their most recent effort, which examines the pros and cons of statics vs. dynamic URLs from Google’s point of view, seem to have resulted in more heat than light, with a number of confused commenters responding to the post.

The gist of the entry was that you should probably leave your dynamic URLs as-is, because Google has no problem crawling them; re-writing them, if done improperly, can increase the difficulty of crawling your site, and is also hard to maintain.

I think Google seems to think that doing URL rewrites is more difficult than it is. I would like to think that most websites are either using a CMS that can adequately handle this (as is the case with WordPress, Drupal and Joomla! for instance), or are being run by someone who has the technical expertise to ensure that this is done appropriately and straightforwardly.

But even if that isn’t the case, Google’s advice here runs counter to the more reasonable advice provded in Tim Berners-Lee‘s W3C Style article, Cool URIs Don’t Change.

There are many reasons for using “cool URIs”, including the fact that they are easier to type, recognize and remember for people. One of the best reasons offered in this article, though, is that if you have a bunch of technology-dependent cruft in your URL, then you decide to switch the underlying technology, you’re going to end up with an entirely different URL structure, thus breaking all the bookmarks and links that have ever been made to your site.

I think the advantages of cool URIs outweigh the risks associated with mapping your dynamic URLs to static URLs, and it is kind of narrow-minded for Google to look at this only as a search engine crawling problem, rather than seeing it in a larger context.

UPDATE

There’s a good post over at SEOmoz on the same topic that lists a bunch of other reasons why, on balance, rewriting your dynamic URLs is still a good idea.