Fixing slow HTTP PUTs with Rails and Typhoeus

by Luke Rodgers on August 13, 2014

I’m working on a project that involves communication between two Rails apps, one as an API and the other as a backend for web app. The web app communicates with the API via the excellent Typhoeus ruby gem, and everything was going swell until I got to implementing updates via HTTP PUTs for a couple resources.

The problem was that they were way slower than any of the other requests, regardless of payload size–slower than GETs, DELETEs, even POSTs. I tried a PATCH and that was as fast as the rest.

Since at the time I was originally debugging this I had no Internet access, I began throwing logging into the internals of Typhoeus everywhere, and eventually narrowed it down to the Ethon gem, which wasn’t too much help since it is basically just an interface to the CURL library. But when I did a plain ol’ CURL PUT to the same endpoint, everything worked just fine.

I tried using tcpump to diagnose the problem, suspecting that it lay in a difference in the way commandline CURL and the Typhoeus use were encoding the request, but my tcpdump-fu was weak: I figured the traffic I was interested would not be going over any of the usual interfaces listed by `ifconfig -a` but didn’t know which one it was, and it was around this time that I temporarily gave up, figuring some quick Googling would solve these problems when I got back online.

It turns out that this traffic goes over the “loopback” interface, which is in fact listed in the output of `ifconfig -a` as the first one on OSX: `lo0`, so I could have run `tcpdump -i lo0`. But now that I had the Internets back, I installed Wireshark, which for some reason I didn’t already have, and fired it up, setting a display filter for http traffic only.

After making a command line CURL PUT request, which was speedy, and a slow ruby Typhoeus request from the rails console, I had a look at the relevant captures in Wireshark and quickly noticed this difference:

Wireshark CURL request

 

Wireshark Typhoeus request

You’ll notice that the second, slow one is composed of two separate TCP frames.

Examining the headers further, I noticed this:

Expect: 100-continue header

Expect: 100-continue header

Turns out that despite all the times I’ve looked at the list of HTTP codes I’ve always just skipped over the 1XXs. Wikipedia has a more detailed breakdown, but the gist is that clients may send an `Expect: Continue-100` header so that the server can check if a request is appropriate based solely on the headers, which can be useful for large requests. It turns out that the libcurl will send this header by default, though you can disable it with the `-H` command line flag, which is what my CURL commands were doing by specifying various content-type and authorization headers.

It also turns out that CURL will just send the second request anyway after one second if it doesn’t receive a 100 code, which explains why my API client was consistently slow but not broken.

Now, how to work around this. Some Googling revealed some people monkeypatching the Rails default Webrick server, which I didn’t want to do. I could also have hacked Typhoeus or Ethon, but that didn’t seem like the way to go either. The best way, it seemed, was to take advantage of this feature and have my application return a 100 code if the user was indeed authorized to make the request.

The sensible place to put this code, it seemed to me, was in Rack middleware. A barebones implementation could look like so:

class Preauthenticator
  def initialize app
    @app = app
  end

  def call env
    if env["HTTP_EXPECT"] =~ /100-continue/
      return [100, {}, [""]]
    else
      @app.call(env)
    end
  end
end

 

(For an example from the Unicorn source code, see here.)

The only tricky part here was figuring out how exactly to get Rails to use it. Middleware generally goes in your `config/application.rb` file (make sure to require your file with the middleware class in it at the top of that file), but including this like so: `config.middleware.use Preauthenticator` resulted in rack complaining about a “app error: deadlock; recursive locking (ThreadError)”. Running `rake middleware` will list all the middleware used by your Rails app; it seemed to me that my Preauthenticator should run before any of this, so I tried `config.middleware.insert_before Rack::Sendfile, Preauthenticator`, Rack::Sendfile being the first middleware listed, and that worked like a charm.

Now my HTTP PUTs were as speedy as the rest of my requests.

TL;DR

Typhoeus and CURL will make some PUTs and POSTs slow by sending an “Expect: 100-Continue” header. Add rack middleware that returns this header to make your PUTs and POSTs faster.

Rails 4 scopes with has_many :through relations

by Luke Rodgers on April 20, 2014

Given the following Rails 4 model:

class Producer < ActiveRecord::Base
  has_many :producer_types
  has_many :types, :through => :producer_types
end

There are a couple ways you could create a scope on this model to retrieve only producers of a certain type.

First:

  scope :manufacturer, -> { where(:types => {:name => 'manufacturer'}).joins(:types) }

Second, to generate a dynamic scope for the given column name, you can take advantage of the following active record functionality:

  scope :with_type_name, lambda {|type_name|
    where(:types => {:name => type_name}).joins(:types)
  }

Configuring SoftEther VPN on Ubuntu with a firewall

by Luke Rodgers on March 30, 2014

DigitalOcean has a good article on setting SoftEther VPN on one of their droplets.

The instructions worked for me without just fine, with a couple exceptions:

1. When you attempt to use the command line vpncmd tool to set up the server, you may have to specify localhost:5555 rather than using the defaults.

2. You may have to create a group for the test user before creating them and assigning them to a test group. This can be done with the command `GroupCreate test`.

3. If you are using a firewall, you will need to open up the ports used by SoftEther. To figure out what ports it’s using, do `sudo netstat -atulpn  | grep vpnserver`. By default, SoftEther will listen on TCP ports 443, 992, and 5555. If you’re using L2TP/IPsec, make sure UDP ports 500 and 4500 are open as well. If you’re using ufw for your firewall, you can see which ports are open/blocked with `sudo ufw status verbose`. To get an idea of which ports each VPN protocol you’re using requires, check out the SoftEther specifications.

Read more literature

by Luke Rodgers on October 13, 2013

“When David called to say he and Patty were coming for a visit, Noel never thought of saying no. And he asked me how he could compete with David. He thought David was coming to his house to win me away. After he reads more literature he’ll realize that is too easy. There will have to be complexities. The complexities will protect him forever.”

From “Vermont,” by Ann Beattie

LAMP on Digital Ocean

by Luke Rodgers on August 25, 2013

I’ve been experimenting lately with using DigitalOcean both for remote dev environments (to facilitate development on a Chromebook) and also for staging/testing servers.

The plugin for Vagrant in concert with chef solo makes spinning up and provisioning new instances a breeze, but this post is about going the slightly more manual route, and using one of the application bundles (currently in beta) for Ubuntu 12.04.

These steps are a combination of things I’ve gleaned from various other places and my experience. I make no claim for their soundness.

Create the droplet

Create a new droplet using the Ubuntu 12.04 x32 (or x64–it shouldn’t matter for our purposes here), and select “LAMP on Ubuntu 12.04″ from the Applications tab.

At this point, I’m going to assume that you’ll also add your SSH public keys, so you don’t have to login with username and password credentials. (If you don’t have an SSH key, here are some instructions for generating one, you can just ignore the git-specific stuff.) An advantage of doing this is that your root password to the new droplet will not be sent over email (because there will no root password).

When you’ve selected all the options, click “Create Droplet.”

Login and Setup

Droplet creation should take a minute at most, after which you’ll see a screen with various information about your newly created vm, including an IP address. If you specified that your SSH keys should be automatically added, you should be able to SSH in now.

ssh root@ip.address.goes.here

First off, let’s change the default mysql root password as suggested by the login banner (note that the banner may continue to say that the password is still “password” even after you’ve changed it and logged in again):

mysqladmin -u root -p'password' password newpassword

Next, because we began with our SSH keys pre-installed, there’s no root password, so set one by typing `passwd` and following the prompts.

Next we’ll issue some commands with the package manager used by Ubuntu, `apt-get` to first update the list of available packages, and then upgrade the installed ones:

apt-get update
apt-get upgrade

Next, install fail2ban, a service that scans logfiles and auto-bans IP addresses that show signs of malicious activity, a good line of defense against crackers:

apt-get install fail2ban

Next, I’ll install my text editor of choice, vim:

apt-get install vim

As well as unzip…

apt-get install unzip

… and ack

apt-get install ack-grep

Adding a user

Because it is, for a variety of reasons, generally not a good idea to do things as root, let’s add a new user, create a home folder for them with the right permissions,  copy the contents of root’s authorized keys files to the new user’s .ssh folder, so we can ssh in as that user, give them a password, and set their default shell to bash.

useradd luke
mkdir /home/luke
mkdir /home/luke/.ssh
chmod 700 /home/luke/.ssh

And add the contents of the root user’s authorized_keys files to that of the new user:

cat .ssh/authorized_keys > /home/luke/.ssh/authorized_keys
chmod 400 /home/luke/.ssh/authorized_keys
chown -R luke:luke /home/luke
passwd luke
chsh -s /bin/bash luke

Now we’ll give that user the ability to run commands as root via `sudo`. Type `visudo` then enter this line, say, below the similar one for root (it doesn’t matter where, actually):

luke ALL=(ALL) ALL

Hit Command-X when done editing.

Now we’ll disable remote root login. Edit this file, /etc/ssh/sshd_config, with vim, or however you prefer, and make change “PermitRootLogin yes” to “PermitRootLogin no”, and uncomment the line “#PasswordAuthentication yes” and change it to “no”. This will mean you can only login on machines that have your SSH private key. Following that, we need to restart ssh:

service ssh restart

Now we’ll install a firewall to control which ports we allow traffic into. We’ll allow SSH and SFTP (port 22), HTTP (80), and HTTPS (443).

apt-get install ufw
ufw allow 22
ufw allow 80
ufw allow 443
ufw enable

You may get a warning after the last command about this disrupting your SSH session, but you should be able to ignore it.

Other dev tools

git, rvm + ruby, tmux

apt-get install git
apt-get install tmux
\curl -L https://get.rvm.io | bash -s stable --ruby

Miscellany

These are just some relevant notes and steps I’m including mostly for myself to remember for later:

FTP

In some cases you may need FTP and not just SFTP.

apt-get install vsftpd

Edit /etc/vsftpd.conf and change the following lines

anonymous_enable=NO
local_enable=YES
write_enable=YES

Save the file, and `/etc/init.d/vsftpd restart` then `ufw allow 21`.

Serving a git repository

If you’re serving files directly from a git repository, make sure you aren’t serving .git.

Throttle outbound Spotify traffic

by Luke Rodgers on August 2, 2013

When working on a shared network with limited bandwidth, it’s sometimes nice to be able to keep listening to Spotify without ruining your co-workers’ Internet connections.

Spotify is P2P (the desktop app is, anyway), so you both receive data from other Spotify users and transmit it to them. Blocking the outgoing traffic entirely would be un-neighbourly (probably also a violation of TOS), and might even prevent streaming from working altogether.

Fortunately, as I just discovered, ipfw allows you to add a pipe to a range of ports, all of which you can then throttle to a certain data transfer rate. It’s a shotgun approach, and will slow down any other services that are trying to send data over those ports as well, but since Spotify seems to stick to the range 10000 to 80000, and I rarely if ever run anything of consequence on those ports, this approach works for me.

sudo ipfw add pipe 1 ip from any to any out dst-port 10000-80000
sudo ipfw pipe 1 config bw 8KBytes/s

Filtering ExtJs Grid

by Luke Rodgers on April 24, 2013

ExtJs Grid provides a nice, quick way to build a UI that can handle a lot of tabular data and support the operations you’d typically like to perform on that data (sorting, filtering). It can be hard to style, but is great for whipping up admin CRUD functionality.

One thing that seems hard to do with it out of the box, however, is trigger filtering directly from your javascript, rather than via the UI.

Ext.grid.Panel does in fact expose the requisite properties and methods to accomplish this, but they’re a bit hard to track down, and not part of the official public API. This gist, GridFilteringExtensions.js, adds some methods to Ext.grid.Panel that make this easier. It probably won’t accomplish everything you want to do, and hasn’t been tested in a wide variety of situations, but it was exactly what I needed, and may be a good base from which to build.

Note that before manually setting a filter for the first time, you’ll need to call `createFilters`.

Closure Compiler externs for Backbone 1.0.0

by Luke Rodgers on April 16, 2013

I think they are reasonably complete. Bug reports, pull requests welcome.

Backbone 1.0.0 externs