Wishlisting Blog

(aka "How I spent my weekend")
 
If you're looking to deploy Rails in a production-quality way on a server, there are a ton of tutorials on how to do that.  None did exactly what I wanted though, so I'm adding another to the mix.  We're using <a href="http://www.mediatemple.net/webhosting/dv/">MediaTemple's (dv) 3.0</a> server.  Design goals:
<ul>
  <li>I wanted to use <a href="http://blog.codahale.com/2006/06/19/time-for-a-grown-up-server-rails-mongrel-apache-capistrano-and-you/">Mongrel instead of FastCGI</a>.</li>
    <li>Because this server comes with Apache pre-installed, and it's not 2.2 (which supports <a href="http://httpd.apache.org/docs/2.2/mod/mod_proxy_balancer.html">mod_proxy_balancer</a>), I decided to use the <a href="http://siag.nu/pen/">Pen</a> load balancer. (Coda Hale <a href="http://blog.codahale.com/2006/11/07/pound-vs-pen-because-you-need-a-load-balancing-proxy/">sold me on this</a>)</li>
  <li>I needed to support <a href="http://rmagick.rubyforge.org/">RMagick</a></li>
  <li>I wanted everything to be as simple as possible.</li>
</ul>
<span id="more-20"></span>
If you're a MediaTemple user, you'll notice they have <a href="http://kb.mediatemple.net/article.php?id=128">instructions for this</a> on the 2.0 edition of their dedicated virtual server, but not 3.0.  Consider this an upgrade/enhancement to those instructions.
 
<h2>Requirements</h2>
<ol>
  <li>This assumes MySQL and Apache are pre-installed (they were for me).</li>  
        <li>You'll need root access to the box.  Request this from MediaTemple.</li>
  <li>You'll need them to install the Developer Tools package (gcc, etc).  Request this from MediaTemple.</li>
  <li>I found that after this, Ruby was installed.  Verify that.
        <code>[root@gamma ~]# ruby --version
ruby 1.8.5 (2006-12-04 patchlevel 2) [i386-linux]

Installation

The biggest pain of this whole process is getting ImageMagick/RMagick working. These servers come without a package manager, and I didn’t want to install one just to install ImageMagick. For all the Ruby stuff, you can just use gems. So, these instructions walk you through downloading and compiling all of the required ImageMagick dependencies, without a package manager.

1. Installing ImageMagick Dependencies (yes, they’re all required for RMagick to work)
1. Install the libjpeg source (you may get errors copying man pages - ignore them I guess)
   wget http://www.ijg.org/files/jpegsrc.v6b.tar.gz
tar -xzvf jpegsrc.v6b.tar.gz
cd jpeg-6b/
./configure --enable-shared
make
make test
make install
 

2. Install the libpng source
   wget http://easynews.dl.sourceforge.net/sourceforge/libpng/libpng-1.2.16.tar.bz2
tar -xvjf libpng-1.2.16.tar.bz2
./configure
make check
 

3. Install the freetype source
   wget http://superb-west.dl.sourceforge.net/sourceforge/freetype/freetype-2.3.2.tar.bz2
tar -xjvf freetype-2.3.2.tar.bz2
cd freetype-2.3.2
make
make
cp objs/.libs/*.so* /usr/local/lib/
cp -rv include/* /usr/local/include/
 

4. Install the Ghostscript source
   wget http://umn.dl.sourceforge.net/sourceforge/ghostscript/ghostscript-8.54-gpl.tar.bz2
tar xjvf ghostscript-8.54-gpl.tar.bz2
cd ghostscript-8.54-gpl
./configure
make
make install
 

5. Install the Ghostscript fonts (thanks Hung)
   wget http://internap.dl.sourceforge.net/sourceforge/gs-fonts/ghostscript-fonts-std-8.11.tar.gz
tar -xvzf ghostscript-fonts-std-8.11.tar.gz
cp -rv fonts /usr/local/share/ghostscript/
 

2. Install ImageMagick
   wget ftp://ftp.fifi.org/pub/ImageMagick/ImageMagick-6.3.2-9.tar.bz2
tar -xjvf ImageMagick-6.3.2-9.tar.bz2
cd ImageMagick-6.3.2
./configure --without-perl --without-magick-plus-plus
 

   Make sure after the configure you see the lines:
   Ghostscript       None                          gs (8.54)
FreeType          --with-freetype=yes           yes
JPEG v1           --with-jpeg=yes               yes
PNG               --with-png=yes                yes
 

   Then install ImageMagick:
   make install
 


3. Install gems, your Ruby package manager
   wget http://rubyforge.org/frs/download.php/17190/rubygems-0.9.2.tgz
tar xzvf rubygems-0.9.2.tgz
cd rubygems-0.9.2
ruby setup.rb
 

4. Add the line /usr/local/lib to /etc/ld.so.conf and run ldconfig -v so Ruby can find ImageMagick
5. Install all of your gems. This will vary according to your needs. For example, we’re not moving to Rails 1.2 yet, which is why that version is restricted. Also, our application uses the hpricot and ruby-openid libraries.
   gem update --system
gem install rails --version '<1.2' -y
gem install mongrel_cluster -y
gem install hpricot -y
gem install ruby-openid -y
gem install rmagick -y
gem install mysql -- --with-mysql-include=/usr/include/mysql --with-mysql-lib=/usr/lib/mysql
 

6. Install the Pen load balancer
   wget ftp://siag.nu/pub/pen/pen-0.17.1.tar.gz
tar -zxvf pen-0.17.1.tar.gz
cd pen-0.17.1
./configure --with-daemon
make
make-install
 

7. Install Coda Hale’s Pen load balancer service, and create a new user for Pen to run as.wget http://blog.codahale.com/wp-content/pen-service_0.2.tar.gz
tar -xzvf pen-service_0.2.tar.gz
cd pen-service
mkdir /etc/pen
cp pen.conf /etc/pen
cp pen-service /etc/init.d/pen
chmod +x /etc/init.d/pen
mkdir -p /var/chroot/pen
useradd pen -s /bin/false -d /var/chroot/pen
chown pen /var/log/pen.log
chown pen /var/run/pen.pid
 


Configuration

1. Configure your mongrel cluster (see here for reference). The simplest way to do it is to run this command to generate a mongrel_cluster.yml file:
mongrel_rails cluster::configure -e production \
    -p 8000 -N 3 -c PATH_TO_YOUR_CURRENT -a 127.0.0.1

   This will create 3 Mongrel servers listening on ports 8000-8002. I check this file into source control alongside the other config files. Once you have this, you should be able to start up your cluster with the command:mongrel_rails cluster::start

2. Customize the load balancer service. If you followed my instructions, you’ve now got Pen v.17, whereas Coda’s service is only compatible with Pen v.10 (the one in Debian). To make his script compatible, I opened up /etc/init.d/pen and commented out the ‘chroot’ line (never worked for me) and ‘poll’ line (not valid in .17) from the “Commands” hash at the top of the script.
   I then customized pen.conf just a bit so that it would reflect the actual paths on my system, but left it listening on port 8080, I also made sure that it was going to connect to ports 8000-8002 (the default), which matches the configuration of our mongrel cluster.

   You should now be able to start pen with /etc/init.d/pen start And ps aux | grep pen to verify that it has started. If pen fails to start, the current init script will give you no feedback, unfortunately. If you need to debug pen, I recommend running it in the foreground (-f) with debug on (-d).

3. Configure Apache to talk to Pen. This involves creating a vhost.conf file here:
   /var/www/vhosts/<YOUR VHOST NAME HERE>/conf/vhost.conf

   And make sure Plesk knows to look at it by running this command:
   /usr/local/psa/admin/sbin/websrvmng -u --vhost-name=<YOUR VHOST NAME HERE>

   My vhost.conf file looks like this, and is largely based off of this article. This assumes pen will be listening on port 8080.

   DocumentRoot /var/www/rails/wishlisting/current/public
   <Directory “/var/www/rails/wishlisting/current/public”>
   Options +FollowSymLinks +Indexes
   Options -Includes -ExecCGI -MultiViews
   AllowOverride All
   Order allow,deny
   Allow from all
   </Directory>

   # I use utf-8 for all my projects, so I force apache to send the good charset by default.
   # This is needed if you use page caching and want apache serves these with the good charset.
   AddDefaultCharset utf-8

   # Do not allow open proxying, allow only requests starting with a /
   <LocationMatch “^[^/]”>
   Deny from all
   </LocationMatch>

   # Avoid open you server to proxying
   ProxyRequests Off

   # Let apache correctly rewrite redirect
   ProxyPassReverse / http://localhost:8080/

   # Let apache pass the original host not the ProxyPass one
   ProxyPreserveHost On

   RewriteEngine On

   # As I mentioned earlier, Capistrano disable_web task allows you to disable your application
   # In fact it only create a simple page named maintenance.html in the directory public/system/ of your application
   # The following mod_rewrite rules will tell apache to directly serve the maintenance.html pages if it find it out.
   RewriteCond %{DOCUMENT_ROOT}/system/maintenance.html -f
   RewriteCond %{SCRIPT_FILENAME} !maintenance.html
   RewriteRule ^.*$ /system/maintenance.html [L]

   # For static files it’s good to avoid hitting your mongrel process
   # so let apache knows it should serve it directly
   # for a rails application it means, files in images / stylesheets / javascripts
   RewriteRule “^/(images|stylesheets|javascripts)/?(.*)” “$0″ [L]
   # Try to match a cached pages
   # RewriteRule ^([^.]+)$ $1.html [QSA]
   # if the cached page does not exists
   RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
   # proxy requests to your pen instance
   RewriteRule “^/(.*)” “http://localhost:8080/$1″ [P,QSA,L]

I had to make a few tweaks to this file for it to work in my scenario. For one, I was noticing that all requests (even for cached content) were going to pen. This was alleviated by adding the “%{DOCUMENT_ROOT}/” part in the rewrite condition test. I also currently have the “.html” rewrite rule commented out for checking page caches. In place of it, on my production box, I have rules specifically tailored to my environment. I was finding that the “.html” rule was in turn sending requests to mongrel with “.html” at the end, which mongrel didn’t know what to do with. I don’t know enough about mod_rewrite to fix this.

Troubleshooting

• mod_rewrite rules don’t seem to be working - You can add the following two lines to your vhosts.conf file to help you debug mod_rewrite rules:
  RewriteLogLevel 3
RewriteLog /[path to a rewrite log file]
  That will dump some clues as to why your rewrite rules might not be working.
• 403 Errors I was getting 403 errors randomly which I didn’t understand, and so one of the things I did was comment out all of my mod_rewrite rules and just send all traffic to pen. I found out it was actually generating 502 errors. Apache was probably generating a 403 while it was looking for an error document to handle the 502. Sometimes the real problems can be obscured like this
• 502 Errors I was seeing proxy errors on pretty much every 3rd request (1 request to each mongrel, then errors intermittently after that). I ran pen in the foreground to watch it’s messages, and all of a sudden the problems went away. For some reason I have not yet determined, adding the -f switch and running pen in the foreground made these errors go away. Without it, I was seeing this in the error logs:
  error reading status line from remote server localhost
Error reading from remote server returned by /
  If anyone knows why this is the case, I’d love to hear it.
• Intermittent errors in general - use httperf, it will save your life. It gives you a way to make “random” errors quickly reproducible. I found that even a simple command like this would reproduce all of my proxy/permissions bugs:
  httperf --hog --server [my server] --num-conn 20 --ra 3 --timeout 40 --verbose
  It opens up 3 connections per second to your server, until it hits 20 connections. So, it should take under 7 seconds to run, and will tell you if you’re hitting problems in the summary.

Advanced Tips

• Serving static files from multiple subdomains. If you’re using Edge rails or the distributed_assets plugin, you know you can configure rails to serve images from multiple asset hosts. This allows you to get past the browser’s two-connection limit and request more files at once from your server. We were doing that, and I was finding that the assets weren’t going through our rewrite code, and as a result they were hitting mongrel. The fix was to add a ServerAlias that enumerates your asset hosts in the vhost.conf file.
  ServerAlias yourserver.com static1.yourserver.com static2.yourserver.com static3.yourserver.com static4.yourserver.com

Other Resources

There are a few links scattered about the instructions above, but these might also be helpful:

• MediaTemple KB: (dv) 3.0 HOWTO: Ruby On Rails using mongrel_cluster

Conclusion

Although getting it setup involved a lot of steps, what you end up with is actually quite simple. Apache talks to Pen, and Pen load balances to a Mongrel cluster. This deployment supports capistrano without any hacks or anything. If you have feedback on this post, please add comments and I will keep the guide up to date. I can add more detail where necessary.