Ok, I admit it. I am a Mongrel cluster user. Yes, I am familiar with Passenger and Thin and most of the other deployment options. Yes, I am aware they are supposedly better or faster or stronger or whatever when compared to a Mongrel cluster. However, up until now we have taken the safe and proven deployment strategy of using Mongrel clusters for our production projects (with all the challenges that entails…).

I have been interested in Apache + Passenger since Hongli Lai announced it because of the supposedly easier configuration and its good performance. I was impressed with the performance increase of Passenger + REE relative to Mongrel just like everyone else. When Passenger was announced we started examining it. We even went as far as deploying it on our staging servers and internal servers with good success. However, for various reasons, we never deployed our main production applications using Passenger.

Well, the recent introduction of Nginx + Passenger has caused us to examine our deployment options again. Nginx has long been regarded as using less memory and having better performance with cached content than Apache. There have been countless articles purporting that one option is better than another.

I don’t like speculation when it comes to deployment options. I prefer proof. In addition, keep in mind that this article is for my own benefit. Whatever option I find to be the best (for me) will be the option we use in the future. So, I have an interest in doing a thorough evaluation.

In this article, I make a very careful analysis comparing the performance of Nginx + Passenger to Apache + Passenger.

After hours of testing and analysis, I have decided that the answer depends upon your skill level. Read on for the details.

The contestants: Engine Yard Solo vs. EC2onRails

Engine Yard Solo

First up, we have Engine Yard’s new Solo product. According to Engine Yard:

“Solo is an inexpensive, web-based platform for on-demand management of your Ruby on Rails web application on Amazon’s cloud computing infrastructure. Manage your web application from development to deployment. You get the Engine Yard Ruby on Rails hosting and deployment expertise wrapped in an easy-to-use interface.”

The idea behind Solo appears to be that many people would like the flexibility of using Amazon’s EC2 environment but lack the technical expertise to configure the server themselves. Solo enables someone with little or no technical knowledge to set up and manage an EC2 server. To get an idea of how easy the system is to use, Engine Yard has a number of screencasts  online.

The Good: I created a Solo account and used it for several hours over the course of a few days in order to write this article. Happily, I can report that Solo performs as advertised. You will be able to very quickly get a Rails project up and running with Solo (assuming you have a fair understanding of deploying Rails applications. See below for a discussion). The Engine Yard team has successfully removed some of the complexity of running your own Rails server. They also provide a well optimized and secure server.

The not so Good: Given that Engine Yard appears to be targeting Solo at less knowledgeable users, I have a few issues with Engine Yard’s Solo.

1. Cost: In order to use Solo, you pay an additional $0.06 per hour for a small instance¹ (plus a premium over Amazon’s prices for the storage and bandwidth) ². This works out to be a minimum of $43.20 per small instance per month (plus storage and bandwidth charges).³
2. Lack of instructions: The instructions for new users are not sufficient.⁴ After you log in to Solo, you are greeted by a very clean layout, with no instructions or tips on what to do next. Should you “Create New Environment” or “Create New Instance” or one of the other choices? An inexperienced user may very well be lost at this stage. One of the things that 37Signals does very well is provide tips in order to get a new user started.
3. Deployment process: The steps that you must take in order to deploy a new application (like adding gems and Unix packages) are spread across numerous places in the system. These items should be placed into a more logical sequence or in some sort of wizard.
4. Log files: The process of viewing deploy log files is less than optimal. My application crashed several times during deployment until I had all of the configuration options set properly.⁵  In order to view the deploy log files, you must view them in the browser. Unfortunately, the deploy log files were very, very large which causes trouble with your browser. This caused trouble even on my machine, which is an 8 core machine with 16GB of ram. The log files should be downloadable.
5. Login process: This is a minor issue but there is no obvious login link. If you go to the Solo page, you will see a sign up link, but no link for returning users.⁶

Overall, these are minor issues that can easily be addressed in future versions of Solo.

The take so far: The current version of Solo is an excellent start. A few wizards and the liberal use of tips and suggestions could make this into a killer app.

EC2 on Rails

Next, we have EC2onRails. EC2onRails is an open source project led by Paul Dowman. According to the EC2 on Rails website:

“EC2 on Rails is an Ubuntu Linux server image for Amazon’s EC2 hosting service that’s ready to run a standard Ruby on Rails application with little or no customization. It’s a Ruby on Rails virtual appliance.”

Paul Dowman and the other EC2onRails committers (like Adam Greene (aka skippy)) deserve recognition for the massive amount of work they have done.⁷

The Good: The best part of the EC2onRails system is that it is open source. There is no charge for using the code and you have the ability to tailor it to your specific application. The EC2onRails code works well for creating an Apache + Mongrel cluster server. The EC2onRails team has done a good job configuring the various parts of the server. You can easily create an all-in-one server (with a web server, application server, and database) or split your application server (Rails) and the database onto separate boxes. Once you have a running server, it is monitored and maintained with tasks like database backups happening automatically.

The not so Good: Keep in mind that we are evaluating the experimental branch of EC2onRails running Nginx+Passenger, not the production branch of 0.9.9.1. So, some of my comments here will be addressed by the time this code reaches production quality. Given that this is an open source project, its weaknesses are like many other open source projects.

1. Support: The EC2onRails project is open source and lacks official support. It is intended for more advanced users. If you have an issue, your best bet is to reach out to the very helpful Google group.
2. Documentation: The documentation is good for deploying an Apache+Mongrel cluster configuration but is under revision for the Nginx+Passenger configuration. There are also items that are not covered in the documentation. For example, there are many configuration options and steps in the process that may require you to search the Google group or read the source code.
3. Ease of use: The system is not as easy to use as Solo. There are a number of steps that must be done manually in order to get a running server (e.g. deploying SSH keys). To be fair, most of these steps only need to be performed one time.
4. Nginx+Passenger is alpha: Support for Nginx+Passenger is currently experimental. Because of this, there is no AMI image available for you to use. This means that you must first create the AMI (a time consuming step) then configure your EC2onRails server.
5. The future? Now that Ubuntu is offering EC2 instances, the future of an important piece of the EC2onRails infrastructure is unclear. Eric Hammond’s ec2ubuntu-build-ami script is used when building a new AMI and it may not be supported by him in the future.⁸ This should not affect EC2onRails end users, but may require changes to the core EC2onRails code.

Overall, the EC2onRails project is a good example of a successful open source project. It has its rough spots but there are a good group of people using it and it is free :). Most of the issues listed above will be addressed by the time the code becomes official.

The take so far: The current version of EC2onRails offers an inexpensive and relatively easy way to get a Rails application running on EC2 using Apache+Mongrel cluster. If you want to deploy a Nginx+Passenger server, you will have a fair bit of work because the code is still experimental.

Here is my analysis of EC2onRails versus Engine Yard Solo so far:

So which one you use depends so far on your expertise. If you have good technical skills, use EC2onRails and save the money. If you are less technically inclined, use Solo. However, we have not yet examined PERFORMANCE! In short, is the Engine Yard Solo server faster than an EC2onRails server?

Hypothesis:

The performance of an Engine Yard Solo server will be the same as an EC2onRails server when configured using Nginx and Passenger.

Setup:

For these tests, I used 4 EC2 instances configured as shown here:

Test server configuration

Great care is required when doing performance tests to ensure your results are legitimate. There are many factors that affect server performance and not accounting for any of these factors can make your results invalid. For example, in many web server performance tests that I read, the authors simply run Apache Bench (ab) against a couple of configurations then proclaim a winner. A couple of potential problems with this approach are 1) the results are not statistically testable and 2) something may have changed on the server between test runs.

In order to get valid results, we need to take multiple samples of the performance. With those samples we need to get both the mean and the standard deviation. When we have this information, we can perform a two-sample t-test to statistically prove that one configuration is better. Fortunately, the httperf tool provides the information that we need.

For these tests, each test server instance was configured exactly the same. Each instance was created using the EC2onRails code from my fork on Github. They were EC2 Small Instances and had minimal services running. The EC2onRails server was also created using my EC2onRails branch. The Engine Yard Solo server was created using the Solo web-based GUI. Both servers used the exact same Rails code⁹ and had the exact same data in the database. All tests were performed at the same time on both configurations. Each test was then repeated on the other configuration to ensure the testing server was not biasing the results.

Details for the EC2onRails server:

• Ubuntu 8.04
• Rails 2.3.2
• Nginx 0.6.36
• Passenger 2.2.2
• MySQL
• Memcached

Testing process: (all of these steps are performed on each testing server at the same time)

• Warm up each server using ab:
  ab -n 100 -c 1 ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com/
• Run this a second time to ensure the server is ready.
• Check that all requests were served successfully.
• Using an automated script:
  • Send 5 requests using httperf to estimate the time required to get 45 samples.
  • Get at least 45 samples using httperf.
• Run the test again using the other testing server.
• Verify that the two samples from different testing servers are not statistically different from one another.

Results:

The test results for the EC2onRails server are not statistically different than the test results in my last post. This gives me confidence in the validity of these results.

No ActiveRecord:

The first set of tests was performed against a page that doesn’t access the database. This page is not cached and has numerous images, scripts, and partials. This should be a test of the instance’s ability to process Rails requests. If the Solo instance is better optimized it should show up in this test.

The results:

Conclusion: The Solo instance is statistically proven to be faster at the 99% confidence level. The Solo instance is about 16% faster than the EC2onRails server at serving standard Rails requests.

Moderate Database usage:

The next set of tests was performed against a page that has moderate database usage. This page accesses eight different models. The page is not cached and has several images, scripts, and partials. This should be a test of the instance’s ability to process Rails requests as well as handle the database load.

The results:

Conclusion: The Solo instance wins again by a fair margin. The Solo instance is statistically proven to be faster at the 99% confidence level. I was not expecting the Solo instance to perform 20% faster.

Heavy Database usage:

This set of tests was performed against a page that has heavy database usage. This page accesses numerous models. The page is not cached and has several images, scripts, and partials. This should be a test of the instance’s ability to process Rails requests as well as handle the database load. With the additional optimization, the Solo instance will probably win by a large margin.

The results:

Conclusion: The Solo instance wins again, although by a smaller margin. The Solo instance is statistically proven to be faster at the 99% confidence level. I thought that the Solo instance would perform better in this test. I was not expecting the Solo instance to only perform 10% faster.

Overall Conclusion:

The test results were surprising. The Engine Yard Solo server is 10%-20% faster than an EC2onRails instance. Hopefully the EC2onRails team can investigate performance and close this performance gap.

The Winner!

Picking a winner in this case is difficult. My recommendation will depend upon your level of expertise. Engine Yard Solo does offer a fast and convenient way to create EC2 instances. Solo is also easier to use than EC2onRails and requires less technical knowledge to operate.

If you are not comfortable building your own server, I suggest that you seriously consider Engine Yard’s Solo. Solo is a reasonably priced way to get a professionally configured Rails server in the EC2 environment. You should also consider some of the other solutions like RightScale (which is much more expensive than Solo).

If you are comfortable setting up your own server, I recommend EC2onRails. It is a great project that is constantly evolving. Personally, I have difficulty justifying the cost of Solo. It seems to me that you are paying $43.20 per month for Engine Yard to configure a server for you. I would find a one time setup charge much more appealing than paying over and over. The EC2onRails project allows you to set up EC2 instances for free that performs almost as well as the Solo instance.

In short, both choices are excellent. The best choice for you, will be up to you

What’s next?

In future posts, I will explore the various deployment options. Next up is Apache+Passenger vs. Nginx+Passenger.

Footnotes:

You will be happy to know that after hours of rigorous testing, I have identified a clear winner. Read on for the details.

For those of you that are not familiar with Amazon’s specifications for the two instance types:

Following the RailsLab recommendation means that you place the web / application server and database on separate instances for best performance. I am sure that two small instances configured as below would outperform a single small instance. However, we wondered how the High-CPU instance would compare against two Small instances when configured like this:

Two Small instances vs. One High-CPU instance

Hypothesis:

Two Small instances will outperform one High-CPU instance in a standard Rails application.

Setup:

For these tests, I used 5 EC2 instances configured as shown here:

Testing setup

Great care is required when doing performance tests to ensure your results are legitimate. There are many factors that affect server performance and not accounting for any of these factors can make your results invalid. For example, in many web server performance tests that I read, the authors simply run Apache Bench (ab) against a couple of configurations then proclaim a winner. A couple of potential problems with this approach are 1) the results are not statistically testable and 2) something may have changed on the server between test runs.

In order to get valid results, we need to take multiple samples of the performance. With those samples we need to get both the mean and the standard deviation. When we have this information, we can perform a two-sample t-test to statistically prove that one configuration is better. Fortunately, the httperf tool provides the information that we need.

For these tests, each instance was configured exactly the same. Each instance was created using the EC2onRails code from my fork on Github. Both the Small Instance cluster (Cluster) and the High-CPU Instance (High-CPU) were running the same version of all code. They also had the exact same data in the database. The code used in these tests is from a live production site. All tests were performed at the same time on both configurations. Each test was then repeated on the other configuration to ensure the testing server was not biasing the results.

Details:

• Ubuntu 8.04
• Rails 2.3.2
• Nginx 0.6.36
• Passenger 2.2.2
• MySQL
• Memcached

Process: (all of these steps are performed on each testing server at the same time)

• Warm up each server using ab:
  ab -n 100 -c 1 ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com/
• Run this a second time to ensure the server is ready.
• Check that all requests were served successfully.
• Using an automated script:
  • Send 5 requests using httperf to estimate the time required to get 45 samples.
  • Get at least 45 samples using httperf.
• Run the test again using the other testing server.
• Verify that the two samples from different testing servers are not statistically different from one another.

Results:

No ActiveRecord:

The first set of tests was performed against a page that doesn’t access the database. This page is not cached and has numerous images, scripts, and partials. This should be a test of the instance’s ability to process Rails requests. Given the additional “Compute Units” available to the High-CPU instance, I expect it to win easily.

The results:

Conclusion: The High-CPU instance is statistically proven to be faster at the 99% confidence level. The High-CPU instance is about 58% faster than the Cluster at serving standard Rails requests.

Moderate Database usage:

The next set of tests was performed against a page that has moderate database usage. This page accesses eight different models. The page is not cached and has several images, scripts, and partials. This should be a test of the instance’s ability to process Rails requests as well as handle the database load. With the additional memory and dedicated machine for the database, the Cluster might have a chance here.

The results:

Conclusion: The High-CPU instance wins again by a large margin. The High-CPU instance is statistically proven to be faster at the 99% confidence level. I thought that the Cluster would perform better in this test. I was not expecting the High-CPU instance to perform 69% faster.

Heavy Database usage:

This set of tests was performed against a page that has heavy database usage. This page accesses numerous models. The page is not cached and has several images, scripts, and partials. This should be a test of the instance’s ability to process Rails requests as well as handle the database load. With the additional memory and dedicated machine for the database, the Cluster might have a chance here.

The results:

Conclusion: The High-CPU instance wins again by a large margin. The High-CPU instance is statistically proven to be faster at the 99% confidence level. I thought that the Cluster would perform better in this test. I was not expecting the High-CPU instance to perform 120% faster.

Cached Page performance:

This set of tests was performed against a page that was cached. It has several images, scripts, and partials. This should be a test of the instance’s ability to server pure web server (nginx) requests. Since nginx uses such small amounts of memory and CPU, both configurations might be similar.

The results:

Conclusion: The High-CPU instance wins again by a very large margin. The High-CPU instance is statistically proven to be faster at the 99% confidence level. I thought that the Cluster would perform better in this test. I was not expecting the High-CPU instance to perform 277% faster.

UPDATE: Testing performance under load¹

Load Testing:

The advantage of having a separate database server should become more pronounced as the site becomes more heavily used. The tests so far were testing responsiveness not ability to handle load. In order to test load, I am using ab with 12 concurrent users like this: “ab -n 6186 -c 12 http://174.129.193.145/page/id” Each test was tailored to last about three minutes.

The results for the medium database usage page:

Conclusion: The High-CPU instance wins again by a very large margin. Notice in particular how the Cluster is slower overall and some of the requests queue up and take a very long time to serve. Only getting 80% of your requests in under 2 seconds is simply not acceptable in a production environment. On the other hand, the High-CPU instance is able to serve 100% of the requests in under 2 seconds.

I thought that the Cluster would perform better in this test. I was not expecting the High-CPU instance to perform 184% faster. Apparently the High-CPU machine is so much more powerful that even having the database on its own box isn’t enough for the small cluster to win.

Overall Conclusion:

The test results were very surprising. I expected the performance between the two configurations to be similar, particularly in database intensive tests.

Given the dramatically better performance of the High-CPU instance in every test situation, I can’t recommend using a cluster of two Small instances for a Rails application. Your money will be much better spent using a single High-CPU instance.

In future posts, I will explore the various deployment options. Next up is Apache+Passenger vs. Nginx+Passenger.

I have decided to set up a blog to talk about Ruby, Rails, and other tech related issues. I hope that you find something useful. If you have questions, comments, or suggestions, feel free to drop me a note.