EC2onRails - 2 Small Instances or 1 High-CPU Medium Instance?

I recently watched the excellent RailsLab videos on tuning and optimizing Rails applications. One of the videos suggests that a great way to scale a Rails application is to separate the web and database components onto separate machines. We have been happily using Amazon’s EC2 system for about a year. Amazon’s recent introduction of the High-CPU Instance (High-CPU) made me curious how it would perform when compared against two Small Instances in a cluster (Cluster), since they would cost the same.

You will be happy to know that after hours of rigorous testing, I have identified a clear winner. Read on for the details.

For those of you that are not familiar with Amazon’s specifications for the two instance types:

Following the RailsLab recommendation means that you place the web / application server and database on separate instances for best performance. I am sure that two small instances configured as below would outperform a single small instance. However, we wondered how the High-CPU instance would compare against two Small instances when configured like this:

Two Small instances vs. One High-CPU instance

Hypothesis:

Two Small instances will outperform one High-CPU instance in a standard Rails application.

Setup:

For these tests, I used 5 EC2 instances configured as shown here:

Testing setup

Great care is required when doing performance tests to ensure your results are legitimate. There are many factors that affect server performance and not accounting for any of these factors can make your results invalid. For example, in many web server performance tests that I read, the authors simply run Apache Bench (ab) against a couple of configurations then proclaim a winner. A couple of potential problems with this approach are 1) the results are not statistically testable and 2) something may have changed on the server between test runs.

In order to get valid results, we need to take multiple samples of the performance. With those samples we need to get both the mean and the standard deviation. When we have this information, we can perform a two-sample t-test to statistically prove that one configuration is better. Fortunately, the httperf tool provides the information that we need.

For these tests, each instance was configured exactly the same. Each instance was created using the EC2onRails code from my fork on Github. Both the Small Instance cluster (Cluster) and the High-CPU Instance (High-CPU) were running the same version of all code. They also had the exact same data in the database. The code used in these tests is from a live production site. All tests were performed at the same time on both configurations. Each test was then repeated on the other configuration to ensure the testing server was not biasing the results.

Details:

• Ubuntu 8.04
• Rails 2.3.2
• Nginx 0.6.36
• Passenger 2.2.2
• MySQL
• Memcached

Process: (all of these steps are performed on each testing server at the same time)

• Warm up each server using ab:
  ab -n 100 -c 1 ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com/
• Run this a second time to ensure the server is ready.
• Check that all requests were served successfully.
• Using an automated script:
  • Send 5 requests using httperf to estimate the time required to get 45 samples.
  • Get at least 45 samples using httperf.
• Run the test again using the other testing server.
• Verify that the two samples from different testing servers are not statistically different from one another.

Results:

No ActiveRecord:

The first set of tests was performed against a page that doesn’t access the database. This page is not cached and has numerous images, scripts, and partials. This should be a test of the instance’s ability to process Rails requests. Given the additional “Compute Units” available to the High-CPU instance, I expect it to win easily.

The results:

Conclusion: The High-CPU instance is statistically proven to be faster at the 99% confidence level. The High-CPU instance is about 58% faster than the Cluster at serving standard Rails requests.

Moderate Database usage:

The next set of tests was performed against a page that has moderate database usage. This page accesses eight different models. The page is not cached and has several images, scripts, and partials. This should be a test of the instance’s ability to process Rails requests as well as handle the database load. With the additional memory and dedicated machine for the database, the Cluster might have a chance here.

The results:

Conclusion: The High-CPU instance wins again by a large margin. The High-CPU instance is statistically proven to be faster at the 99% confidence level. I thought that the Cluster would perform better in this test. I was not expecting the High-CPU instance to perform 69% faster.

Heavy Database usage:

This set of tests was performed against a page that has heavy database usage. This page accesses numerous models. The page is not cached and has several images, scripts, and partials. This should be a test of the instance’s ability to process Rails requests as well as handle the database load. With the additional memory and dedicated machine for the database, the Cluster might have a chance here.

The results:

Conclusion: The High-CPU instance wins again by a large margin. The High-CPU instance is statistically proven to be faster at the 99% confidence level. I thought that the Cluster would perform better in this test. I was not expecting the High-CPU instance to perform 120% faster.

Cached Page performance:

This set of tests was performed against a page that was cached. It has several images, scripts, and partials. This should be a test of the instance’s ability to server pure web server (nginx) requests. Since nginx uses such small amounts of memory and CPU, both configurations might be similar.

The results:

Conclusion: The High-CPU instance wins again by a very large margin. The High-CPU instance is statistically proven to be faster at the 99% confidence level. I thought that the Cluster would perform better in this test. I was not expecting the High-CPU instance to perform 277% faster.

UPDATE: Testing performance under load¹

Load Testing:

The advantage of having a separate database server should become more pronounced as the site becomes more heavily used. The tests so far were testing responsiveness not ability to handle load. In order to test load, I am using ab with 12 concurrent users like this: “ab -n 6186 -c 12 http://174.129.193.145/page/id” Each test was tailored to last about three minutes.

The results for the medium database usage page:

Conclusion: The High-CPU instance wins again by a very large margin. Notice in particular how the Cluster is slower overall and some of the requests queue up and take a very long time to serve. Only getting 80% of your requests in under 2 seconds is simply not acceptable in a production environment. On the other hand, the High-CPU instance is able to serve 100% of the requests in under 2 seconds.

I thought that the Cluster would perform better in this test. I was not expecting the High-CPU instance to perform 184% faster. Apparently the High-CPU machine is so much more powerful that even having the database on its own box isn’t enough for the small cluster to win.

Overall Conclusion:

The test results were very surprising. I expected the performance between the two configurations to be similar, particularly in database intensive tests.

Given the dramatically better performance of the High-CPU instance in every test situation, I can’t recommend using a cluster of two Small instances for a Rails application. Your money will be much better spent using a single High-CPU instance.

In future posts, I will explore the various deployment options. Next up is Apache+Passenger vs. Nginx+Passenger.