Engine Yard Solo vs. EC2onRails: Pay Engine Yard extra or do it yourself?

Engine Yard is well known in the Rails community for their expertise in deploying and scaling Rails applications. When Engine Yard recently introduced Engine Yard Solo, I was very interested since I have been using the open source project EC2onRails. In particular, I wondered if you should pay Engine Yard for the Solo product when you can get EC2 running on your own for free. (excluding Amazon EC2 charges.)

After hours of testing and analysis, I have decided that the answer depends upon your skill level. Read on for the details.

The contestants: Engine Yard Solo vs. EC2onRails

Engine Yard Solo

First up, we have Engine Yard’s new Solo product. According to Engine Yard:

“Solo is an inexpensive, web-based platform for on-demand management of your Ruby on Rails web application on Amazon’s cloud computing infrastructure. Manage your web application from development to deployment. You get the Engine Yard Ruby on Rails hosting and deployment expertise wrapped in an easy-to-use interface.”

The idea behind Solo appears to be that many people would like the flexibility of using Amazon’s EC2 environment but lack the technical expertise to configure the server themselves. Solo enables someone with little or no technical knowledge to set up and manage an EC2 server. To get an idea of how easy the system is to use, Engine Yard has a number of screencasts  online.

The Good: I created a Solo account and used it for several hours over the course of a few days in order to write this article. Happily, I can report that Solo performs as advertised. You will be able to very quickly get a Rails project up and running with Solo (assuming you have a fair understanding of deploying Rails applications. See below for a discussion). The Engine Yard team has successfully removed some of the complexity of running your own Rails server. They also provide a well optimized and secure server.

The not so Good: Given that Engine Yard appears to be targeting Solo at less knowledgeable users, I have a few issues with Engine Yard’s Solo.

1. Cost: In order to use Solo, you pay an additional $0.06 per hour for a small instance¹ (plus a premium over Amazon’s prices for the storage and bandwidth) ². This works out to be a minimum of $43.20 per small instance per month (plus storage and bandwidth charges).³
2. Lack of instructions: The instructions for new users are not sufficient.⁴ After you log in to Solo, you are greeted by a very clean layout, with no instructions or tips on what to do next. Should you “Create New Environment” or “Create New Instance” or one of the other choices? An inexperienced user may very well be lost at this stage. One of the things that 37Signals does very well is provide tips in order to get a new user started.
3. Deployment process: The steps that you must take in order to deploy a new application (like adding gems and Unix packages) are spread across numerous places in the system. These items should be placed into a more logical sequence or in some sort of wizard.
4. Log files: The process of viewing deploy log files is less than optimal. My application crashed several times during deployment until I had all of the configuration options set properly.⁵  In order to view the deploy log files, you must view them in the browser. Unfortunately, the deploy log files were very, very large which causes trouble with your browser. This caused trouble even on my machine, which is an 8 core machine with 16GB of ram. The log files should be downloadable.
5. Login process: This is a minor issue but there is no obvious login link. If you go to the Solo page, you will see a sign up link, but no link for returning users.⁶

Overall, these are minor issues that can easily be addressed in future versions of Solo.

The take so far: The current version of Solo is an excellent start. A few wizards and the liberal use of tips and suggestions could make this into a killer app.

EC2 on Rails

Next, we have EC2onRails. EC2onRails is an open source project led by Paul Dowman. According to the EC2 on Rails website:

“EC2 on Rails is an Ubuntu Linux server image for Amazon’s EC2 hosting service that’s ready to run a standard Ruby on Rails application with little or no customization. It’s a Ruby on Rails virtual appliance.”

Paul Dowman and the other EC2onRails committers (like Adam Greene (aka skippy)) deserve recognition for the massive amount of work they have done.⁷

The Good: The best part of the EC2onRails system is that it is open source. There is no charge for using the code and you have the ability to tailor it to your specific application. The EC2onRails code works well for creating an Apache + Mongrel cluster server. The EC2onRails team has done a good job configuring the various parts of the server. You can easily create an all-in-one server (with a web server, application server, and database) or split your application server (Rails) and the database onto separate boxes. Once you have a running server, it is monitored and maintained with tasks like database backups happening automatically.

The not so Good: Keep in mind that we are evaluating the experimental branch of EC2onRails running Nginx+Passenger, not the production branch of 0.9.9.1. So, some of my comments here will be addressed by the time this code reaches production quality. Given that this is an open source project, its weaknesses are like many other open source projects.

1. Support: The EC2onRails project is open source and lacks official support. It is intended for more advanced users. If you have an issue, your best bet is to reach out to the very helpful Google group.
2. Documentation: The documentation is good for deploying an Apache+Mongrel cluster configuration but is under revision for the Nginx+Passenger configuration. There are also items that are not covered in the documentation. For example, there are many configuration options and steps in the process that may require you to search the Google group or read the source code.
3. Ease of use: The system is not as easy to use as Solo. There are a number of steps that must be done manually in order to get a running server (e.g. deploying SSH keys). To be fair, most of these steps only need to be performed one time.
4. Nginx+Passenger is alpha: Support for Nginx+Passenger is currently experimental. Because of this, there is no AMI image available for you to use. This means that you must first create the AMI (a time consuming step) then configure your EC2onRails server.
5. The future? Now that Ubuntu is offering EC2 instances, the future of an important piece of the EC2onRails infrastructure is unclear. Eric Hammond’s ec2ubuntu-build-ami script is used when building a new AMI and it may not be supported by him in the future.⁸ This should not affect EC2onRails end users, but may require changes to the core EC2onRails code.

Overall, the EC2onRails project is a good example of a successful open source project. It has its rough spots but there are a good group of people using it and it is free :). Most of the issues listed above will be addressed by the time the code becomes official.

The take so far: The current version of EC2onRails offers an inexpensive and relatively easy way to get a Rails application running on EC2 using Apache+Mongrel cluster. If you want to deploy a Nginx+Passenger server, you will have a fair bit of work because the code is still experimental.

Here is my analysis of EC2onRails versus Engine Yard Solo so far:

So which one you use depends so far on your expertise. If you have good technical skills, use EC2onRails and save the money. If you are less technically inclined, use Solo. However, we have not yet examined PERFORMANCE! In short, is the Engine Yard Solo server faster than an EC2onRails server?

Hypothesis:

The performance of an Engine Yard Solo server will be the same as an EC2onRails server when configured using Nginx and Passenger.

Setup:

For these tests, I used 4 EC2 instances configured as shown here:

Test server configuration

Great care is required when doing performance tests to ensure your results are legitimate. There are many factors that affect server performance and not accounting for any of these factors can make your results invalid. For example, in many web server performance tests that I read, the authors simply run Apache Bench (ab) against a couple of configurations then proclaim a winner. A couple of potential problems with this approach are 1) the results are not statistically testable and 2) something may have changed on the server between test runs.

In order to get valid results, we need to take multiple samples of the performance. With those samples we need to get both the mean and the standard deviation. When we have this information, we can perform a two-sample t-test to statistically prove that one configuration is better. Fortunately, the httperf tool provides the information that we need.

For these tests, each test server instance was configured exactly the same. Each instance was created using the EC2onRails code from my fork on Github. They were EC2 Small Instances and had minimal services running. The EC2onRails server was also created using my EC2onRails branch. The Engine Yard Solo server was created using the Solo web-based GUI. Both servers used the exact same Rails code⁹ and had the exact same data in the database. All tests were performed at the same time on both configurations. Each test was then repeated on the other configuration to ensure the testing server was not biasing the results.

Details for the EC2onRails server:

• Ubuntu 8.04
• Rails 2.3.2
• Nginx 0.6.36
• Passenger 2.2.2
• MySQL
• Memcached

Testing process: (all of these steps are performed on each testing server at the same time)

• Warm up each server using ab:
  ab -n 100 -c 1 ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com/
• Run this a second time to ensure the server is ready.
• Check that all requests were served successfully.
• Using an automated script:
  • Send 5 requests using httperf to estimate the time required to get 45 samples.
  • Get at least 45 samples using httperf.
• Run the test again using the other testing server.
• Verify that the two samples from different testing servers are not statistically different from one another.

Results:

The test results for the EC2onRails server are not statistically different than the test results in my last post. This gives me confidence in the validity of these results.

No ActiveRecord:

The first set of tests was performed against a page that doesn’t access the database. This page is not cached and has numerous images, scripts, and partials. This should be a test of the instance’s ability to process Rails requests. If the Solo instance is better optimized it should show up in this test.

The results:

Conclusion: The Solo instance is statistically proven to be faster at the 99% confidence level. The Solo instance is about 16% faster than the EC2onRails server at serving standard Rails requests.

Moderate Database usage:

The next set of tests was performed against a page that has moderate database usage. This page accesses eight different models. The page is not cached and has several images, scripts, and partials. This should be a test of the instance’s ability to process Rails requests as well as handle the database load.

The results:

Conclusion: The Solo instance wins again by a fair margin. The Solo instance is statistically proven to be faster at the 99% confidence level. I was not expecting the Solo instance to perform 20% faster.

Heavy Database usage:

This set of tests was performed against a page that has heavy database usage. This page accesses numerous models. The page is not cached and has several images, scripts, and partials. This should be a test of the instance’s ability to process Rails requests as well as handle the database load. With the additional optimization, the Solo instance will probably win by a large margin.

The results:

Conclusion: The Solo instance wins again, although by a smaller margin. The Solo instance is statistically proven to be faster at the 99% confidence level. I thought that the Solo instance would perform better in this test. I was not expecting the Solo instance to only perform 10% faster.

Overall Conclusion:

The test results were surprising. The Engine Yard Solo server is 10%-20% faster than an EC2onRails instance. Hopefully the EC2onRails team can investigate performance and close this performance gap.

The Winner!

Picking a winner in this case is difficult. My recommendation will depend upon your level of expertise. Engine Yard Solo does offer a fast and convenient way to create EC2 instances. Solo is also easier to use than EC2onRails and requires less technical knowledge to operate.

If you are not comfortable building your own server, I suggest that you seriously consider Engine Yard’s Solo. Solo is a reasonably priced way to get a professionally configured Rails server in the EC2 environment. You should also consider some of the other solutions like RightScale (which is much more expensive than Solo).

If you are comfortable setting up your own server, I recommend EC2onRails. It is a great project that is constantly evolving. Personally, I have difficulty justifying the cost of Solo. It seems to me that you are paying $43.20 per month for Engine Yard to configure a server for you. I would find a one time setup charge much more appealing than paying over and over. The EC2onRails project allows you to set up EC2 instances for free that performs almost as well as the Solo instance.

In short, both choices are excellent. The best choice for you, will be up to you

What’s next?

In future posts, I will explore the various deployment options. Next up is Apache+Passenger vs. Nginx+Passenger.

Footnotes: