A technical post: Ruby interpreter comparison—MRI 1.9.1 vs. REE 1.8.7

I ran a small, controlled “real-world” experiment comparing these two interpreters.  The results show identical time performance.

My interest is Rails application speed

This site, OregonLaws.org, is a Rails 2.3.4 application.  (These blog pages are the sole exception, running on WordPress.)  I manage the architecture and deployment so that server response time is 200 ms or less — a goal I borrowed from Google.  I’ve been interested in moving to Ruby 1.9.1 now that all of the packages I use are compatible with it.  There are still some issues running Rails on 1.9.1 (such as the string encoding problems) and I wanted to see if there’s really anything to be gained by spending the time on it vs. using Ruby Enterprise Edition.

Results

Here are the data from my trials (lower is better):

REE vs MRI experimental data

Methods

I ran the OregonLaws.org Rails 2.3.4 app in development mode with WEBrick in an Ubuntu 9.10 VM on my Mac laptop.  I have the latest of each interpreter type installed:  (1) The 1.9.1.243-2 MRI (standard) Ruby interpreter, installed via the Ubuntu package system.  (2) The 1.8.7-2009.10 REE Ruby interpreter installed from the source code. I switch between these by changing my PATH.  The entire environments are identical except for the Ruby interpreters.

After starting the server (“./script/server”) I run one trial of each page and discard the results, in order to prime any caches.  I then just clicked through the pages—a mix of static and dynamic content—typing in the “Completed in” time from the development log.

I calculated cumulative average times of 73 ms for MRI and 75 ms for REE, +/- 0.5 ms.  I’m reporting these as whole numbers because the decimals must be dropped according to the rules of significant digits.  I believe that the 2 ms difference between them is too small to be considered a real finding.  (It’s been a while since I’ve calculated statistical significance; but my gut tells me that this difference is just random noise/chance.)

Interpretation

These results show that these two interpreters have essentially identical execution times in a real-world scenario.  Based on this, it makes sense for me to stay with REE, as it is 100% compatible with my Rails application.

Caveats

  1. Small sample size
  2. App is running development mode

The results might look a little different if done in a production application, and over a longer period of time with more pages requested.  For my purposes, though — to get a fairly good idea about how these two interpreters compare — I believe the results are very helpful.

5 thoughts on “A technical post: Ruby interpreter comparison—MRI 1.9.1 vs. REE 1.8.7

  1. The other advantage of staying with REE is unknowns. Ruby 1.8 is more tested and used at present, giving you more of a community to depend upon when that odd error does crop up in a production environment. Like new OSes, it’s often good to wait until the initial unexpected items are found and cleared/solved before putting your production website to the test.

  2. Running in production mode is going to make a HUGE difference in terms of performance overall. Development reloads classes on each request which significantly slows down the process (but makes it much more convenient to develop in).

    In addition while WEBrick is the default server that comes with Rails it is well known to be absurdly slow. I’d highly recommend using a better server like Mongrel, Thin, or FastCGI. With all three you should see a lot more performance.

  3. @Jonathan,

    Sure – this was just for *comparative* testing.

    On the actual website, I use production mode with Apache + mod_rails. Now, one issue may be that the VMs in my tests were doing different tasks then they would in a production system. E.g., more time spent loading classes vs. running the application code.

  4. Yes, I had the same reaction as Jonathan: Your test scenario is going to be dominated by the class reloading workload, which may not be relevant to the “real world”. In fact, you might find that your response time drops from 73 vs. 75 to 8 vs. 10. In other words, by removing the noise, you might make your results statistically significant.

    Also, one big advantage of REE is memory consumption, which can affect response time when work is I/O bound because you can have more workers.

  5. I’m sorry to tell you, but 1.9.x is not MRI… Matz’s Ruby Interpreter only implements the Ruby 1.8 language specs. You probably meant YARV (1.9).

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s