Month: October 2015

HOW DO I BENCHMARK RUBY CODE?

HOW DO I BENCHMARK RUBY CODE?

So you’ve got some Ruby code and you want to make it faster. Maybe, you’ve already got a new implementation in mind, or maybe you are still cooking that up. But how do you make certain that your new implementation is faster?

Getting started

The Benchmark module is in the standard library, so you do not need to install any gems to get it.The Benchmark module provides methods to measure and report the time used to execute Ruby code. The Benchmark module offers several different ways for you to benchmark your code. I’ll take you through the different options and their use cases. The simplest way to measure your Ruby code is with Benchmark.measure. It returns the time used to execute the given block.

Screen Shot 2015-10-21 at 6.19.34 AM

This will return something that looks like this:

Screen Shot 2015-10-21 at 6.20.24 AM

With no context, these might look like magic numbers. Here’s what they mean:

                         Screen Shot 2015-10-21 at 5.46.11 AMScreen Shot 2015-10-21 at 5.41.35 AM

This report shows the user CPU time, system CPU time, the sum of the user and system CPU times, and the elapsed real time. The unit of time is seconds.

Generally, the number farthest to the right is the most important one. It tells how long it actually took to perform the operation. If you are curious about why the clock time is so high, the other numbers can help you drill down to see if you are spending time in system functions or your own code.

Now that you know what those magic numbers mean, we can move on to the core Benchmark API. The truth is that I rarely use the measure method on its own. It only prints the benchmark for a single block of code. The most common way to use Benchmark is to compare the execution time of different approaches to the same problem. Benchmark has some built-in methods for this exact purpose.

Benchmark #bm

This method lets you define several blocks of code to benchmark, then prints the results side-by-side in the same format you saw earlier.

Screen Shot 2015-10-20 at 12.58.20 PM

Screen Shot 2015-10-20 at 1.02.01 PM

You call the Benchmark#bm method passing a block. The block variable is a special object provided by Benchmark. It gives you a report method that you call with the block of code that you want to measure. Benchmark then runs both blocks of code and prints their execution times side-by-side.

Often, when doing benchmarks that test code that executes very quickly, you need to do many iterations to see a meaningful number. In this case, I did 100,000 iterations of each variant just to get the execution time up to half a second so I could grasp the difference.

Labels

In this benchmark, I buried some comments in the source that said what each block of code was doing. That is not so helpful when looking at the results! Benchmark allows you to pass in a label to the report method that will be printed along with the results.I have now removed the comments describing the blocks and pass them in to the report method as an argument. Now the output describes itself. Make sure you pass a value thats greater than the length of your longest label. Thats the happy path.

Screen Shot 2015-10-20 at 11.18.39 PM

Screen Shot 2015-10-20 at 11.27.30 PM

Benchmark#bmbm

The Benchmark#bm you just saw is really the core of Benchmark, but there is one more method I should mention: Benchmark#bmbm. Thats right its the same method name, repeated twice. Sometimes, with a benchmark that creates a lot of objects, the results start to get skewed because of interactions with Ruby’s memory allocation or garbage collection routines. When creating a lot of objects, one block may need to run garbage collector, while the other does not; or just one block may get stuck with the cost of allocating more memory for Ruby to use. In this case, the benchmark can produce unbalanced results. This is when you want to use Benchmark#bmbm.The method name is suitable because it actually benchmarks your blocks of code twice. First, it runs the code as a rehearsal to force any initialization that needs to happen, then it forces the GC(Garbage Collection) to run, then it runs the benchmark again for real. This ensures that the system is fully initialized and the benchmark is fair.

This last example benchmark allocates a lot of objects. When this runs at the rehearsal stage, Ruby has to allocate more memory to make room for all the objects. Then when the ‘real benchmark happens, the memory is already available and just the actual implementation is tested.

Screen Shot 2015-10-20 at 11.33.10 PM

Screen Shot 2015-10-20 at 11.34.19 PM

Notice the discrepancy between the rehearsal and the benchmark! Thanks bmbm!

Conclusion

When you want to try your hand at speeding up some of your Ruby code, make sure that you measure, measure, measure to prove that your new implementation is faster than the old one. This great little benchmarking library ships with Ruby right in the standard library, so theres no excuses!

Advertisements