May 2009

At Citrusbyte we’ve launched production websites with the major three Ruby testing frameworks: Test::Unit, RSpec, and Shoulda. In our experience using these libraries, we never felt like we found our ideal framework. As we wanted to move the company toward a single testing platform, we looked at what was out there and realized none of the current offerings was a good fit for our needs.

Test::Unit

Test::Unit is old and well tested, comes with the standard library and is easy to pick. The downside is that after using Shoulda and RSpec we fell in love with nested contexts - without them the code gets too large and not DRY. It leads to pulling out lots of helpers to reuse in your tests, which in turn takes your context setup away from your tests (a similar problem to what occurs with fixtures). As a test suite grows larger, the helper suite grows in a different place and it leads to hard-to-maintain tests.

RSpec

RSpec adds a magic syntax for no good reason. For example, compare these two assertions:

With RSpec:

@user.should have(4).friends

With the rest, using plain old Ruby:

assert @user.friends.size == 4

Or using a helper assertion:

assert_equal 4, @user.friends.size

While the magic syntax is helpful for getting into the BDD mindset, once you get it you can do BDD with any similar tool.

Another point against RSpec is the fact that its codebase is big and somehow unstable. Even if the developers are fast at patching bugs, the testing framework is a place where you really don’t want to deal with bugs. A larger codebase also means that you need more time to understand its internals in case you need to patch it yourself. Finally, RSpec is the slowest of the lot, and with large test suites the difference can amount to many seconds.

Shoulda

Shoulda provides what we want, but also provides features we don’t want to use. We don’t want macros, before_* statements, Rails or ActiveRecord integration. We want our testing framework to test our code, not code from other libraries.

In the end, Shoulda and RSpec give us roughly the same power (in terms of testing, ignoring that spec/mocks is included in RSpec), and Shoulda provides it with far less overhead (code that might need debugging but is difficult to understand). Shoulda forgoes the `foo.should be_valid?` syntax in order to avoid a couple of code smells. See the Shoulda announcement for more information about these code smells and the differences with RSpec.

We found that we actually really like RSpec and Shoulda because of BDD, but not because of their particular functionality or DSLs.

Our Ideal Framework

Reflecting on all these facts, we agreed that our ideal framework should have nested contexts to suit our BDD needs, be as small as possible, stable, and have no unnecessary magic.

As we looked at our needs for a framework, we realized that while RSpec and Shoulda are powerful, we could just add nested contexts to Test::Unit and that would fulfill our criteria. We did that with less than 100 lines of very simple code and called it “Contest”.

Here’s a quick sample of how it looks like:

require "rubygems"
require "contest"

class Array
  def rotate_left(n = 1)
    n.times { push(shift) }
    self
  end

  def rotate_right(n = 1)
    n.times { unshift(pop) }
    self
  end
end

class TestArray < Test::Unit::TestCase
  context "Array" do
    should "rotate elements to the left when sent rotate_left" do
      assert_equal [2, 3, 4, 5, 1], [1, 2, 3, 4, 5].rotate_left
    end

    should "rotate elements to the right when sent rotate_right" do
      assert_equal [5, 1, 2, 3, 4], [1, 2, 3, 4, 5].rotate_right
    end

    should "rotate elements to the left 2 places when sent rotate_left(2)" do
      assert_equal [3, 4, 5, 1, 2], [1, 2, 3, 4, 5].rotate_left(2)
    end

    should "rotate elements to the right 2 places when sent rotate_right(2)" do
      assert_equal [4, 5, 1, 2, 3], [1, 2, 3, 4, 5].rotate_right(2)
    end
  end
end

Contest is available in our labs and at github. Feel free to use it!

{ 9 comments }