Having over three thousand RSpec tests in a single project has become difficult to manage. We chose to organize these into suites, somewhat mimicking our directory structure. And while we succeeded at making our Capybara integration tests more reliable (see Reliably Testing Asynchronous UI with RSpec and Capybara), they continue relying on finicky timeouts. To avoid too many false positives we've put together a system to retry failed tests. We know that a spec that fails twice in a row is definitely not a fluke!
Create a new Rake file in
lib/tasks/test_suites.rake and declare an array of test suites.
1 2 3 4 5
RSpec::Core contains a module called
RakeTask that will programmatically create Rake tasks for you.
1 2 3 4 5 6 7 8 9 10 11
rake -T to ensure that the suites have been generated. To execute a suite, run
rake test:suite:models:run. Having a test suite will help you separate spec failures and enables other organizations than by directory, potentially allowing you to tag tests across multiple suites.
1 2 3
Retrying failed specs has been a long requested feature in RSpec (see #456). A viable approach has been finally implemented by Matt Mitchell in #596. There're a few issues with that pull request, but two pieces have already been merged that make retrying specs feasible outside of RSpec.
A fix for incorrect parsing input files specified via
A fix for making the
-eoption cumulative, so that one can pass multiple example names to run.
Both will appear in the 2.11.0 version of RSpec, in the meantime you have to point your
rspec-core dependency to the latest version on Github.
Don't forget to run
bundle update rspec-core.
The strategy to retry failed specs is to output a file that contains a list of failed ones and to feed that file back to RSpec. The former can be accomplished with a custom logger. Create
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
In order to use the formatter, we must tell RSpec to
require it with
--require and to use it with
--format. We don't want to lose our settings in
.rspec either - all these options can be combined in the Rake task.
1 2 3 4 5 6 7 8 9 10
Once a file is generated, we can feed it back to RSpec in another task, called
1 2 3 4 5 6 7 8 9
Finally, lets combine the two tasks and invoke
retry when the
run task fails.
1 2 3 4 5 6 7 8 9
A complete version of our
test_suites.rake, including a
spec:suite:all task that executes all specs can be found in this gist. Our Jenkins CI runs
rake spec:suite:all, with a much improved weather report since we started using this system.