Having over three thousand RSpec tests in a single project has become difficult to manage. We chose to organize these into suites, somewhat mimicking our directory structure. And while we succeeded at making our Capybara integration tests more reliable (see Reliably Testing Asynchronous UI with RSpec and Capybara), they continue relying on finicky timeouts. To avoid too many false positives we've put together a system to retry failed tests. We know that a spec that fails twice in a row is definitely not a fluke!
Create a new Rake file in lib/tasks/test_suites.rake
and declare an array of test suites.
1 2 3 4 5 |
|
RSpec::Core
contains a module called RakeTask
that will programmatically create Rake tasks for you.
1 2 3 4 5 6 7 8 9 10 11 |
|
Run rake -T
to ensure that the suites have been generated. To execute a suite, run rake test:suite:models:run
. Having a test suite will help you separate spec failures and enables other organizations than by directory, potentially allowing you to tag tests across multiple suites.
1 2 3 |
|
Retrying failed specs has been a long requested feature in RSpec (see #456). A viable approach has been finally implemented by Matt Mitchell in #596. There're a few issues with that pull request, but two pieces have already been merged that make retrying specs feasible outside of RSpec.
- #610:
A fix for incorrect parsing input files specified via
-O
. - #614:
A fix for making the
-e
option cumulative, so that one can pass multiple example names to run.
Both will appear in the 2.11.0 version of RSpec, in the meantime you have to point your rspec-core
dependency to the latest version on Github.
1
|
|
Don't forget to run bundle update rspec-core
.
The strategy to retry failed specs is to output a file that contains a list of failed ones and to feed that file back to RSpec. The former can be accomplished with a custom logger. Create spec/support/formatters/failures_formatter.rb
.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
|
In order to use the formatter, we must tell RSpec to require
it with --require
and to use it with --format
. We don't want to lose our settings in .rspec
either - all these options can be combined in the Rake task.
1 2 3 4 5 6 7 8 9 10 |
|
Once a file is generated, we can feed it back to RSpec in another task, called suite:suite[:id]:retry
.
1 2 3 4 5 6 7 8 9 |
|
Finally, lets combine the two tasks and invoke retry
when the run
task fails.
1 2 3 4 5 6 7 8 9 |
|
A complete version of our test_suites.rake
, including a spec:suite:all
task that executes all specs can be found in this gist. Our Jenkins CI runs rake spec:suite:all
, with a much improved weather report since we started using this system.
Comments