We now have over 4700 RSpec examples in one of our projects. They are stable, using the techniques described in an earlier post and organized in suites. But they now take almost 3 hours to run, which is clearly unacceptable.
To solve this, we have parallelized parts of the process with existing tools, and can turn a build around in just under an hour. This post will dive into our Jenkins build flow setup.
To keep things simple, we're going to only build the
master branch. When a change is committed on
master we're going to push
master to a
master-ci branch and trigger a distributed build on
master-ci. Once all the parts have finished, we'll complete the build by pushing
master-succeeded and notify the dev team of success or failure.
Here's a diagram of what's going on.
Create the following Jenkins jobs.
A free-style job that connects to the SCM, in our case Git.
- Set SCM repository URL to your Git repo, eg.
- Change the default branch specifier from
master. We'll be pushing a
master-cibranch, which could, in turn, cause more builds if you don't do this.
- Add a post-build action to build another project. Trigger the
masterproject if the build succeeds.
This is a build-flow job. We'll describe the individual tasks that the flow invokes further. The flow DSL looks as follows.
1 2 3 4 5 6 7
This is a good place to add an e-mail notification post-build action for every unstable build.
A free-style job that creates the
master-ci branch from master. It needs to be connected to your SCM and executes the following shell script.
1 2 3
Note that we cannot combine this task with
master-prequel, because we have to make sure the branch creation runs once under the entire flow, while
master-prequel can be run multiple times, once per check-in. Otherwise the
master-ci branch could get updated before a
master-ci-task runs from a previous flow execution.
A parameterized build that accepts a
tasks parameter that the flow will pass in.
Change the default branch specifier to
master-ci and execute the following shell script.
1 2 3
This example runs
rake $tasks, which we define to be various test suites in our flow DSL. Our test suite setup is described in this post. Your mileage may vary.
This is an optional step. We use this free-style job to tag
master-succeeded with the following shell script.
1 2 3
Our deployment to production will pickup the
master-succeeded branch when it's time.
I see a few possible improvements here that might require a bit of work.
- The ability to split an RSpec suite up across an arbitrary number N sub-jobs and M executors would create an optimal parallel split based on the resources available.
- Passing the value of
GIT_COMMITacross these jobs would enable building any branch and eliminate the need for
- Build flow could support SCM polling the same way as free-style jobs, avoiding the need for
master-prequel. We weren't able to get a stable notification of changes from Github with the Jenkins Github plugin.
Please suggest further improvements in the comments below!
(Update: See Splitting up a large test suite for a modified approach that splits work approximately evenly among an arbitrary number of sub-jobs.)