- The dependency of a specific versions of Qt has been causing frustrations to set it up properly both on engineers' local machines and on CI.
- The roadmap of capybara-webkit development is unclear.
- It's been hard to truly identify the root cause of "flickering" feature specs (i.e. tests that fail intermittently and are hard to reliably reproduce), while retrying tended to resolve it on CI.
- The entire RSpec tests took about 16 mins to complete on CI, with 6 parallelism. The slowness made it unrealistic to run the whole tests locally.
Headless Chrome has gained a lot of attention in the past few years and migrations done by companies such as GitLab and thoughtbot have proven it to be a promising alternative to capybara-webkit. In fact, it's been officially included in Rails 5.1 for system tests.
The goal of this project is to switch to Headless Chrome and maintain the same feature sets we have now. This includes:
- Making all existing specs pass
- Running in container environments and using Artsy Hokusai
- Bonus point to improve the stability of feature specs
- Bonus point to improve the speed of running the entire test suite
First, we replaced
Second, we registered both
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
We were on Rails v5.0.2 and Capybara v2.18.0 during the migration. We will be able to simplify the configuration by using the default
:selenium_chrome_headless drivers introduced in Capybara v3.11.1. In addition, Rails v5.1 introduced the new system tests, and it'll be even simpler by using the
Naively switching to Headless Chrome caused about 60 spec failures on my local machine. We simply went through them one by one and fixed them. A big part of failures was due to capybara-webkit's non-standard driver methods, such as setting cookies, inspecting console logs, etc., and we just had to migrate to Selenium WebDriver's equivalents.
However, we still observed flickering specs on CI, while the exact failures seemed to be different than previously observed with Capybara Webkit. We will have to investigate further for possible causes. Regarding speed, we didn't see significant improvement after switching to Headless Chrome, as mentioned in GitLab's and others' blog post, too.
The naive migration to Chrome (and removal of the Qt dependency) already improved the developer experience quite a lot (e.g. no more wrestling with installing Capybara Webkit and Qt 5.5 on every engineer's local machine and CI.) There are many next steps we can keep experimenting with and improving our tests, for example
- Updating Volt to Rails >= 5.1 and switching to system tests
- Investigating the causes of the flickering specs by looking into intermittent failures reported on CI
- Improving speed by using Docker multi-stage builds, caching, writing the right type and amount of tests, etc.
It's a long journey, and we were all excited about the migration and the new future. We'd love to hear your experience, too!