We've recently been quite excited about the Ruby CDP project over here at bitcrowd. It's a fleet of open source Ruby projects which utilize the Chrome Dev Tools Protocol (CDP) to instrument a Chrome/Chromium browser. We started using some of them as replacements for bulkier, outdated or just "not-so-nice" tools and wanted to give a quick summary of our first impressions.

What is CDP?

First up, what is this CDP thing actually? In brief, it's a low-level JSON-API running over a WebSocket-connection to control Chrome's browser engine. It was originally developed for Chrome's DevTools UI, but Google at some point stabilized and documented it, so that it can be used by external tools as well. It's an alternative to the old HTTP-based WebDriver standard.

Outlook: WebDriver BiDi

While CDP is tied to Chrome alone, there is also a new version of the WebDriver standard currently evolving. It's named WebDriver BiDi - as it's "bi-directional" - and essentially taking all the good parts from WebDriver and CDP and mixing them together. At the time of writing, it's still work in progress though and using it in Selenium requires dropping down to the raw driver.browser-level API.¹

Ruby CDP

The Ruby CDP project makes an effort to build higher level Ruby APIs on top of the CDP protocol. At the time of writing, they have three projects:

1. Ferrum - a high-level API to control Chrome in Ruby
2. Cuprite - a pure-Ruby driver for Capybara based on Ferrum
3. Vessel - a high-level web crawling framework also based on Ferrum

The idea is to have pure-Ruby tooling based on the Chrome DevTools protocol without requiring too many external dependencies from other ecosystems.

First impressions

We've been using FerrumPDF, a PDF library built on top of Ferrum to replace Grover in one of our projects. Grover's dependency on the Puppeteer NPM package was a bit of a maintenance burden and it's nicer to have less individual tools involved. Up until now, we're happy with the switch, but more on that in a different post…

Cuprite

We also switched the test suite of a large Rails app from using Selenium to Cuprite as a driver for our Capybara end-to-end feature tests.

The pitch is:

Cuprite is a pure Ruby driver (read as no Selenium/ WebDriver/ ChromeDriver dependency) for Capybara. It allows you to run Capybara tests on a headless Chrome or Chromium. Under the hood it uses Ferrum which is high-level API to the browser by CDP protocol. The design of the driver is as close to Poltergeist as possible though it's not a goal.

Before

The app is quite aged already and has a rather big and long running suite of feature tests. The tests used to run in a headless Chrome browser driven through Selenium. Under the hood, Selenium used Chromedriver, which is a standalone server that implements the WebDriver standard to then talk to a Chrome browser to perform actions in there. The setup still works well and we used it in a lot of projects over the years.

But for this particular app, the feature test suite was rather flaky. While being a classic server-side rendered Rails app, the app is quite JavaScript-heavy and contains sections which are pure ReactJS with state management on the client. With all of this in place, the feature tests of this app were unreliable: Elements may have been re-rendered and therefore become stale while Capybara wanted to interact with them, some interactions involved loading screens and animations and were taking longer than what Capybara expected, etc.

One error that appeared often was Selenium having problems with elements being re-rendered:

Selenium::WebDriver::Error::UnknownError:
  unknown error: unhandled inspector error: {"code":-32000,"message":"Node with given id does not belong to the document"}
    (Session info: chrome=144.0.7559.96)

In order to mitigate all the falkiness, there were things in place like this:

config.around :each, :js do |ex|
 ex.run_with_retry retry: 3
end

config.default_max_wait_time = 10

Capybara was set to wait up until 10 seconds - instead of the default two seconds - for asynchronous processes to finish. Test tagged with "js" would be retried up to three times before they would finally be reported as failing. Plus, on CI the failed tests were also retried once more before they would be marked as failing:

bundle exec rspec spec/features --exclude-pattern || bundle exec rspec spec/features --only-failures

With all of this in place, the feedback of CI was super slow. Failing tests would be retried up to six times until CI was finally done and red.

Running the test suite locally was equally painful. Skipping the retry mechanisms in favor of faster feedback meant dealing with falkiness: for any failing test, developers first needed to find out whether it's a failure related to their change or "just" a flaky test. This can be quite a time sink…

Switch

People on the internet reported their feature tests being more reliable and less flaky when using Cuprite as a driver for Capybara compared to Selenium. So we were curious if this could improve our situation and went on a journey (together with Claude…) to switch the drivers and compare the speed and reliability to the Selenium setup.

We observed, that the tests did not run faster than with Selenium. But there was a huge difference when it came to reliability: we were able to remove all mechanisms in place for mitigating the falkiness with Selenium, from the retry logic up to the high wait time for Capybara.

The tests ran super stable, both locally and on CI 🎉

Migrating to Cuprite

Migrating our test suite required some small adjustments to the tests here and there where Selenium and Cuprite behave differently.

External URLs

This may be quite specific, but Cuprite won't load external, non-existing URLs. Instead, it will show a Chrome error page. So one cannot match on the current path for those URLs like it's possible with Selenium.

- expect(page).to have_current_path("https://localhost/non-existing-path", url: true)
+ expect_external_url("https://localhost/non-existing-path")

What we ended up doing instead, was checking the details of the resulting error page using a helper:

def expect_external_url(url)
  expect(page.current_url).to eq('chrome-error://chromewebdata/')
  expect(page).to have_css('h1', text: 'This site can’t be reached')
  expect(page).to have_button('Reload') do |button|
  expect(button['data-url']).to eq(url)
end

Form values

For some form fields, the values extracted via Capybara where yielding a different type. A non-checked checkbox for instance:

- expect(find('.my-selector')[:checked]).to be_nil
+ expect(find('.my-selector')[:checked]).to be(false)

Matching across elements

Again a quite specific issue: when matching the text across multiple elements, Cuprite is separating the elements via tabs instead of whitespace:

- expect(page).to have_content("foo bar")
+ expect(page).to have_content("foo\tbar")

Conclusion

Migrating an existing codebase from Selenium to Cuprite did not require much adaption of the tests itself, but it helped to stabilize the overall suite a lot. Alone for that it's definitely worth considering!

But if you have time, maybe it's also fine to wait for full BiDi support in Selenium.

References

Footnotes

1. https://www.selenium.dev/documentation/webdriver/bidi/ ↩