Measuring Test Code Coverage For Non-Ruby Runners

When we think about Ruby code coverage, our go-to gem for this is SimpleCov opens a new window , which works great when the test suite uses Minitest opens a new window , RSpec opens a new window , Cucumber opens a new window , Capybara opens a new window , and all these tools that are integrated with Ruby and Rails. But many applications also use other tools like Playwright opens a new window or Cypress opens a new window to run e2e tests, and we can’t use SimpleCov the same way.

Most of the time, what we have seen is that the Ruby code executed when running these tools ends up left behind and not being counted for the total code coverage, even though we know the code is actually being tested.

Sample Application

To make it easier to try this, we created a sample application opens a new window that uses the cypress-on-rails gem opens a new window along with Minitest, and includes the instructions and custom Rake tasks to get the final code coverage when running both test suites.

Note that this could be done with any other tool like Playwright or Protractor opens a new window (deprecated) since the actual integration is independent of the tool as long as it runs the Rails server.

Approach Summary

The main problem that we are facing when using these external tools is that the Rails server is controlled by this runner and not started by a Ruby runner, so we can’t easily control the lifecycle of the server to measure and extract the coverage information.

We can see how the cypress-on-rails gem already overcomes some of these limitations with custom helpers to execute fixtures, factories, and db cleanup in their generated sample tests opens a new window .

We have a few moving pieces here, so this is a quick summary of the process:

• when we run rails test we get the code coverage of the Minitest test suite
• then we’ll run Coverband’s standalone server on the side, so it collects stats from the Rails servers started by Cypress
• once the e2e tests are done (running rails cypress:run), we can use a custom Rake task to extract the code coverage from Coverband into a JSON file
• finally, we can merge the code coverage of the 2 tools into a single final report

The Details

Minitest Code Coverage

This step is the standard SimpleCov setup: we only need to add the SimpleCov gem in the :test group opens a new window , and enable SimpleCov at the beginning of our test_helper.rb file opens a new window .

After this we can do a quick test, run rails test and see the generated report:

We can see the total coverage is around 35%, and we can see the details of the UsersController coverage that we’ll use for this example too:

Note the total coverage of this file is around 39%, with many lines not covered (lines 15, 24, and 26). This is expected, as we are only testing the index action opens a new window in our Minitest suite.

But we know we have more tests for this file; the Cypress test suite is testing an invalid form submission of the new user form opens a new window .

Coverband Standalone Server

As mentioned above, we can’t simply add SimpleCov to the Cypress tests, so we are going to use Coverband opens a new window (which uses SimpleCov internally) to capture the code that is being executed by the tests, similar to what we would do if running Coverband in production.

Coverband provides a standalone server that we can use so we can control the lifecycle of the code coverage tracking independently of the lifecycle of the Cypress runner.

First we have to add the coverband gem (note that we are adding it only in the test group, we are not using it to track production coverage!). Then, we can execute the server with RAILS_ENV=test rails coverband:coverage_server and leave it running on the side.

Note that, now that Coverband is added, we get an error message at the end of the Minitest run. Ideally, we could use a different Rails environment for the e2e tests so the gem wouldn’t be loaded, but the cypress-on-rails gem hardcodes the test Rails environment.

E, [2026-01-02T16:29:34.221521 #322239] ERROR -- : coverage failed to store
E, [2026-01-02T16:29:34.221579 #322239] ERROR -- : Coverband Error: #<RuntimeError: coverage measurement is not enabled> coverage measurement is not enabled

Running Cypress Tests

Now it’s time to run the e2e tests. In this example we run rails cypress:run, but it would be the same for any other tool, as long as we set the same RAILS_ENV variable as the Coverband server (so the gem is required).

When the tests finish, we won’t see any information about the code coverage, but we can open the Coverband server in the browser to check how the percentage is being tracked in http://localhost:9022.

It’s important to note that Coverband will calculate all the coverage of any code that gets executed (even for Ruby files we don’t really care about when looking at the final code coverage value) and we get some information like coverage of config files or test files that will affect the coverage percentage. With SimpleCov, we use the 'rails' profile opens a new window to ignore those extra files, but we don’t need to worry about it at this point for Coverband. We’ll clean that up at the end.

Extracting the Code Coverage from Coverband

Coverband provides a coverband:coverage_html task that uses SimpleCov to process the internal coverage data. There’s also a coverband:coverage_json but these tasks don’t generate the raw data we need to merge it with the Minitest resultset. To solve this, we created a custom Rake task that will extract the raw data and format the JSON file to have a structure that we can then merge.

# https://github.com/fastruby/coverage-with-cypress-sample-app/blob/main/lib/tasks/json_coverage.rake#L2

desc "JSON formatted report of Coverband code coverage"
task :simplecov_json_report do
  require "coverband"
  require "coverband/utils/result"
  require "coverband/utils/file_list"
  require "coverband/utils/source_file"
  require "coverband/utils/lines_classifier"
  require "coverband/utils/results"

  require "simplecov"
  require "simplecov_json_formatter"

  `mkdir -p #{SimpleCov.coverage_path}`

  # SimpleCov hardcodes this constant as `coverage.json`, we are renaming this here to make it more
  # clear, but this step is not necessary
  SimpleCovJSONFormatter::ResultExporter.send(:remove_const, "FILENAME")
  SimpleCovJSONFormatter::ResultExporter::FILENAME = "cypress_coverage.json"

  coverband_reports = Coverband::Reporters::Base.report(Coverband.configuration.store)
  Coverband::Reporters::Base.fix_reports(coverband_reports)
  result = Coverband::Utils::Results.new(coverband_reports)
  SimpleCov::Formatter::JSONFormatter.new.format(result)

  # fix json structure so it can be merged with coverage:merge
  # we want a `"Cypress"` key at the root of the JSON object, with the `"coverage"` inside it
  generated_json_file = File.join(SimpleCov.coverage_path, SimpleCovJSONFormatter::ResultExporter::FILENAME)
  content = { "Cypress" => JSON.parse(File.read(generated_json_file)) }
  File.write(generated_json_file, content.to_json)
end

Now we can run rails simplecov_json_report to generate the coverage/cypress_coverage.json file.

Note that this step will show a really high code coverage percentage, over 75%, but this is not correct since it includes config, test, and other files for this calculation!

Merging Results

Now we get to the final step: we have the coverage/.resultset.json and coverage/cypress_coverage.json files that we need to merge into a single file.

For this, we have created another custom Rake task that uses SimpleCov’s collate method:

# https://github.com/fastruby/coverage-with-cypress-sample-app/blob/main/lib/tasks/json_coverage.rake#L34

namespace :coverage do
  desc "Merge Minitest's and Cypress' code coverage json files into one"
  task :merge do
    require "simplecov"

    # change this if you use different json result names
    coverage_files = Dir["#{SimpleCov.coverage_path}/.resultset.json"] + Dir["#{SimpleCov.coverage_path}/cypress_coverage.json"]

    # make sure to use the `rails` profile to not add noise with files we won't test
    SimpleCov.collate coverage_files, "rails"
  end
end

And we can run this task with rails coverage:merge and see the new generated code coverage report:

Here we can see the total percentage went up to almost 53%.

And here we can see the UsersController coverage is over 60% and we can see the lines inside the new and create actions are now green as we expected from the Cypress test.

Conclusion

This is a basic example of the approach; in a real production application it may require some tweaks depending on the tool being used, along with the proper setup for CI and proper automation of the extraction and merging of the coverage data.

A similar problem happens when we want to measure the JavaScript/TypeScript code coverage and we are using a Ruby runner like Capybara along with runners like Jest or Cypress. You can read about capturing all the code coverage and merge results together in this article: JavaScript Test Code Coverage in Rails opens a new window .

Do you need help with your tests? Let’s talk! opens a new window