Merging Multiple SimpleCov Coverage Results

As part of our Roadmap service at FastRuby.io, we have to analyze the test suite of the application we are upgrading to give a proper estimate on how long it will take us to upgrade. We use SimpleCov for this.

Most of our clients use parallelization in their continuous integration tools. SimpleCov generates multiple .resultset.json files for the same codebase. Our goal was to have a single result for the whole application, so in this blog post we are going to show you how we solved that problem.

Some of the applications we upgrade are outdated and setting them up can be difficult. Sometimes they have no documentation, missing steps on how to set them up, or outdated installation steps. Even after setup, the test suite of these applications can take several hours to complete and generate a coverage report.

So, we recently decided to take a different approach for this problem. Instead of executing the tests locally and generating the report, we rely on our client's CI configuration to get the coverage data and move on. After all, the coverage report is just a metric to give us an idea on how much effort it will take us to complete the upgrade.

Not everything is rosy, we found a problem with this approach too. Continuous integration services (like CircleCI) allow you to parallelize the execution of any command and a common pattern is to execute different parts of your test suite in different containers. This will make it faster, since now you'll spread the load of your test in different containers. The problem with this is that if you are running SimpleCov it will generate a result for each of your containers. So, to have the full coverage report you'll have to merge all results to generate one final coverage result.

We want to share a little script on how to do the merging and generate a complete coverage.

class SimpleCovMerger
  def self.report_coverage(base_dir:, ci_project_path:, project_path:)
    new(base_dir: base_dir, ci_project_path: ci_project_path, project_path: project_path).merge_results
  end

  attr_reader :base_dir, :ci_project_path, :project_path

  def initialize(base_dir:, ci_project_path:, project_path:)
    @base_dir = base_dir
    @ci_project_path = ci_project_path
    @project_path = project_path
  end

  def merge_results
    require "simplecov"
    require "json"

    results = resultsets.map do |file|
      hash_result = JSON.parse(clean(File.read(file)))
      SimpleCov::Result.from_hash(hash_result)
    end

    result = SimpleCov::ResultMerger.merge_results(*results)

    SimpleCov::ResultMerger.store_result(result)
  end

  private

  def resultsets
    Dir["#{base_dir}/.resultset-*.json"]
  end

  def clean(results)
    results.gsub(ci_project_path, project_path)
  end
end

To use it you'll have to be aware of a couple of parameters:

  • base_dir - This is the directory where you stored all your .resultset.json from your different containers/machines from your CI service
  • ci_project_path - The path where your project is stored in your CI service
  • project_path - The path of the project you are generating a coverage report
SimpleCovMerger.report_coverage(base_dir: "./resultsets", ci_project_path: "/home/ubuntu/the_project/", project_path: "/Users/bronzdoc/projects/fastruby/the_project/")

Conclusion

We hope you'll find this code snippet helpful for your purposes. Please let us know if you have a better way or any ideas for improving it.

Get the book