Coverage dropped in Rails 7.1 (fixed)

[ArthurWD]

I've had some trouble understanding why my test coverage dropped after upgrading to Rails 7.1. I finally fixed the issue which I'll explain below. I'm not sure if this is intended this way, but if so, this might be useful for others to find.

After upgrading Rails, I noticed the coverage dropped significantly. After some diagnosing, I realised lib files which are required in config/application.rb had a coverage of 0%, although they were being covered by tests.

After a while, I noticed the coverage only dropped when using rails test. When I tried rails test test/integration test/channel test/controllers test/jobs test/lib test/mailers test/models, the coverage was correct.

Then, I reduced the issue to this Rails commit, causing the test:prepare to be run when calling rails test without args. Via this code path, config/application.rb was loaded before test_helper.rb.

I've now moved SimpleCov.start to bin/rails, like suggested in the Readme. This works, although I now have to specify RAILS_ENV=test when running rails test (this was done automatically in test_helper.rb before).

Do we need to update the Readme to warn that files required in config/application.rb are not included in the coverage when running rails test in Rails 7.1?

[zonque]

Thanks a lot for sharing this. I've been searching for hours for a solution on this.

[chriscz]

Thanks for sharing this.

It's probably best to put the suggested change inside of your config/boot.rb so that it works both with bin/rails as well as bundle exec rails. For anyone intereseted, this is what we went with:

# config/boot.rb
ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../Gemfile", __dir__)

require "bundler/setup" # Set up gems listed in the Gemfile.
require "bootsnap/setup" # Speed up boot time by caching expensive operations.

if File.basename($PROGRAM_NAME) == "rails" && ARGV[0] =~ /^test/ || ENV["RAILS ENV"] == "test"
  require_relative "../test/test_helper/coverage"
end

# test/test_helper.rb (top of the file)
ENV["RAILS_ENV"] ||= "test"
require_relative "test_helper/coverage"

# test/test_helper/coverage.rb
require "simplecov"

if ENV["CI"]
  require "simplecov-json"

  SimpleCov.formatters = SimpleCov::Formatter::MultiFormatter.new([
    SimpleCov::Formatter::HTMLFormatter,
    SimpleCov::Formatter::JSONFormatter,
  ])
end

SimpleCov.start("rails") if ENV["COVERAGE"] || ENV["CI"]