Simplecov support

[franzliedke]

SimpleCov tries to automatically determine the test framework / tools being used based on process name and environment variables.

Since this project is based on parallel_tests, for which SimpleCov claims support, I was expecting this to "just work" as well.

Apparently, all that needs to be done is exposing two environment variables to the sub processes, as explained here. Would that be feasible? (Happy to send a PR if you approve.)

[franzliedke]

Please excuse my incorrect idioms. 😆

[ilyazub]

@franzliedke Thanks for opening this issue and for offering your help!

Apparently, all that needs to be done is exposing two environment variables to the sub processes, as explained here. Would that be feasible? (Happy to send a PR if you approve.)

Sure! Environment variables can be passed in TurboTests::Runner#start_subprocess.

I've unclearly described the usage of parallel_tests in this project. turbo_tests uses Ruby threads. parallel_tests are used to group spec files per process. Clarified the difference between two projects in fb33eb4.

[franzliedke]

Nice! Thanks for the pointers.

I'm in project mode at work right now. Roughly one month from now, we will have time for stuff like this. See you then. 🙂

[john-h-k]

Hi, this issue is relevant to me and I could probably pick it up earlier than @franzliedke . I'm slightly confused by the SimpleCov docs, as they seem to imply TEST_ENV_NUMBER is enough, but turbo_tests already sets that does it not?

[ilyazub]

turbo_tests already sets that does it not

Yes, turbo_tests sets TEST_ENV_NUMBER. But there's a bug — value for the first process is incorrect (#12).

[mrmason]

If it helps anyone, I managed to get simplecov working with turbo_tests by making sure that each test run was named seperately for simplecov - so for example, add this to spec/rails_unit_helper.rb:

SimpleCov.command_name "test:#{ENV['TEST_ENV_NUMBER']}"

which then results in the following output:

Coverage report generated for test:1, test:2, test:3, test:4 to /../coverage. 12129 / 12784 LOC (94.88%) covered.
Coverage report generated for test:1, test:2, test:3, test:4 to /../coverage.xml. 12129 / 12784 LOC (94.88%) covered

without this, each test runner was overwriting the same coverage files.

[ilyazub]

@mrmason, thank you! SimpleCov users reported thread-safety bugs with parallel_tests: simplecov-ruby/simplecov#559. Have you experienced any of these with turbo_tests?

P.S. SimpleCov.command_name detects TEST_ENV_NUMBER automatically: https://github.com/simplecov-ruby/simplecov/blob/216b2d530bee7c4b8a8fe2898684924bfccfa79a/lib/simplecov/command_guesser.rb#L23-L28

[franzliedke]

P.S. SimpleCov.command_name detects TEST_ENV_NUMBER automatically: https://github.com/simplecov-ruby/simplecov/blob/216b2d530bee7c4b8a8fe2898684924bfccfa79a/lib/simplecov/command_guesser.rb#L23-L28

Yep, and that's what's currently missing in turbo_tests, right? That's what this ticket is about, if I remember correctly. 😆

[mrmason]

@mrmason, thank you! SimpleCov users reported thread-safety bugs with parallel_tests: simplecov-ruby/simplecov#559. Have you experienced any of these with turbo_tests?

P.S. SimpleCov.command_name detects TEST_ENV_NUMBER automatically: https://github.com/simplecov-ruby/simplecov/blob/216b2d530bee7c4b8a8fe2898684924bfccfa79a/lib/simplecov/command_guesser.rb#L23-L28

This is interesting, because without adding the code that I put above, the outputs over write each other and the test coverage isn't correct - not sure why it's not working.

We're not seeing any direct thread-safety bugs - however I am seeing the following error occasionally when the test suite runs:

Finished in 2 minutes 17.2 seconds (files took 5.22 seconds to load)
3130 examples, 0 failures, 3 pending, 1 error occurred outside of examples

This causes the job to fail with an exit code of 1 - but there are no errors logged that I can see - do you have any suggestions to try and find out where/what the 1 error occurred outside of examples is ?

[ilyazub]

Yep, and that's what's currently missing in turbo_tests, right? That's what this ticket is about, if I remember correctly.

@franzliedke Right :-) I've missed that SimpleCov relies on PARALLEL_TEST_GROUPS and TEST_ENV_NUMBER environment variables for automatic detection: https://github.com/simplecov-ruby/simplecov/blob/216b2d530bee7c4b8a8fe2898684924bfccfa79a/lib/simplecov.rb#L434-L436

[ilyazub]

@mrmason Is 1 error occurred outside of examples occurring without SimpleCov too?

Here's what people on StackOverflow suggest:

• Use --bisect with a specific --seed: https://stackoverflow.com/a/49120567/1291371
• Check the example_status_persistence_file_path: https://stackoverflow.com/a/52873584/1291371

P.S. Can be related to https://github.com/serpapi/turbo_tests/pull/20/files/074f68f6d782e1435b779d5631150f114713dc21#diff-c68c40e74f2dc811296e7f0a1088dc810d5cb72d326f703782ad7a22ced5835fR112

[Mayurifag]

Did anyone have success on using simplecov or any other rails code coverage tool with turbo_tests?

[ilyazub]

It's also possible that PARALLEL_TEST_GROUPS environment variable is required for simplecov.

[aprescott]

I ran into this issue while trying to get coverage to work correctly. Setting command_name explicitly in spec_helper.rb as mentioned in #24 (comment) seems to work fine for me.

Setting PARALLEL_TEST_GROUPS explicitly did not work. While SimpleCov does have built-in support for parallel_tests, setting PARALLEL_TEST_GROUPS makes SimpleCov think parallel_tests is in use, and it then attempts to rely on other parallel_tests specifics, directly calling methods within that gem that don't seem applicable to turbo_tests:

~/.gem/ruby/3.2.3/gems/parallel_tests-4.6.0/lib/parallel_tests.rb:42:in `fetch': key not found: "PARALLEL_PID_FILE" (KeyError)
Did you mean?  "PARALLEL_TEST_GROUPS"
	from ~/.gem/ruby/3.2.3/gems/parallel_tests-4.6.0/lib/parallel_tests.rb:42:in `pid_file_path'
	from ~/.gem/ruby/3.2.3/gems/parallel_tests-4.6.0/lib/parallel_tests.rb:38:in `pids'
	from ~/.gem/ruby/3.2.3/gems/parallel_tests-4.6.0/lib/parallel_tests.rb:90:in `number_of_running_processes'
	from ~/.gem/ruby/3.2.3/gems/parallel_tests-4.6.0/lib/parallel_tests.rb:86:in `wait_for_other_processes_to_finish'
	from ~/.gem/ruby/3.2.3/gems/simplecov-0.22.0/lib/simplecov.rb:279:in `wait_for_other_processes'
	from ~/.gem/ruby/3.2.3/gems/simplecov-0.22.0/lib/simplecov.rb:110:in `result'
	from ~/.gem/ruby/3.2.3/gems/simplecov-0.22.0/lib/simplecov/configuration.rb:197:in `block in at_exit'
	from ~/.gem/ruby/3.2.3/gems/simplecov-0.22.0/lib/simplecov.rb:189:in `run_exit_tasks!'
	from ~/.gem/ruby/3.2.3/gems/simplecov-0.22.0/lib/simplecov.rb:179:in `at_exit_behavior'
	from ~/.gem/ruby/3.2.3/gems/simplecov-0.22.0/lib/simplecov/defaults.rb:30:in `block in <top (required)>'

Setting command_name avoided all these problems, but it would of course be great if Simplecov just worked without any extra config!

[jamesst20]

#24 (comment) is also working for me, but i'm getting multiple output for simplecov instead of a summarized one. Is there a workaround for that?

[phyzical]

were wondering the same, i think thats the last thing missing from this imo

Ah nvm can confirm being explicit does give a summary

SimpleCov.command_name "test:#{ENV['TEST_ENV_NUMBER']}"