How to run second test example with RSpec::Core::Runner for the same test file

[ArturT]

Subject of the issue

I use RSpec::Core::Runner to run specs https://relishapp.com/rspec/rspec-core/v/3-3/docs/running-specs-multiple-times-with-different-runner-options-in-the-same-process

I run test files and test examples (i.e. spec/example_spec.rb[1:1]) at the same time. I'm getting test files and test example paths from external API. Sometimes it can happen that RSpec::Core::Runner will have to run a second test example B1 from the test file spec/example_spec.rb for which we already run first test example A1 but the second test example B1 is ignored. I don't know how to force RSpec to run B1 test :(

I tried to dig into the source code of RSpec for the last 3 days but I'm stuck. Any help or tips where can I look for a solution would be great.

Your environment

• Ruby version: 2.6.5
• rspec-core version: 3.9.1

Steps to reproduce

Source code: https://github.com/ArturT/rspec-test-examples

# spec/example_spec.rb
describe 'Example 1' do
  it 'A1' do
    expect(true).to be true
  end

  it 'B1' do
    expect(true).to be true
  end
end

# spec/example2_spec.rb
describe 'Example 2' do
  it 'A2' do
    expect(true).to be true
  end

  it 'B2' do
    expect(true).to be true
  end
end

# spec/example3_spec.rb
describe 'Example 3' do
  it 'A3' do
    expect(true).to be true
  end

  it 'B3' do
    expect(true).to be true
  end
end

Script to run tests:

# run_tests.rb
puts 'Run tests'

require 'rspec/core'

def start_runner(cli_args)
  puts '-'*50
  args = ['--format', 'documentation'] + cli_args
  options = RSpec::Core::ConfigurationOptions.new(args)
  RSpec::Core::Runner.new(options).run($stderr, $stdout)

  RSpec.clear_examples
end


cli_args = ['spec/example_spec.rb[1:1]', 'spec/example2_spec.rb']
start_runner(cli_args)

cli_args = ['spec/example_spec.rb[1:2]', 'spec/example3_spec.rb']
start_runner(cli_args)

Expected behavior

$ ruby run_tests.rb
Run tests
--------------------------------------------------
Run options: include {:ids=>{"./spec/example_spec.rb"=>["1:1"]}}

Example 1
  A1

Example 2
  A2
  B2

Finished in 0.00182 seconds (files took 0.16182 seconds to load)
3 examples, 0 failures

--------------------------------------------------
Run options: include {:ids=>{"./spec/example_spec.rb"=>["1:2"]}}

Example 1
  B1              <---- example B1 should run here

Example 3
  A3
  B3

Finished in 0.00056 seconds (files took 0.00535 seconds to load)
2 examples, 0 failures

Actual behavior

$ ruby run_tests.rb
Run tests
--------------------------------------------------
Run options: include {:ids=>{"./spec/example_spec.rb"=>["1:1"]}}

Example 1
  A1

Example 2
  A2
  B2

Finished in 0.00182 seconds (files took 0.16182 seconds to load)
3 examples, 0 failures

--------------------------------------------------
Run options: include {:ids=>{"./spec/example_spec.rb"=>["1:2"]}}

Example 3
  A3
  B3

Finished in 0.00056 seconds (files took 0.00535 seconds to load)
2 examples, 0 failures

[ArturT]

Hi @myronmarston , I saw you shared some knowledge with a bit similar problem rspec/rspec#27

Maybe you will be able to point me where to look for? Thank you.

[pirj]

Hey @ArturT!
Maybe this may give you a clue.
You seem to have asked a similar question in the past, maybe you could consider using RSpec.clear_examples now?

[JonRowe]

You've only cleared your examples, not your filters, so you end up building combined filters.

There is a private api on configuration to reset your filters, but the public way to fix this is to change clear_examples for reset. e.g RSpec.reset

[ArturT]

@pirj @JonRowe Thank you for the help. The RSpec.reset does help for the example repo I've created and I get expected output but when I try to use it with Rails and controller spec then with 2nd run of RSpec::Core::Runner that gets path to controller spec I get error like:

Failures:

  1) ArticlesController#index should receive all(*(any args)) 1 time
     Failure/Error: get :index

     NoMethodError:
       undefined method `get' for #<RSpec::ExampleGroups::ArticlesController::Index:0x00007fc98f711a00>

....
### NOTE: Here is an example for not working rails routes path

  2) Calculator when add two numbers result is 0
     Failure/Error: visit calculator_index_path

     NameError:
       undefined local variable or method `calculator_index_path' for #<RSpec::ExampleGroups::Calculator::WhenAddTwoNumbers:0x00007fdd2aec2318>

...
### NOTE: Here is are missing shared examples:

An error occurred while loading ./spec/services/calculator_spec.rb.
Failure/Error: it_behaves_like 'calculator'

ArgumentError:
  Could not find shared examples "calculator"

I added RSpec.configuration.load_spec_files but this seems to be not enough.

I suspect RSpec config from spec/rails_helper.rb was reset and also shared examples removed.

options = RSpec::Core::ConfigurationOptions.new(cli_args)
RSpec.configuration.load_spec_files

RSpec::Core::Runner.new(options).run($stderr, $stdout)
           
RSpec.reset

I'm not sure what's the good direction to go. Should I somehow try to force Ruby to reload the content of spec/rails_helper.rb file to get my initial RSpec config state or maybe instead of using RSpec.reset I should try to monkey patch RSpec to somehow let me run test example B1.

Any tips would be great. Thank you.

[JonRowe]

Yes RSpec.reset is a complete reset of rspec back to basics and you will need to reload your config. As you are programatically loading specs here I would make your config programatic rather than relying on files.

However you can also use RSpec.configuration.clear_filters with clear_examples its a private API so may change at a later date.

[ArturT]

@JonRowe Thanks for tips. I can't find a definition for RSpec.configuration.clear_filters https://github.com/rspec/rspec-core/search?q=clear_filters&unscoped_q=clear_filters and it's undefined method when I try to call it in the code.

There is a method RSpec.configuration.reset_filters. I guess it does what you meant but it still did not allow me to run example B1.

I found that clear_examples already calls reset_filters

[agis]

I'm too bumping into the same issue when using RSpec.clear_examples in the same process and then issue two RSpec::Core::Runner.new().run with id filters that point to the same spec file. I also observe the what the original issue describes, the second time that the same file is executed (albeit with a different ID filter), that example is not executed.

(Note: I've tried RSpec.reset and RSpec.world.reset but I got a lot of weird failures. I've then tried to load spec_helper.rb everytime before the run but still no luck.)

Anyway, I did some digging around and this is what I've found so far.

The reason that the example is not executed, is not because the filters are wrong. It's because the example metadata that RSpec populates are wrong, the second time the file is loaded. So if you have a spec file like this:

# foo_spec.rb
describe "foo"
  it "bar" do
    expect(true).to be true
  end
end

The expected filter to run the example would be foo_spec.rb[1:1]. However, when we do:

2.times do
  RSpec.clear_examples
  RSpec.world.prepare_example_filtering
  opts = RSpec::Core::ConfigurationOptions.new(["foo_spec[1:1]"])
  RSpec::Core::Runner.new(opts).run($stdout, $stderr)
end

The filter (1:1) is correctly picked up by RSpec and in the first iteration, the published example Metadata#scoped_id is populated as expected, 1:1. In the second iteration though, the metadata scoped_id becomes 2:1 instead of 1:1. This is set in:

rspec-core/lib/rspec/core/metadata.rb

Lines 185 to 189 in b77634d

	def build_scoped_id_for(file_path)
	index = @index_provider.call(file_path).to_s
	parent_scoped_id = metadata.fetch(:scoped_id) { return index }
	"#{parent_scoped_id}:#{index}"
	end

That index variable is set to 2 . From there, the FilterManager naturally doesn't detect any examples to run, since 1:1 (what we requested) doesn't match 2:1 (what RSpec ended up publishing):

rspec-core/lib/rspec/core/world.rb

Lines 34 to 38 in b77634d

	def prepare_example_filtering
	@filtered_examples = Hash.new do |hash, group|
	hash[group] = filter_manager.prune(group.examples)
	end
	end

So filter_manager.prune(group.examples) returns an empty array.

I haven't had the time to investigate further, so I don't know if there's anything that we can do in RSpec to make it handle such cases in a backwards compatible way. I'm not even sure that this behavior of the index_provider is expected.

@JonRowe, @pirj, @myronmarston any ideas?

[JonRowe]

The issue you've encountered @agis is due to the filters not reseting between runs, .reset is your solution, you just need to capture the config before your first run, duplicate and reuse it, rather than letting the default occur (as you have rails helpers). You could also reload the rails helpers either would work but these are not mainstream use cases.

[agis]

@JonRowe To make sure I understand, by "filters" you're referring FilterManager? If so, I believe this is not the culprit because I see that it is set as expected in RSpec.world:

# 1st iteration/run
=> #<RSpec::Core::FilterManager:0x000055698a282028
 @exclusions=
  #<RSpec::Core::FilterRules:0x000055698a282000
   @opposite=#<RSpec::Core::InclusionRules:0x000055698a281fb0 @opposite=#<RSpec::Core::FilterRules:0x000055698a282000 ...>, @rules={:ids=>{"./spec/controllers/validation_controller_spec.rb"=>["1:1:1"]}}>,
   @rules={}>,
 @inclusions=
  #<RSpec::Core::InclusionRules:0x000055698a281fb0
   @opposite=#<RSpec::Core::FilterRules:0x000055698a282000 @opposite=#<RSpec::Core::InclusionRules:0x000055698a281fb0 ...>, @rules={}>,
   @rules={:ids=>{"./spec/controllers/validation_controller_spec.rb"=>["1:1:1"]}}>>


# 2nd iteration/run
=> #<RSpec::Core::FilterManager:0x000055699d80f0c8
 @exclusions=
  #<RSpec::Core::FilterRules:0x000055699d80f0a0
   @opposite=#<RSpec::Core::InclusionRules:0x000055699d80f050 @opposite=#<RSpec::Core::FilterRules:0x000055699d80f0a0 ...>, @rules={:ids=>{"./spec/controllers/validation_controller_spec.rb"=>["1:1:1"]}}>,
   @rules={}>,
 @inclusions=
  #<RSpec::Core::InclusionRules:0x000055699d80f050
   @opposite=#<RSpec::Core::FilterRules:0x000055699d80f0a0 @opposite=#<RSpec::Core::InclusionRules:0x000055699d80f050 ...>, @rules={}>,
   @rules={:ids=>{"./spec/controllers/validation_controller_spec.rb"=>["1:1:1"]}}>>

They do seem set up properly to me, but perhaps I'm missing something?

[JonRowe]

Ah sorry I think I'm getting two people confused here, if the id is generating 2:1 on a second pass thats a bug and needs to be fixed.

[agis]

Ah sorry I think I'm getting two people confused here, if the id is generating 2:1 on a second pass thats a bug and needs to be fixed.

Good to know, I've found the culprit and I'm opening a new PR soon.

[shadre]

Ah sorry I think I'm getting two people confused here, if the id is generating 2:1 on a second pass thats a bug and needs to be fixed.

Good to know, I've found the culprit and I'm opening a new PR soon.

Glad you discovered this @agis. We've found this behavior was interrupting with our use case, too (as we are also working on a solution that is reusing the runner and wanted to avoid reloading the whole config).

At the time we didn't think the lack of reset for the count was a bug - just thought our use case lies outside of what RSpec interface expects. Glad this has been cleared up! For what it's worth, we ended up resetting the hash by the hacky RSpec.world.instance_variable_set(:@example_group_counts_by_spec_file, Hash.new(0)). 🙈

[agis]

@shadre wow, that would've definitely saved me 2 or 3 hours of debugging 😅 Nevertheless it's good to know you've found the solution.

[shadre]

@shadre wow, that would've definitely saved me 2 or 3 hours of debugging sweat_smile Nevertheless it's good to know you've found the solution.

Yeah, I'm gutted to know you could have avoided that. :( Glad you too got it working, though!