Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix cucumber coverage #898

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

Fix cucumber coverage #898

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
d44a464
Restore coverage collection for cucumber features
mscottford Apr 6, 2023
b9f98f8
Suppress console output generated by Simplecov HTML formatter
mscottford Apr 6, 2023
7c71251
Adjusts timeouts based on the extra work that's being performed
mscottford Apr 6, 2023
f8ed82e
Adjusts timings to account for extra work being done by SImplecov
mscottford Apr 7, 2023
edd5992
Removes coverage directory before running full test suite
mscottford Apr 7, 2023
f196a14
Merge branch 'cucumber:main' into fix-cucumber-coverage
mscottford Jun 20, 2023
0cbb23b
Switches to double quotes
mscottford Jul 7, 2023
fb37cfe
Adds class level documentation
mscottford Jul 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion Rakefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ Bundler.setup
require "cucumber/rake/task"
require "rspec/core/rake_task"
require "rubocop/rake_task"
require "rake/clean"

Cucumber::Rake::Task.new do |t|
t.cucumber_opts = %w(--format progress)
Expand All @@ -21,8 +22,10 @@ end

RSpec::Core::RakeTask.new

CLOBBER << "coverage/**/*"

desc "Run the whole test suite."
task test: [:spec, :cucumber]
task test: [:clobber, :spec, :cucumber]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This shouldn't normally be needed. Can you remove this commit please?


RuboCop::RakeTask.new

Expand Down
12 changes: 6 additions & 6 deletions features/02_configure_aruba/basics.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ Feature: Usage of configuration
Given a file named "spec/support/aruba_config.rb" with:
"""ruby
Aruba.configure do |config|
config.exit_timeout = 0.5
config.exit_timeout = 0.7
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather avoid these changes to the timeouts. Aruba's tests suite takes a long time as it is. Also, I ran the full test suite without these timeout changes and there were no failures.

Are you able to run the test suite without failures from the main branch?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I remember correctly, I increased the timeouts so that the tests would pass the test-windows (3.0) GitHub Action. The log from that run isn't showing up anymore when I take a look at it, so I'm not able to confirm that. I'll take out the changes and see what happens.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's also worth noting, that simplecov takes some time to start, so that could throw the timing off.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm .. maybe aruba-test-cli inherits RUBYOPT and therefore loads simplecov_setup? That shouldn't happen...

end
"""
And a file named "spec/support/hooks.rb" with:
Expand Down Expand Up @@ -96,7 +96,7 @@ Feature: Usage of configuration
Given a file named "features/support/aruba_config.rb" with:
"""ruby
Aruba.configure do |config|
config.exit_timeout = 0.5
config.exit_timeout = 0.7
end
"""
And a file named "features/run.feature" with:
Expand All @@ -107,7 +107,7 @@ Feature: Usage of configuration
Then the exit status should be 0

Scenario: Slow command
When I run `aruba-test-cli 1.0`
When I run `aruba-test-cli 1.5`
Then the exit status should be 128
"""
When I run `cucumber`
Expand All @@ -122,7 +122,7 @@ Feature: Usage of configuration
Given a file named "features/support/aruba_config.rb" with:
"""ruby
Aruba.configure do |config|
config.exit_timeout = 0.2
config.exit_timeout = 0.7
end
"""
And a file named "features/support/hooks.rb" with:
Expand All @@ -140,11 +140,11 @@ Feature: Usage of configuration

@slow-command
Scenario: Slow command known by the developer
When I run `aruba-test-cli 0.5`
When I run `aruba-test-cli 1`
Then the exit status should be 0

Scenario: Slow command which might be a failure
When I run `aruba-test-cli 0.5`
When I run `aruba-test-cli 1`
Then the exit status should be 128
"""
When I run `cucumber`
Expand Down
4 changes: 2 additions & 2 deletions features/03_testing_frameworks/cucumber/steps/command/check_for_exit_statuses.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ Feature: Check exit status of commands
"""
Feature: Failing program
Scenario: Run command
Given the default aruba exit timeout is 0.4 seconds
Given the default aruba exit timeout is 0.7 seconds
When I successfully run `aruba-test-cli`
"""
When I run `cucumber`
Expand All @@ -112,7 +112,7 @@ Feature: Check exit status of commands
Feature: Failing program
Scenario: Run command
Given the default aruba exit timeout is 0 seconds
When I successfully run `aruba-test-cli` for up to 0.4 seconds
When I successfully run `aruba-test-cli` for up to 0.7 seconds
"""
When I run `cucumber`
Then the features should all pass
Expand Down
22 changes: 11 additions & 11 deletions features/03_testing_frameworks/cucumber/steps/command/stop_command.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,8 @@ Feature: Stop commands
"""
Feature: Run it
Scenario: Run command
Given the default aruba exit timeout is 0.2 seconds
And I wait 0.2 seconds for a command to start up
Given the default aruba exit timeout is 1.5 seconds
And I wait 0.7 seconds for a command to start up
When I run `aruba-test-cli1` in background
And I run `aruba-test-cli2` in background
And I terminate the command started last
Expand Down Expand Up @@ -97,8 +97,8 @@ Feature: Stop commands
Background:

Scenario: Run command
Given the default aruba exit timeout is 0.2 seconds
And I wait 0.2 seconds for a command to start up
Given the default aruba exit timeout is 0.7 seconds
And I wait 0.5 seconds for a command to start up
When I run `aruba-test-cli1` in background
And I run `aruba-test-cli2` in background
And I stop the command started last
Expand Down Expand Up @@ -145,10 +145,10 @@ Feature: Stop commands
"""
Feature: Run it
Background:
Given the default aruba exit timeout is 0.2 seconds
Given the default aruba exit timeout is 0.3 seconds

Scenario: Run command
Given I wait 0.2 seconds for a command to start up
Given I wait 0.3 seconds for a command to start up
When I run `aruba-test-cli1` in background
When I run `aruba-test-cli2` in background
And I terminate the command "aruba-test-cli1"
Expand Down Expand Up @@ -197,10 +197,10 @@ Feature: Stop commands
"""
Feature: Run it
Background:
Given the default aruba exit timeout is 0.2 seconds
Given the default aruba exit timeout is 0.3 seconds

Scenario: Run command
Given I wait 0.2 seconds for a command to start up
Given I wait 0.3 seconds for a command to start up
When I run `aruba-test-cli1` in background
And I run `aruba-test-cli2` in background
When I stop the command "aruba-test-cli1"
Expand Down Expand Up @@ -239,7 +239,7 @@ Feature: Stop commands
Feature: Run it
Scenario: Run command
Given the default aruba stop signal is "HUP"
And the default aruba exit timeout is 0.2 seconds
And the default aruba exit timeout is 0.7 seconds
When I run `aruba-test-cli`
Then the exit status should be 155
"""
Expand Down Expand Up @@ -271,8 +271,8 @@ Feature: Stop commands
"""
Feature: Run it
Scenario: Run command
Given the default aruba exit timeout is 0.3 seconds
And I wait 0.2 seconds for a command to start up
Given the default aruba exit timeout is 0.7 seconds
And I wait 0.7 seconds for a command to start up
When I run `aruba-test-cli1` in background
And I terminate the command started last
Then the exit status should be 100
Expand Down
14 changes: 7 additions & 7 deletions features/04_aruba_api/command/run_command.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ Feature: Run command
"""ruby
require 'spec_helper'

RSpec.describe 'Run command', type: :aruba, exit_timeout: 0.1, startup_wait_time: 0.4 do
RSpec.describe 'Run command', type: :aruba, exit_timeout: 0.5, startup_wait_time: 1.5 do
before do
run_command('aruba-test-cli')
last_command_started.send_signal 'HUP'
Expand Down Expand Up @@ -154,7 +154,7 @@ Feature: Run command
"""ruby
require 'spec_helper'

RSpec.describe 'Run command', type: :aruba, exit_timeout: 0.4 do
RSpec.describe 'Run command', type: :aruba, exit_timeout: 1.0 do
before { run_command('aruba-test-cli') }

it 'runs the command with the expected results' do
Expand Down Expand Up @@ -233,10 +233,10 @@ Feature: Run command
"""ruby
require 'spec_helper'

RSpec.describe 'Run command', type: :aruba, exit_timeout: 0.1 do
RSpec.describe 'Run command', type: :aruba, exit_timeout: 1.0 do
before do
run_command 'aruba-test-cli1', startup_wait_time: 0.5
run_command 'aruba-test-cli2', startup_wait_time: 0.2
run_command 'aruba-test-cli1', startup_wait_time: 1.0
run_command 'aruba-test-cli2', startup_wait_time: 0.7
last_command_started.send_signal 'HUP'
end

Expand Down Expand Up @@ -289,8 +289,8 @@ Feature: Run command

RSpec.describe 'Run command', type: :aruba do
before do
run_command 'aruba-test-cli1', exit_timeout: 0.4
run_command 'aruba-test-cli2', exit_timeout: 0.1
run_command 'aruba-test-cli1', exit_timeout: 1.5
run_command 'aruba-test-cli2', exit_timeout: 0.7
end

it 'runs both commands with the expected results' do
Expand Down
8 changes: 4 additions & 4 deletions features/04_aruba_api/command/stop_single_command.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ Feature: Stop command
"""ruby
require 'spec_helper'

RSpec.describe 'Run command', type: :aruba, exit_timeout: 0.2 do
RSpec.describe 'Run command', type: :aruba, exit_timeout: 0.5 do
before { run_command('aruba-test-cli') }
before { last_command_started.stop }
it { expect(last_command_started).to be_successfully_executed }
Expand All @@ -52,7 +52,7 @@ Feature: Stop command
"""ruby
require 'spec_helper'

RSpec.describe 'Run command', type: :aruba, startup_wait_time: 0.1, exit_timeout: 0.3 do
RSpec.describe 'Run command', type: :aruba, startup_wait_time: 0.1, exit_timeout: 0.5 do
before { run_command('aruba-test-cli') }
before { find_command('aruba-test-cli').stop }
it { expect(last_command_started).to be_successfully_executed }
Expand Down Expand Up @@ -85,7 +85,7 @@ Feature: Stop command

Aruba.configure do |config|
config.stop_signal = 'HUP'
config.exit_timeout = 0.2
config.exit_timeout = 0.5
end

RSpec.describe 'Run command', type: :aruba do
Expand Down Expand Up @@ -120,7 +120,7 @@ Feature: Stop command

Aruba.configure do |config|
config.stop_signal = 'HUP'
config.exit_timeout = 0.2
config.exit_timeout = 0.5
end

RSpec.describe 'Run command', type: :aruba do
Expand Down
10 changes: 5 additions & 5 deletions features/04_aruba_api/filesystem/use_fixtures.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Feature: Use fixtures in your tests
Given I use a fixture named "cli-app"

Scenario: Use a fixture for your tests
Given a file named "spec/fixture_spec.rb" with:
Given a file named "spec/fixture_a_spec.rb" with:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When you remove the changes to the timeouts (see my comment above) please keep the changes in this file.

"""
require 'spec_helper'

Expand All @@ -30,7 +30,7 @@ Feature: Use fixtures in your tests
Then the specs should all pass

Scenario: Pass file from fixtures to your command
Given a file named "spec/fixture_spec.rb" with:
Given a file named "spec/fixture_b_spec.rb" with:
"""
require 'spec_helper'

Expand All @@ -56,7 +56,7 @@ Feature: Use fixtures in your tests
Then the specs should all pass

Scenario: Use a fixture for your tests in test/
Given a file named "spec/fixture_spec.rb" with:
Given a file named "spec/fixture_c_spec.rb" with:
"""
require 'spec_helper'

Expand All @@ -76,7 +76,7 @@ Feature: Use fixtures in your tests
Then the specs should all pass

Scenario: Use a fixture for your tests in spec/
Given a file named "spec/fixture_spec.rb" with:
Given a file named "spec/fixture_d_spec.rb" with:
"""
require 'spec_helper'

Expand All @@ -96,7 +96,7 @@ Feature: Use fixtures in your tests
Then the specs should all pass

Scenario: Use a fixture for your tests in features/
Given a file named "spec/fixture_spec.rb" with:
Given a file named "spec/fixture_e_spec.rb" with:
"""
require 'spec_helper'

Expand Down
15 changes: 9 additions & 6 deletions features/support/env.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,29 +1,32 @@
$LOAD_PATH.unshift File.expand_path("../../lib", __dir__)

# Has to be the first file required so that all other files show coverage information
require "simplecov" unless RUBY_PLATFORM.include?("java")
require_relative "simplecov/simplecov_setup" unless RUBY_PLATFORM.include?("java")

# Standard Library
require "fileutils"
require "pathname"

# Gems
require "aruba/cucumber"
require "aruba/config/jruby"
require "rspec/expectations"

Before do |test_case|
Around do |test_case, block|
command_name = "#{test_case.location.file}:#{test_case.location.line} # #{test_case.name}"

# Used in simplecov_setup so that each scenario has a different name and
# their coverage results are merged instead of overwriting each other as
# 'Cucumber Features'
ENV["SIMPLECOV_COMMAND_NAME"] = command_name.to_s
set_environment_variable 'SIMPLECOV_COMMAND_NAME', command_name.to_s

simplecov_setup_pathname =
Pathname.new(__FILE__).expand_path.parent.join("simplecov_setup")
Pathname.new(__FILE__).expand_path.parent.join('simplecov').to_s

# set environment variable so child processes will merge their coverage data
# with parent process's coverage data.
ENV["RUBYOPT"] = "-r#{simplecov_setup_pathname} #{ENV['RUBYOPT']}"
prepend_environment_variable 'RUBYOPT', "-I#{simplecov_setup_pathname} -rsimplecov_setup "

with_environment do
block.call
end
end
31 changes: 31 additions & 0 deletions features/support/simplecov/simplecov_setup.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# @note this file is loaded in env.rb to setup simplecov using RUBYOPTs for
# child processes and @in-process
unless RUBY_PLATFORM.include?("java")
require "simplecov"
root = File.expand_path("../../..", __dir__)
command_name = ENV["SIMPLECOV_COMMAND_NAME"] || "Cucumber Features"
SimpleCov.command_name(command_name)
SimpleCov.root(root)

# Run simplecov by default
SimpleCov.start unless ENV.key? "ARUBA_NO_COVERAGE"

module SimpleCov
module Formatter
class HTMLFormatter
def format(result)
Dir[File.join(File.dirname(__FILE__), "../public/*")].each do |path|
FileUtils.cp_r(path, asset_output_path)
end
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't work because the original code refers to a public/ directory inside simplecov-html. So, if you start with no coverage/ directory and just run a cucumber scenario, there will be no styling in the HTML result.

One way to fix this would be to prepend a module instead that overrides format with something that temporarily redirects $stdout.

However another option would be to set SimpleCov.formatter = SimpleCov::Formatter::SimpleFormatter when ENV["SIMPLECOV_COMMAND_NAME"] is set. That should avoid the message and the needless copying of files. Also, it will make the SimpleCov message be printed at the very end of the Cucumber run.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! I'll give the SimpleFormatter approach a try.


File.open(File.join(output_path, "index.html"), "wb") do |file|
file.puts template("layout").result(binding)
end
File.open(File.join(output_path, "summary.txt"), "w") do |file|
file.puts output_message(result)
end
end
end
end
end
end
12 changes: 0 additions & 12 deletions features/support/simplecov_setup.rb
View file
Open in desktop

This file was deleted.