Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cucumber 7 HTML formatter shows scenario as failed even if it passed in retry. #1533

Closed
FrankieDcom opened this issue Feb 19, 2021 · 5 comments · Fixed by #1631
Closed

Cucumber 7 HTML formatter shows scenario as failed even if it passed in retry. #1533

FrankieDcom opened this issue Feb 19, 2021 · 5 comments · Fixed by #1631
Assignees
Labels
🐛 bug Defect / Bug library: cucumber-react ✅ accepted The core team has agreed that it is a good idea to fix this

Comments

@FrankieDcom
Copy link

Hello we have recently upgraded to Cucumber 7, and have started using the out of the box HTML formatter for our test reports. We also have retries enabled on our tests.

We are noticing when a test fails on the first run, but passes on a retry, the HTML formatter incorrectly records it as a failed test, while the summary recording it as a success. With a simple scenario:

var count = 0;

When('Will fail first run', function () {
    count++;
    if (count < 2) {
        throw Error(`Fail on first try ${count}`);
    }
});
Feature: Test retry
  Will fail on the first run

  Scenario: Fail first run
    When Will fail first run

Running the following generates the following report and summary:

image

image

As you can see, the HTML report displays a failure while the summary correctly shows the scenario as passing with a warning.

@FrankieDcom FrankieDcom changed the title Cucumber 7 HTML formatter shows feature as failed even if it passed in retry. Cucumber 7 HTML formatter shows scenario as failed even if it passed in retry. Feb 19, 2021
@davidjgoss
Copy link
Contributor

Thanks for the detailed report @FrankieDcom - I'll take a look at this.

@thiagojacinto
Copy link

I just want to add that the same thing happens if we use builtin message formatter. @FrankieDcom, did you find any workaround for that?

@aslakhellesoy aslakhellesoy transferred this issue from cucumber/cucumber-js May 10, 2021
@aslakhellesoy aslakhellesoy added 🐛 bug Defect / Bug ✅ accepted The core team has agreed that it is a good idea to fix this library: cucumber-react labels May 10, 2021
@aslakhellesoy
Copy link
Contributor

I just want to add that the same thing happens if we use builtin message formatter.

@thiagojacinto could you please clarify what you mean by that? The message formatter outputs ndjson messages. Are you seeing something incorrect in the outputted messages?

@thiagojacinto
Copy link

thiagojacinto commented May 11, 2021

@thiagojacinto could you please clarify what you mean by that?

Sure, @aslakhellesoy. If you follow the same steps described by Frankie to reproduce this but adding the message formatter with --format message:./report.json the same thing happens:

the test fails on the first run, but passes on a retry, the message formatter incorrectly records it as a failed test, while the summary recording it as a success

@davidjgoss
Copy link
Contributor

@thiagojacinto the message formatter only reports status at the step level and doesn't aggregate/summarise - this is down to consuming tools. You'll see all the test step results for every attempt, so when there have been retries you probably just want to disregard the data from previous attempts. This is what we're doing now in @cucumber/query, using the testCaseStarted message as a signal to clear previous results if they exist.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🐛 bug Defect / Bug library: cucumber-react ✅ accepted The core team has agreed that it is a good idea to fix this
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants