Jenkins test report shows test passed but in actual it failed

[nikhilesh009]

when I ran the test on the test runner if a test failed after retires, it shows test failed, but i ran on Jenkins test failed but in the test result, it shows test passed but screenshot generated that shows test failed.
is it possible that generated screenshot shows the first attempt failed the test ?

[kuceb]

@nikhilesh009 is it the last test in the suite (last it in a describe block for example)? There may be some edge-cases where the logic is off on knowing whether a test will re-run. I have changes in cypress-core that fixes this, but not sure it can be fixed in this plugin

[nikhilesh009]

Yes, its last test in a describe block. is it possible that the test is stuck too at the end?

[kuceb]

@nikhilesh009 I wasn't able to reproduce this after trying different combinations of nested tests with various hooks. I would be happy to look more into this if you have a minimal reproducible spec file that demonstrates the problem.

[panli666]

hi, i meet same issue in CI, the case was failed, but in report it is pass!

[lavender21]

Also, have the same issue in CI, we use buildkite. Any update about this issue?

[nikhilesh009]

I was concern about test passing/failing, so i intentionally failed my test and then test on Jenkins, On Jenkins test report it shows correctly, the test failed, and in output 3 screenshot of the failed test.
but still, i am not sure its always the case.I am testing always third parties supplier, and sometimes it's up-down.

when I ran the test on the test runner if a test failed after retires, it shows test failed, but i ran on Jenkins test failed but in the test result, it shows test passed but screenshot generated that shows test failed.
is it possible that generated screenshot shows the first attempt failed the test ?

[kuceb]

Can anyone post the simple spec code that can reproduce this? I don't have time to guess and check dozens of combinations of passes/failures with/out hooks and such.

[panli666]

Hi @bkucera I found the issue, when i add after hook to clear the cookies, the case will be marked as pass, even it is failed.

`context('Window', () => {
beforeEach(() => {
cy.visit('https://example.cypress.io/commands/window')
})

it('cy.window() - get the global window object', () => {
Cypress.currentTest.retries(2)
// https://on.cypress.io/window
cy.window().should('have.property', 'top')
cy.get('#navbar > :nth-child(1) > :nth-child(2) > a1').click();
})
after(() => {
cy.clearCookies();
});
})`

[kuceb]

@PansyLi awesome, I'll try to reproduce this again

[kuceb]

🎉 This issue has been resolved in version 1.3.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀