Plugin seems to break test.state property in cy.state('runnable') of a test suite

[proCrichard]

When a test fails and is retried, that test's details in the cy.state('runnable') object is somehow broken. From an example I have here, I can see in the cypress logger that my test has failed:

However the cy.state('runnable') output on that test considers it 'passed':

This is throwing off a plugin I use to report test results. Any information you have on this issue would be helpful.

[kuceb]

@proCrichard could you mention what plugin you are using?

[proCrichard]

I wrote a custom plugin to send test results to datadog.

Starts with an after hook:

after(() => {
  const findTests = suite => {
    suite.tests.forEach(test => {
      cy.task('logTestResults', {
        title: `${test.parent.title}${test.title}`,
        state: test.state,
      });
    });

    if (suite.suites.length < 1) {
      return;
    }
    suite.suites.forEach(nestedSuite => {
      findTests(nestedSuite);
    });
  };

  cy.state('runnable').parent.suites.forEach(suite => {
    findTests(suite);
  });
});

Which triggers this event listener:

const dogapi = require('dogapi');
const changeCase = require('change-case');

module.exports = (on, app) => {
  const dogapiOptions = {
    api_key: process.env.DATADOG_API_KEY,
    app_key: process.env.DATADOG_APP_KEY,
  };
  dogapi.initialize(dogapiOptions);

  on('task', {
    logTestResults({ title, state }) {
      const tags = [];

//This if statement is to catch state=undefined which is sent from tests that failed and were retried.
      if (!state) {
        state = 'failed';
      }

      tags.push(`outcome:${state}`);
      tags.push(`description:${changeCase.snakeCase(title)}`);
      tags.push(`app:${app}`);
      dogapi.metric.send(`app.e2e`, 1, { tags }, (error, results) => {
        console.dir(results); // eslint-disable-line no-console
      });
      return null;
    },
  });
};

[kuceb]

did your test that failed (but had state: passed) fail from a beforeEach hook or in the test body?

What retry number do you have enabled?

[proCrichard]

Retries set to 2
I had the same outcome when I forced a test to fail in:

• before() hook
• beforeEach()
• test body

[kuceb]

@proCrichard alright, and what version of cypress and this plugin?

I just added your after() to my spec for this project, and I am getting the correct state set on the runnables

[proCrichard]

"cypress": "3.4.0",
"cypress-plugin-retries": "1.2.2"

[kuceb]

@proCrichard ok, I think I am reproducing it now. Thanks

[proCrichard]

Ok great. I appreciate you jumping on this for me.

[kuceb]

@proCrichard after spending a couple hours on this, I'll need to edit the Cypress source directly to fix this, but I have a PR to do so here cypress-io/cypress#3968

I would recommend using the module API to pull results out instead of using runnable instances. https://docs.cypress.io/guides/guides/module-api.html#cypress-run. However, I know that can be cumbersome