Support retries in snapshot queue

[kuceb]

Cypress version 5.0 now support test retries, so a test may have multiple attempts meaning cy.percySnapshot may be called multiple times per-test if the test retries after the snapshot
This causes the following error:

[percy] snapshot taken: 'Test retry'
[percy] snapshot taken: 'Test retry'
[percy] StatusCodeError 400 - {"errors":[{"status":"bad_request"},
{"source":{"pointer":"/data/attributes/base"},
"detail":"The name of each snapshot must be unique, and this name already exists in the build: 
'Test retry' -- You can fix this by passing a 'name' param when creating the snapshot. 
See the docs for more info on identifying snapshots for your specific client: 
https://percy.io/docs"}]} 

(see cypress-io/cypress-react-unit-test#362 an example error'd run)

In order to support Cypress test retries, the plugin will need to hold off on sending the DOM snapshot until after the final attempt has run. Likely:

let testSnapshots = {}
snapshot() {
  // ...
  testSnapshots[name] = snapshotData
  // ...
}

// runs for every attempt
Cypress.on('test:after:run:async', (test) => {
  if (test.final) {
    // Cypress adds .final === true on the final attempt of a test
    sendSnapshots(testSnapshots)
	  testSnapshots = {}
  }
})

see the PR to fix cypress-plugin-snapshots as an example of adding tests for multiple Cypress versions meinaart/cypress-plugin-snapshots#141

[EvanLovely]

Here's how I'm handling it for now, I've made cy.vrt() that takes the same function signature as cy.percySnapshot(), then does two things to the snapshot title:

1. Prepends it with the nested test titles and formats like "My Spec > The Test >" and puts that before the text I pass into cy.vrt()
2. Appends "attempt X" where X is the attempt number. Nothing is appended during the first attempt.

import type { SnapshotOptions } from '@percy/core';

Cypress.Commands.add('vrt', (name: string, options?: SnapshotOptions) => {
  const { titlePath } = Cypress.currentTest;
  const attempt = cy.state('runnable')._currentRetry + 1;
  const title = attempt > 1 ? `${name} attempt ${attempt}` : name;
  // eslint-disable-next-line no-restricted-properties
  cy.percySnapshot([...titlePath, title].join(' > '), options);
});

Inside support/index.d.ts I have this to get proper TypeScript auto-completion:

type SnapshotOptions = import('@percy/core').SnapshotOptions;

declare namespace Cypress {
  interface Chainable<Subject> {
    /** Take Percy Snapshot for Visual Regression Testing */
    vrt(name: string, options?: SnapshotOptions): Chainable<Subject>;
  }
}

Finally, I've added this to my eslint config so that linting will fail if anyone tries to use cy.percySnaphot():

module.exports = {
  rules: {
    'no-restricted-properties': [
      2,
      {
        object: 'cy',
        property: 'percySnapshot',
        message:
          'Please use `cy.vrt` instead of `cy.percySnapshot` as it handles Cypress retries better',
      },
    ],
  },
};

[Robdel12]

We'll likely be shipping retries support in all SDKs in Q1 2022. Most of the internal work to @percy/core has been done (queues & differing uploads).

That would work as a work around, but you will lose consistent comparisons because the snapshot names are changing.

[sigginjals]

@EvanLovely Thanks for the example. It surprised me a lot that this wasn't default behaviour from Percy.

[Robdel12]

@sigginjals this approach will work but not really. You’ll see which tests are retrying easily in Percy because their name will change and they won’t compare unless your baseline also has snapshot name - retryCount. Basically you’re going to have new and missing snapshots all the time.

[cipivanov]

Any updates on this?

[monsieurnebo]

We'll likely be shipping retries support in all SDKs in Q1 2022. Most of the internal work to @percy/core has been done (queues & differing uploads).

@Robdel12 Any update on this issue? That's a pain :(

[Robdel12]

👋🏼 No updates, more pressing work has taken precedence since then. When there's updates, this issue will be updated and referenced in the relevant PRs

[monsieurnebo]

Thanks for the update :)

[Robdel12]

Please stop bumping. When there’s anything new to share we will, the current status is the same: #281 (comment)

[Robdel12]

Hey everyone! We have an update that you've been patiently waiting for 🎉 Percy's SDKs as of v1.11.0 (#1077) now support retries. This works by waiting to create the snapshots until the end of the tests. This can be configured in the Percy config file. For example, as a .percy.yml:

version: 2
percy:
  defer-uploads: true

This will allow snapshots to be duplicated. Proper documentation will be coming shortly but given the popularity of this issue I figured I'd give a heads up to those who are clamoring for it. 🏎️ 💨