testing how Percy handles test retries

[bahmutov]

it('retries', { retries: 3 }, () => {
  mount(<Hello />)
  cy.contains('retry') // ensure the component has rendered

  // now let's fail the test - it will retry several times and pass
  cy.percySnapshot('Test retry')
  cy.contains('retry 3', { timeout: 1500 })
})

Right now it fails with

[percy] snapshot taken: 'Test retry'
[percy] snapshot taken: 'Test retry'
[percy] StatusCodeError 400 - {"errors":[{"status":"bad_request"},
{"source":{"pointer":"/data/attributes/base"},
"detail":"The name of each snapshot must be unique, and this name already exists in the build: 
'Test retry' -- You can fix this by passing a 'name' param when creating the snapshot. 
See the docs for more info on identifying snapshots for your specific client: 
https://percy.io/docs"}]} 
| Fri Jul 31 2020 13:44:31 GMT-0400 (Eastern Daylight Time)

Suggestion: Percy Cypress plugin can take the test retry into account and ONLY consider the last test

[Robdel12]

👋 Hey-o! This one is going to be quite tricky. The Percy API enforces snapshot names to be unique and once percySnapshot is called it'll create that snapshot.

Is there anyway to only call percySnapshot once this test succeeds? Is there anything within Cypress that would make it possible to defer this POST until the test succeeds? https://github.com/percy/percy-cypress/blob/master/lib/index.ts#L20 🤔 That's pretty much the crux here -- we'll need a way to defer POST'ing the DOM snapshot + options to the local Percy server until it's a successful/passing test.

[brian-mann]

@Robdel12 yes - we should probably just submit a PR here. The implementation should buffer the snapshot in memory until the test finishes - with retries overwriting the previously stored snapshots buffered in memory, and then flushed when the test finishes.

It'd probably be easier for us to do the PR since we know the implementation.