test: change some skipped tests to use retries

[flotwig]

• Closes

User facing changelog

Additional details

• Followup to chore: skip or fix flaky tests #23167 and chore: skip or fix flaky tests II #23386
• Instead of skipping, sets retries to 15 on tests marked as flaky in those PRs
• Some consistent failures exist now, where previously there was flake:
  1. uses cy.origin twice in origin/validation.cy.ts https://dashboard.cypress.io/projects/ypt4pf/runs/40003/test-results/0c4b2400-e352-49e2-90f6-fdc91506aa9e
  2. xhr log has response body/status code when xhr response is logged second in proxy_logging.cy.ts https://dashboard.cypress.io/projects/ypt4pf/runs/40003/test-results/7d0e5b2a-c299-4ebb-8478-fdfbfcb39ef2
  3. xhr log has response body/status code when xhr response is logged first in proxy_logging.cy.ts https://dashboard.cypress.io/projects/ypt4pf/runs/40003/test-results/d4c78b51-b525-462d-a844-a4803d263799
     • seems to pass rarely
  4. verifies fetch requests made while a secondary origin is active eventually update with snapshots of the secondary origin in origin/snapshots.cy.ts
     • seems to pass rarely
  5. sends Studio usage metadata in record_spec https://app.circleci.com/pipelines/github/cypress-io/cypress/44422/workflows/25bf1f35-9741-4f30-b29c-6ec11cf7e89c/jobs/1861142
  6. passes in parallel with group in record_spec https://app.circleci.com/pipelines/github/cypress-io/cypress/44422/workflows/25bf1f35-9741-4f30-b29c-6ec11cf7e89c/jobs/1861142
  7. makes the right command for pnpm / makes the right command for npm in project-setup.cy.ts https://app.circleci.com/pipelines/github/cypress-io/cypress/45902/workflows/14823615-1e9c-48fd-8d1b-b75a314110dc/jobs/1932073
  8. saves the filter when navigating to a spec and back in specs_list_e2e.cy.ts https://app.circleci.com/pipelines/github/cypress-io/cypress/45902/workflows/14823615-1e9c-48fd-8d1b-b75a314110dc/jobs/1932099
  9. passes filtered options to Project#open and sets cli config in cypress_spec.js https://app.circleci.com/pipelines/github/cypress-io/cypress/46741/workflows/a44414d3-e3ab-4614-aeb3-5c041a87400f/jobs/1960772
  10. takes the user to first step of ct setup when switching from app in project-setup.cy.ts https://app.circleci.com/pipelines/github/cypress-io/cypress/46741/workflows/a44414d3-e3ab-4614-aeb3-5c041a87400f/jobs/1960739
  11. shows no icon and file system timestamp for files in specs_list_no_git_repo.cy.ts https://app.circleci.com/pipelines/github/cypress-io/cypress/47574/workflows/c287ce0b-245b-449e-9bab-86a66f0265dd/jobs/1990772

Steps to test

How has the user experience changed?

PR Tasks

• Have tests been added/updated?
• [na] Has the original issue (or this PR, if no issue exists) been tagged with a release in ZenHub? (user-facing changes only)
• [na] Has a PR for user-facing changes been opened in cypress-documentation?
• [na] Have API changes been updated in the type definitions?

[cypress-bot]

Thanks for taking the time to open a PR!

• Create a Draft Pull Request if your PR is not ready for review. Mark the PR as Ready for Review when you're ready for a Cypress team member to review the PR.
• Become familiar with the Code Review Checklist for guidelines on coding standards and what needs to be done before a PR can be merged.

[cypress]

---

Test summary

199 • 0 • 23 • 0 • 4

---

Run details

Project	cypress
Status	Passed
Commit	3a5b016
Started	Dec 20, 2022 1:18 AM
Ended	Dec 20, 2022 1:34 AM
Duration	15:58 💡
OS	Windows 10.0.17763
Browser	Chrome 108

View run in Cypress Dashboard ➡️

---

Flakiness

	cypress\e2e\migration.cy.ts		1
1	Full migration flow for each project > completes journey for migration-e2e-component-default-with-types
	cypress\e2e\scaffold-component-testing.cy.ts		3
1	scaffolding component testing > create-react-app > scaffolds component testing for Create React App v5 project
2	scaffolding component testing > angular-cli-unconfigured > scaffolds component testing for Angular
3	scaffolding component testing > svelte-webpack-unconfigured > Scaffolds component testing for Svelte using webpack

---

This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Dashboard

[rachelruderman]

Thank you for implementing this!!! Definitely preferable to boost retries where possible than to skip altogether. I see some CI is failing, is that expected? Ideally the changes would keep CI green (unless there's unrelated CI failures happening)

[flotwig]

Ideally the changes would keep CI green (unless there's unrelated CI failures happening)

@rachelruderman it seems to be flake, but not flake that was touched by these changes

[rachelruderman]

Sounds good!! Thank you for taking this on 🙏

[flotwig]

Had to re-skip some of the retries, there's a full list in the OP comment. I think this is now at a good spot where it's not going to bring any flake/failures back.