fix(testIsolation): improve the behavior, clarify config options and sync with session command #24316
Conversation
|
Thanks for taking the time to open a PR!
|
Co-authored-by: Matt Henkes <[email protected]>
There was a problem hiding this comment.
Looks really well tested! Main questions I have are around the comments in cypress.d.ts and wonder if we want to reflect similar language in the Docs PR?
cli/types/cypress.d.ts
Outdated
| * | ||
| * NOTES: | ||
| * - Turning test isolation off may improve performance of end-to-end tests, however, previous tests could impact the | ||
| * browser state of the next test and cause inconsistency when using .only(). |
There was a problem hiding this comment.
This could also have an impact if the user changes the order of their tests. I think what we are trying to convey is that tests are not guaranteed idempotency when run by themselves and I am not sure that is obvious.
There was a problem hiding this comment.
This could also have an impact if the user changes the order of their tests.
This would depend on how the user has written their E2E tests -- you could write basic, repeatable tests that don't impact the state of the next test and could be ran standalone. Though most people may accidentally write tests that do rely on one another. If you / others feel strongly, I'd prefer to elaborate more in about the risks in the documentation vs the Typescript description....It might be worth removing this entire blur and adding an onlink to learn more info.
There was a problem hiding this comment.
I think that is pretty clear here with what you added. I wonder if we want to generalize this more as to your point, this is a TypeScript declaration file and not docs 😄 . Maybe something like:
Turning test isolation off may improve performance of end-to-end tests, however, previous tests may not be idempotent. @see <LINK_TO_DOCS>
which would give the clear description of what is meant here when we mention idempotent, since its probably an appropriate word to capture what we mean here, but not suitable for an in depth explanation which the docs provide.
| .then((win) => { | ||
| win.cookie = 'key=value; SameSite=Strict; Secure; Path=/fixtures' | ||
| win.localStorage.setItem('animal', 'bear') | ||
| win.sessionStorage.setItem('food', 'burgers') |
There was a problem hiding this comment.
I wonder if we need a similar test check if document.cookie and the cookie-jar are cleared between tests when going back to cy.origin?
|
@AtofStryker I tried to match the language to the docs changes while also make it comprehensive in the "what is test isolation" blurb. I'll take a look through your comments, but if you could make notes on the docs PR as well where you feel these diverge, that'd be helpful. I've been thinking/talking about this for so long it all makes sense to me 😄 |
@emilyrohrbough after taking a look, I think the docs capture that context very well. I think some of the updates proposed to the |
Co-authored-by: Bill Glesias <[email protected]>
End to end testing
experimentalSessionAndOrigin=false|testIsolation | beforeEach test | cy.session ||----------------|---------------------------------------|------------------------------------||legacy | clear cookies & local storage in current domain | not available |experimentalSessionAndOrigin=trueComponent Testing
No real impact on Component testing. When sessions & origin goes GA, session storage will also be cleared between tests. Currently only the page, cookies and local storage are cleared.
User facing changelog
The behavior and implementation of the experimental
stricttest isolation mode has been evaluated per user feedback.strictmode has been replaced test isolationonandoffmodes, withonmode being the default behavior observed whenexperimentalSessionAndOrigin=trueand thecy.session()command now inherits the test isolation behavior for the suite it runs in. When enabled,legacymode is not longer a valid mode and to persist the browser context between testsoffmode can be used.PR Tasks
cypress-documentation? update documentations around test isolation cypress-documentation#4797type definitions?