chore: make feature.spec and segements.spec more resilient #7289
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
1 Ignored Deployment
|
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.OpenSSF Scorecard
Scanned Manifest Files |
thomasheartman
commented
Jun 5, 2024
frontend/src/component/project/NewProjectCard/ProjectOwners/ProjectOwners.tsx
Outdated
Show resolved
Hide resolved
thomasheartman
changed the title
thomasheartman
changed the title
chriswk
approved these changes
Jun 6, 2024
|
Appears to at least mitigate some of the issues. There's still errors that can pop up, but I ran it four times in a row without getting any errors. Got an error on the fifth run, but that was a new error, so that's progress for sure. Also, four successful runs in a row vs 24 runs to make everything pass sounds like a net improvement to me. |
thomasheartman
added a commit
that referenced
this pull request
Jun 12, 2024
…messages This change adds a wait statement before entering the name of the segment when checking for error messages that this segment name already exists. This is the same workaround that we did in #7289, which seems to have worked. Like in that PR, using waits is still an antipattern, but it appears to be working.
thomasheartman
added a commit
that referenced
this pull request
Jun 12, 2024
…messages (#7377) This change adds a wait statement before entering the name of the segment when checking for error messages that this segment name already exists. This is the same workaround that we did in #7289, which seems to have worked. Like in that PR, using waits is still an antipattern, but it appears to be working. Maybe it’s time to look more deeply at why it happens? Why isn’t the field ready to receive input even though it’s on the page? Is it mui’s fault or ours?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR attempts to make the feature.spec and segements.spec test suites more reliable. They have been flaking out a lot recently, and this will hopefully make them less flaky.
The way of handling it is a little different for each test suite.
feature.spec
Some of the failures we're seeing for the feature/feauture.spec test suite are due to uncaught resize observer issues (possibly triggered by the banners).
We can ignore these errors as they don't impact functionality, only rendering, and are likely to resolve themselves quickly in real-world scenarios.
On the other hand, it might also ignore actual errors, so I'm not a 100% on this. Would love some input.
However, MDN has some info on observation errors:
Based on that, I think this is a pretty safe error to ignore.
I'm unsure whether catching this exception is only set in the
aftercleanup or whether it pollutes the cy object for all tests, but I think it's fine either way. But if you have ideas, I'd love to hear them.segments.spec
The issue here appears to be that when we first input the segment's name in the form, it takes a little time for the UI to become ready, so the first characters of the string are cut off.
This is a known issue that the cypress team are aware, but that isn't likely to get fixed any time soon because no one can give them a reproducible example.
You can see the effect of this on segments that haven't been cleaned up in the preview:
To work around it, we add a 500ms wait before we start filling out the form. Yes, adding waits in your tests is an antipattern, but it's the easiest way around in this case.
We could investigate and find a way not to need that, but that would likely be a much larger project. This appears to mitigate the issue immediately, so is at least a pretty good temporary fix in my opinion. We also already do this in other tests, so there is a precedent for it.