Fix flaky widgets-related E2E tests #33066
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
adamziel
added
[Block] Legacy Widget
|
Size Change: +150 B (0%) Total Size: 1.05 MB
ℹ️ View Unchanged
|
|
Maybe we should group "marquee" tests with the |
|
I just saw some more failures, but different ones than without this PR – hopefully these are just flaky tests, I am restarting the workflows and let's see. |
draganescu
approved these changes
Jun 29, 2021
ockham
reviewed
Jun 29, 2021
|
|
||
| await addMarquee(); | ||
| it( 'Should add and save the marquee widget', async () => { | ||
| await activatePlugin( 'gutenberg-test-marquee-widget' ); |
There was a problem hiding this comment.
How about moving this into a newly created beforeEach() (so setup and teardown are symmetrical)? Or better yet maybe, move this one into beforeAll(), and change the afterEach()into anafterAll()`? This would allow adding other tests to the 'Function widgets' group later and would only activate the plugin once at the very beginning, and deactivated it at the very end.
There was a problem hiding this comment.
That's a good note, thank you! I went with beforeEach and afterEach for now, let's see if the tests pass before tweaking that further, I am worried that calling deleteAllWidgets without deactivating and reactivating the plugin could cause the tests to fail.
|
Thanks a lot for tackling this! Best viewed without whitespace: https://github.com/WordPress/gutenberg/pull/33066/files?diff=split&w=1 One small note here: #33066 (comment) 🙂 |
|
All e2e tests are failing with the same error. It might be related to the RC release. |
7 tasks
adamziel
force-pushed
the
fix-widgets-tests
branch
from
14e3c9e to
5b4609b
Compare
|
@adamziel, I think you just removed @kevin940726 latest commit on this branch 🙇 |
|
@Mamaduka ay thanks for bringing it up, I rebased and didn't notice we had more changes here :-) |
adamziel
added
the
Backport to WP 6.7 Beta/RC
desrosj
pushed a commit
that referenced
this pull request
Jul 2, 2021
* Remove empty keys from the compared object * Revert dev artifacts * Disable the gutenberg-test-marquee-widget plugin * Wrap marquee tests in their own describe statement * Lint * Update snapshots * Replace hello with howdy * Move plugin activation to beforeEach * Move deleteAllWidgets to beforeEach * Update tests * use data-testid rather than name attribute selectors * Remove any existing marquees before running the tests, use the "save" form button * Remove dev artifact Co-authored-by: Kai Hao <[email protected]> (cherry picked from commit 5cafe02)
desrosj
pushed a commit
that referenced
this pull request
Jul 2, 2021
* Remove empty keys from the compared object * Revert dev artifacts * Disable the gutenberg-test-marquee-widget plugin * Wrap marquee tests in their own describe statement * Lint * Update snapshots * Replace hello with howdy * Move plugin activation to beforeEach * Move deleteAllWidgets to beforeEach * Update tests * use data-testid rather than name attribute selectors * Remove any existing marquees before running the tests, use the "save" form button * Remove dev artifact Co-authored-by: Kai Hao <[email protected]> (cherry picked from commit 5cafe02)
gziolo
pushed a commit
that referenced
this pull request
Jul 5, 2021
…33174) * Only calculate the compressed size when necessary. (#32161) This updates the compressed size workflow to run only when the files changes would have an impact on the package size. (cherry picked from commit 23d84b3) * Limit when workflows run on forks (#32114) (cherry picked from commit 435dde4) * Improvements to NPM package caching across workflows (#32458) * Use a consistent cache key for NPM packages across all workflows. Also, include the NodeJS version in the cache key in case different package versions are required. * Update the `actions/cache` action to the latest version. * Remove the strategy matrix from jobs with a single NodeJS version. For some reason, GitHub Actions will attach matrix values to job names in parenthesis. Because required checks are configured by job name, these need to stay consistent. (cherry picked from commit 7e54162) * Limit when release artifacts are built on forks: pt. 2 (#32494) * Limit when release artifacts are built on forks. Previously in #32114, adjustments were made to GHA workflow files to limit when workflows ran on forked repositories. But this missed the second job in the workflow that builds the release artifact. Because it was set to always run even though the previous job is required, it has continued to run on forks. * Ensure the artifact is built always, even when the preceding job does not run. (cherry picked from commit 8fbc4d6) * Use a different cache key for the PR automation workflow (#32588) * Use a different cache key for the PR automation workflow. * Update the `chalk` dependency to the latest version. (cherry picked from commit 05ed04a) * Fix flaky widgets-related E2E tests (#33066) * Remove empty keys from the compared object * Revert dev artifacts * Disable the gutenberg-test-marquee-widget plugin * Wrap marquee tests in their own describe statement * Lint * Update snapshots * Replace hello with howdy * Move plugin activation to beforeEach * Move deleteAllWidgets to beforeEach * Update tests * use data-testid rather than name attribute selectors * Remove any existing marquees before running the tests, use the "save" form button * Remove dev artifact Co-authored-by: Kai Hao <[email protected]> (cherry picked from commit 5cafe02) * Navigation: skip flakey tests (#33074) (cherry picked from commit 4d0959e) Co-authored-by: Adam Zielinski <[email protected]> Co-authored-by: Kerry Liu <[email protected]>
youknowriad
pushed a commit
that referenced
this pull request
Jul 5, 2021
* Remove empty keys from the compared object * Revert dev artifacts * Disable the gutenberg-test-marquee-widget plugin * Wrap marquee tests in their own describe statement * Lint * Update snapshots * Replace hello with howdy * Move plugin activation to beforeEach * Move deleteAllWidgets to beforeEach * Update tests * use data-testid rather than name attribute selectors * Remove any existing marquees before running the tests, use the "save" form button * Remove dev artifact Co-authored-by: Kai Hao <[email protected]> (cherry picked from commit 5cafe02)
youknowriad
pushed a commit
that referenced
this pull request
Jul 5, 2021
This reverts commit 305ca72.
youknowriad
pushed a commit
that referenced
this pull request
Jul 5, 2021
…33174) * Only calculate the compressed size when necessary. (#32161) This updates the compressed size workflow to run only when the files changes would have an impact on the package size. (cherry picked from commit 23d84b3) * Limit when workflows run on forks (#32114) (cherry picked from commit 435dde4) * Improvements to NPM package caching across workflows (#32458) * Use a consistent cache key for NPM packages across all workflows. Also, include the NodeJS version in the cache key in case different package versions are required. * Update the `actions/cache` action to the latest version. * Remove the strategy matrix from jobs with a single NodeJS version. For some reason, GitHub Actions will attach matrix values to job names in parenthesis. Because required checks are configured by job name, these need to stay consistent. (cherry picked from commit 7e54162) * Limit when release artifacts are built on forks: pt. 2 (#32494) * Limit when release artifacts are built on forks. Previously in #32114, adjustments were made to GHA workflow files to limit when workflows ran on forked repositories. But this missed the second job in the workflow that builds the release artifact. Because it was set to always run even though the previous job is required, it has continued to run on forks. * Ensure the artifact is built always, even when the preceding job does not run. (cherry picked from commit 8fbc4d6) * Use a different cache key for the PR automation workflow (#32588) * Use a different cache key for the PR automation workflow. * Update the `chalk` dependency to the latest version. (cherry picked from commit 05ed04a) * Fix flaky widgets-related E2E tests (#33066) * Remove empty keys from the compared object * Revert dev artifacts * Disable the gutenberg-test-marquee-widget plugin * Wrap marquee tests in their own describe statement * Lint * Update snapshots * Replace hello with howdy * Move plugin activation to beforeEach * Move deleteAllWidgets to beforeEach * Update tests * use data-testid rather than name attribute selectors * Remove any existing marquees before running the tests, use the "save" form button * Remove dev artifact Co-authored-by: Kai Hao <[email protected]> (cherry picked from commit 5cafe02) * Navigation: skip flakey tests (#33074) (cherry picked from commit 4d0959e) Co-authored-by: Adam Zielinski <[email protected]> Co-authored-by: Kerry Liu <[email protected]>
youknowriad
removed
the
Backport to WP 6.7 Beta/RC
youknowriad
pushed a commit
that referenced
this pull request
Jul 6, 2021
This reverts commit 305ca72.
youknowriad
pushed a commit
that referenced
this pull request
Jul 6, 2021
…33174) * Only calculate the compressed size when necessary. (#32161) This updates the compressed size workflow to run only when the files changes would have an impact on the package size. (cherry picked from commit 23d84b3) * Limit when workflows run on forks (#32114) (cherry picked from commit 435dde4) * Improvements to NPM package caching across workflows (#32458) * Use a consistent cache key for NPM packages across all workflows. Also, include the NodeJS version in the cache key in case different package versions are required. * Update the `actions/cache` action to the latest version. * Remove the strategy matrix from jobs with a single NodeJS version. For some reason, GitHub Actions will attach matrix values to job names in parenthesis. Because required checks are configured by job name, these need to stay consistent. (cherry picked from commit 7e54162) * Limit when release artifacts are built on forks: pt. 2 (#32494) * Limit when release artifacts are built on forks. Previously in #32114, adjustments were made to GHA workflow files to limit when workflows ran on forked repositories. But this missed the second job in the workflow that builds the release artifact. Because it was set to always run even though the previous job is required, it has continued to run on forks. * Ensure the artifact is built always, even when the preceding job does not run. (cherry picked from commit 8fbc4d6) * Use a different cache key for the PR automation workflow (#32588) * Use a different cache key for the PR automation workflow. * Update the `chalk` dependency to the latest version. (cherry picked from commit 05ed04a) * Fix flaky widgets-related E2E tests (#33066) * Remove empty keys from the compared object * Revert dev artifacts * Disable the gutenberg-test-marquee-widget plugin * Wrap marquee tests in their own describe statement * Lint * Update snapshots * Replace hello with howdy * Move plugin activation to beforeEach * Move deleteAllWidgets to beforeEach * Update tests * use data-testid rather than name attribute selectors * Remove any existing marquees before running the tests, use the "save" form button * Remove dev artifact Co-authored-by: Kai Hao <[email protected]> (cherry picked from commit 5cafe02) * Navigation: skip flakey tests (#33074) (cherry picked from commit 4d0959e) Co-authored-by: Adam Zielinski <[email protected]> Co-authored-by: Kerry Liu <[email protected]>
desrosj
pushed a commit
that referenced
this pull request
Jul 7, 2021
* Remove empty keys from the compared object * Revert dev artifacts * Disable the gutenberg-test-marquee-widget plugin * Wrap marquee tests in their own describe statement * Lint * Update snapshots * Replace hello with howdy * Move plugin activation to beforeEach * Move deleteAllWidgets to beforeEach * Update tests * use data-testid rather than name attribute selectors * Remove any existing marquees before running the tests, use the "save" form button * Remove dev artifact Co-authored-by: Kai Hao <[email protected]> (cherry picked from commit 5cafe02)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Partially related to #32978.
The widgets tests fail randomly on the latest trunk version. The reasons for failures are:
wp_inactive_widgetskey in the snapshot of sidebarWidgets data. The culprit is tests execution order – we enable thegutenberg-test-marquee-widgetplugin but we never disable it. With the plugin enabled, thewp_inactive_widgetskey is there, without it it's not there.How has this been tested?
Just confirm the snapshot-based tests pass
Types of changes
Bug fix