Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: Fix Sanity Test 1 image paths #5600

Merged
merged 3 commits into from
Mar 29, 2023
Merged

Conversation

dcalhoun
Copy link
Member

@dcalhoun dcalhoun commented Mar 28, 2023

  • Fix Sanity Test 1 image paths. Previously, the test description was updated, but the tests were not re-run to generate new file paths.
  • Nest Columns slider test beneath Sanity Test 1 description. The lack of nesting was an oversight. The rest of the tests are nested.
  • Remove decreased failure threshold for sliders test. This caused false failures. It appears the test is able to identify true failures with the default 0.1% failure threshold

To test: n/a

PR submission checklist:

  • I have considered adding unit tests where possible.
  • I have considered if this change warrants user-facing release notes more info and have added them to RELEASE-NOTES.txt if necessary.

Previously, the test description was updated, but the tests were not
re-run to generate new file paths.
The lack of nesting was an oversight. The rest of the tests are nested.
@dcalhoun dcalhoun added the Testing Anything related to automated tests label Mar 28, 2023
@dcalhoun dcalhoun self-assigned this Mar 28, 2023
This caused false failures. It appears the test is able to identify true
failures with the default 0.1% failure threshold.
@peril-wordpress-mobile
Copy link

Wanna run full suite of Android and iOS UI tests? Click here and 'Approve' CI job!

@dcalhoun dcalhoun marked this pull request as ready for review March 28, 2023 15:01
@dcalhoun dcalhoun enabled auto-merge March 28, 2023 15:01
@derekblank derekblank self-requested a review March 29, 2023 02:07
Copy link
Contributor

@derekblank derekblank left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! 🚀

@dcalhoun dcalhoun merged commit b112cae into trunk Mar 29, 2023
@dcalhoun dcalhoun deleted the test/fix-sanity-test-1-image-paths branch March 29, 2023 02:07
@fluiddot
Copy link
Contributor

fluiddot commented Mar 29, 2023

Seems the E2E tests related to Sanity Test 1 are failing in trunk after this PR was merged (reference). I noticed the optional UI tests weren't triggered in the last commit of this PR, since the Sanity Test suite 1 is only run in the Test iOS/Android on Device - Full job, it would be great to run it to check for potential failures.

I'll restore the branch and run them as a last check.

@fluiddot fluiddot restored the test/fix-sanity-test-1-image-paths branch March 29, 2023 08:22
@fluiddot
Copy link
Contributor

Seems the E2E tests related to Sanity Test 1 are failing in trunk after this PR was merged (reference). I noticed the optional UI tests weren't triggered in the last commit of this PR, since the Sanity Test suite 1 is only run in the Test iOS/Android on Device - Full job, it would be great to run it to check for potential failures.

I'll restore the branch and run them as a last check.

As it happened in the trunk commit, the E2E tests failed when triggering the full tests via gutenberg-mobile/Optional UI Tests (reference). In any case, this issue has been fixed in #5606 🎊 .

@fluiddot fluiddot deleted the test/fix-sanity-test-1-image-paths branch March 29, 2023 09:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Testing Anything related to automated tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants