Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test(e2e): restore report generation and fix location of received images #3145

Merged
merged 6 commits into from
Aug 21, 2024

Conversation

tbouffard
Copy link
Member

@tbouffard tbouffard commented Aug 16, 2024

When an test failure occurred, the report was not generated and the development server did not stop at the end of the test execution.
This was due to our custom function toMatchImageSnapshot becoming asynchronous in commit bffd25b.
The original function is synchronous, so the custom function must be too.

In addition to this fix, the logic of the code managing image attachments has been improved, mainly to better separate responsibilities.
Storing the expected image copy and calculating the image location are specific to j-i-s, so there's no need to do this. to j-i-s, and are therefore performed first.
The function for attaching images to the report only uses the previously calculated image location and is unaware that j-i-s is being used.

For tests that redefined the location of output images (usually in a subfolder), the path to the received images was not updated accordingly. It was stored in the same location for each test, so only one file was kept and not stored with the diff and the expected image.

Also regenerate the image for the “subprocess.03.collapsed.with.elements” test.

  • Previously, the test failed when run with firefox and this was for good reasons that were hidden with other browsers due to the threshold value used.
  • The sub-process on the top left is a "collapsed adhoc sub-process", so 2 markers must be displayed with a space between them. Currently, the adhoc marker is not displayed, but the expand marker is correctly positioned (not centered, but on the left) as if the adhoc marker were displayed.
  • Commit c0c50aa removed additional spacing between markers, so the expand marker has moved to the right. But the reference snapshot was not updated accordingly.
  • The test probably accepted a minor visual change like this, but at some point the new version of Firefox added other changes that are above the threshold, and the error now appears.
  • The problem didn't occur on macOS because the threshold was increased in commit bfa02fc, but this hid the problem.

Notes

First detection of problems fixed in this PR:

Error reproductions and progressive fix with the "overlays" e2e tests

Command run: DEBUG=bv:test:jest:img npm run test:e2e -- --runTestsByPath test/e2e/overlays.rendering.test.ts

Examples of reports generated with all tests involving "custom received directory"

The report was no longer generated and the development server did not stop at the end of the test execution.
This was due to our custom function `toMatchImageSnapshot` becoming asynchronous in commit bffd25b.
The original function is synchronous, so the custom function must be too.

In addition to this fix, the logic of the code managing image attachments has been improved, mainly to better separate responsibilities.
Storing the expected image copy and calculating the image location are specific to j-i-s, so there's no need to do this. to j-i-s, and are therefore performed first.
The function for attaching images to the report only uses the previously calculated image location and is unaware that j-i-s is being used.

For tests that redefined the location of output images (usually in a subfolder), the path to the received images was not updated accordingly. It was stored in the same location for each test, so only one file was kept and not stored with the diff and the expected image.
  - disable "fit on load" to generate e2e test errors
  - disable panning and mouse zoom to generate e2e test errors
  - reproduce the pb with overlays tests
This reverts commit 8e53f9fcb52b028d168d448316b567b39ddd997e.
@tbouffard tbouffard added the chore Build, CI/CD or repository tasks (issues/PR maintenance, environments, ...) label Aug 16, 2024
@tbouffard tbouffard marked this pull request as ready for review August 16, 2024 11:54
test/config/jest.image.ts Outdated Show resolved Hide resolved
@tbouffard tbouffard marked this pull request as draft August 21, 2024 09:33
@tbouffard
Copy link
Member Author

ℹ️ Converted to draft to take the time to integrate review feedback.

@tbouffard tbouffard marked this pull request as ready for review August 21, 2024 12:02
Copy link

sonarcloud bot commented Aug 21, 2024

@tbouffard tbouffard merged commit 7a9f2b4 into master Aug 21, 2024
21 checks passed
@tbouffard tbouffard deleted the test/fix_e2e_received_custom_dir branch August 21, 2024 12:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
chore Build, CI/CD or repository tasks (issues/PR maintenance, environments, ...)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants