Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Render test passes even though actual image is different from expected. #4862

Open
NathanMOlson opened this issue Oct 20, 2024 · 5 comments · May be fixed by #4864
Open

Render test passes even though actual image is different from expected. #4862

NathanMOlson opened this issue Oct 20, 2024 · 5 comments · May be fixed by #4864
Labels
benchmark Related to testing performance bug Something isn't working PR is more than welcomed Extra attention is needed

Comments

@NathanMOlson
Copy link
Contributor

The terrain/fog render test passes, even though the actual image is different from the expected image.

Expected:
expected

Actual:
actual

Actual (drawn over a red background to demonstrate the transparent "hole" near the horizon):
actual_red_background

Computed Diff:
diff
@NathanMOlson NathanMOlson mentioned this issue Oct 20, 2024
Open
@HarelM
Copy link
Collaborator

HarelM commented Oct 20, 2024

The render test are using an external library to compare images: pixelmatch with some configuration.
The configuration is there to prevent false positive test failures.
Feel free to investigate what's causing this...

@HarelM HarelM added bug Something isn't working PR is more than welcomed Extra attention is needed benchmark Related to testing performance labels Oct 20, 2024
@NathanMOlson
Copy link
Contributor Author

Feel free to investigate what's causing this...

The root cause of this bug is that pixelmatch does not detect any difference between white and transparent pixels: mapbox/pixelmatch#26

@HarelM
Copy link
Collaborator

HarelM commented Oct 20, 2024

Ahh, in that case we should make sure either not to use white or use a solid non-white background.
Another approach we can try is contribute the fix to picelmatch, but I don't have high hopes for this route.

@NathanMOlson NathanMOlson linked a pull request Oct 20, 2024 that will close this issue
Open
3 tasks
@NathanMOlson
Copy link
Contributor Author

I submitted a PR to pixelmatch to fix this issue: mapbox/pixelmatch#142. I think it would be a good idea to incorporate this fix, as it may affect other render tests in our test suite.

@HarelM
Copy link
Collaborator

HarelM commented Oct 21, 2024

Worst case we can patch-package it I guess.
Let's wait a few days and see where this lands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
benchmark Related to testing performance bug Something isn't working PR is more than welcomed Extra attention is needed
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix render test false negative NathanMOlson/maplibre-gl-js
2 participants
@HarelM @NathanMOlson