Switch headless screenshot error to MSE, add benchmarking #16038
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hrydgard
approved these changes
Sep 17, 2022
|
Looks good, buildbot has some complaints though, should be easy to fix |
To be able to add additional options.
This is more useful than 1/0 on whether a pixel doesn't match, when many are close but not exact.
This makes it easier to allow a certain amount of deviation across devices.
This outputs the average time for comparison.
unknownbrackets
force-pushed
the
headless
branch
from
db24cc3
to
749268c
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This allows you to run a frame dump or test to get an average time taken, which can be convenient for comparing the relative performance of two changes.
It also returns screenshot differences as mean squared error instead of the previous simple % of pixels that are not exact matches. I considered PSNR, but I feel like the output is less obvious for what a "pass" means. There's also an option to tolerate a certain amount of MSE, which can be helpful when comparing things that might have small differences (i.e. hardware rendering.)
-[Unknown]