Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
This PR makes that all
cucumber-js
steps that contain "I should see" take a screenshot AFTER the step is done.The problem with screenshot-based testing is that you may get very small pixel differences between baseline.
This is caused by UI elements like a spinner indicating a loading (the difference could just be the spinner at a different rotation)
The rationale for only taking screenshots on "I should see" tests is that these tests should BLOcK on UI elements and "see" a stable view of the UI after the blocking is done
This is exactly what we want to take a screenshot of to compare with.
Changes made to
cucumber
libraryUnfortunately Cucumber doesn't let you add a hook for every step (which is what we need)
To fix this, I did the following:
cucumber-js
to 5.1.0 (changelog here)cucumber-js
into my own repocucumber-js
(notably,lib
folder was required (otherwise in .gitignore)Future work
You can see for example in my PR, one of our tests said "I should see" but in fact it had no blocking UI call at all (it only checked the value of some variable)
Therefore, to effectively do screenshot-based testing, one would have to
This PR doesn't make screenshot testing as part of our CI but that is the ultimate goal.
Ideally, our CI would cache the screenshot tests for specific commits. Then, when you make a PR, it checks at which commit you forked from
develop
and compare your PR to that (Note: you CANNOT compare to currentdevelop
as too many things may have changed. You have to compare to where the branch forked)Save Location
Note: Screenshots folder is deleted every time you run a new test
Example
Test definition
Run test
Created files
Screenshot generated