CI/tests: CI integration tests on Edge and windows #1621
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
peaBerberian
force-pushed
the
misc/try-other-browsers
branch
5 times, most recently
from
8c7abe0 to
3e4d33d
Compare
peaBerberian
changed the title
peaBerberian
added
work-in-progress
peaBerberian
changed the title
peaBerberian
force-pushed
the
misc/try-other-browsers
branch
10 times, most recently
from
dfa0c7c to
393cfaa
Compare
Until now, CI/npm scripts only runs our integration tests on Chrome and Firefox. This is a proposal to also run them on Edge and Safari. There have been very few browser-dependent issues found with integration tests for now but it may still be useful and wasn't too hard to do. Testing on both edge and Safari also allows to run a lot of MS-specific and Webkit-specific workarounds which may allow to better detect browser-specific regressions. I tested it and it seemed to work. Issues with this is that we have more chance to be annoyed by flaky tests (which would be better fixed by updating the tests anyway) and that CI steps would take longer. We may want to split CI scripts per-browser for that last part? To see.
peaBerberian
force-pushed
the
misc/try-other-browsers
branch
from
393cfaa to
ffae398
Compare
peaBerberian
removed
the
work-in-progress
peaBerberian
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Listing things I've done:
I've added a script for integration tests on the Edge browser, which I also run on Windows in the CI
I explicitly call
bashandnodein package.json'sscriptsto be compatible with windows (cmd?).bashis technically already a dependency anyway due to some of our scriptsI updated Github's
actions/checkoutandactions/setup-nodein the CI'schecks.ymlconfiguration file, as new examples from GitHub seems to indicate those should be the versions used today (differences seems to be things like versions of dependencies like node.js)I updated scripts's
path.joinusage as I thought it was provoking an issue on windows but turned out it was not. I still left it the way I updated it, we could put it like it was before, I don't really careI've raised MANY tests timeout and reduced the playback rates. Turns out that the machine/VMs that GitHub actions run is not performant at all, often reaching timeouts and often not being able to play as fast as we ask it
Due to that last point, tests take much more time though.
Things I didn't succeed to do:
I didn't succeed running tests on Safari, only because I didn't find a way anywhere to authorize autoplay without user interaction. Other browsers provide flags for this, but it seems that Safari doesn't.
Muting the audio volume on the corresponding video element didn't seem to have an effect.
I tried for a very long time to run tests on MacOS, but the performance of GitHub actions are so low that our corresponding integration tests were beginning to loose in quality. I chose to remove the macos part instead.