Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallelize tests run on GitHub Actions #425

Merged
merged 1 commit into from
May 20, 2022

Conversation

lread
Copy link
Collaborator

@lread lread commented May 20, 2022

In addition to being broken down by OS, tests are now also broken down by
major type (ide, api, unit) and browser (chrome, firefox, edge, safari)
for ide and api tests.

Observations:

  • can now more readily see what we are testing by the job breakdown in
    GitHub Actions UI
  • nice to have an overall shorter test run. I'm seeing ~8m to completion instead
    of the former ~15m
  • faster feedback for what is passing/failing
  • some windows tests might be a bit flaky, but good to know, so that we can
    address, right?
  • very nice to be able to rerun a single failed job, for example, if
    unit windows has failed, I can rerun it alone via the GitHub UI.

The current tests only invoke chrome and firefox for ide testing, so
I've stuck with that. Api testing matches current browser defaults.

I'm liking keeping the brunt of the work out of yaml and in babashka
tasks. The yaml has a setup job which:

  • brings down Clojure deps to be cached
  • learns what tests should be run via
    bb script/test.clj matrix-for-ci --format=json

The build task then:

  • runs the tests found by the setup task. Test runs on ubuntu include
    launching a virtual display (necessary for GitHub Actions)

Of note:

  • New custom test reporting spits out the browser (for api and ide
    tests) and the name of the test being run. This feedback is nice
    especially for api tests which are long-running. It also helps to
    validate we are testing what we think we are testing.
  • Added ^:unit metadata to unit test namespaces
  • I noticed GitHub Actions was passing failing Windows tests and learned
    how powershell needs a little help in propagating return codes.
  • Because browser sets are different for api and ide tests, I left
    the existing ETAOIN_TEST_DRIVERS env var for api tests and introduced
    ETAOIN_IDE_TEST_DRIVERS for ide tests.
  • Brought in some more scripting helpers from other projects, refactored
    existing bb tools-versions script accordingly

Closes #420

In addition to being broken down by OS, tests are now also broken down by
major type (ide, api, unit) and browser (chrome, firefox, edge, safari)
for ide and api tests.

Observations:
- can now more readily see what we are testing by the job breakdown in
GitHub Actions UI
- nice to have an overall shorter test run. I'm seeing ~8m to completion instead
of the former ~15m
- faster feedback for what is passing/failing
- some windows tests might be a bit flaky, but good to know, so that we can
address, right?
- very nice to be able to rerun a single failed job, for example, if
`unit windows` has failed, I can rerun it alone via the GitHub UI.

The current tests only invoke chrome and firefox for ide testing, so
I've stuck with that. Api testing matches current browser defaults.

I'm liking keeping the brunt of the work out of yaml and in babashka
tasks. The yaml has a setup job which:
- brings down Clojure deps to be cached
- learns what tests should be run via
`bb script/test.clj matrix-for-ci --format=json`

The build task then:
- runs the tests found by the setup task. Test runs on ubuntu include
launching a virtual display (necessary for GitHub Actions)

Of note:
- New custom test reporting spits out the browser (for api and ide
tests) and the name of the test being run. This feedback is nice
especially for api tests which are long-running. It also helps to
validate we are testing what we think we are testing.
- Added `^:unit` metadata to unit test namespaces
- I noticed GitHub Actions was passing failing Windows tests and learned
how powershell needs a little help in propagating return codes.
- Because browser sets are different for api and ide tests, I left
the existing `ETAOIN_TEST_DRIVERS` env var for api tests and introduced
`ETAOIN_IDE_TEST_DRIVERS` for ide tests.
- Brought in some more scripting helpers from other projects, refactored
existing bb tools-versions script accordingly

Closes clj-commons#420
@lread lread requested review from borkdude and slipset as code owners May 20, 2022 19:11
@borkdude
Copy link
Contributor

Huge improvement!

@lread lread merged commit 1e0c80f into clj-commons:master May 20, 2022
@lread
Copy link
Collaborator Author

lread commented May 20, 2022

Thanks, @borkdude! I especially enjoyed enumerating the matrix jobs to run on GitHub Actions from a babashka task.

@borkdude
Copy link
Contributor

@lread Haha, wow, I missed that, nice.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Parallelize Browser Testing on CI
2 participants