-
Notifications
You must be signed in to change notification settings - Fork 195
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Split tests into four groups for CI job matrices #872
Conversation
Codecov Report
@@ Coverage Diff @@
## master #872 +/- ##
==========================================
- Coverage 72.50% 71.09% -1.41%
==========================================
Files 187 187
Lines 5331 5217 -114
==========================================
- Hits 3865 3709 -156
- Misses 1466 1508 +42
Continue to review full report at Codecov.
|
Alright I think our CI is back online: Travis, GitLab, Appveyor, and Docker CI all work again. And I think CodeCov should be more lenient about failing PRs now so we won't get red crosses on most PRs anymore. Tests will take longer to run now though. |
Found a trick to configure Travis CI to build pull requests but only pushes to master (thus reducing redundancy as default Travis CI behavior is to build both pushes and merge commits on PRs). |
Successful in 65m... |
This PR splits the tests into four test groups, selectable with the
TEST_GROUP
environment variable:unit
,integration
,regression
, andscripts
. UsingTEST_GROUP=all
(the default ifTEST_GROUP
is not defined) will run all tests.The purpose of this PR is to address long test build times (see #860) that time out at 50 minutes on Travis CI and at 60 minutes on GitLab CI by splitting tests into multiple jobs (i.e. building a job matrix) that each should individually run much faster.
If the different jobs could be run in parallel this would speed up our test builds significantly, but alas we are stuck with free-tier CI pipelines so we can't run too many jobs in parallel and will probably end up waiting longer. But at least our jobs won't time out.
In working on this PR I was also able to revive the Appveyor and Docker CI pipelines (I think Appveyor will only show up for future PRs).
Resolves #139