Switch CI to use Github Actions, run tests in parallel and fast fail when possible #1234

jakebolewski · 2020-06-11T00:11:47Z

Run subset on every push in parallel
Kill jobs that fail quickly
Kill stale jobs quickly upon new commits to a branch
Run all OS unit tests only on bors try / bors merge
Remove Azure Devops Pipelines

Description

A clear and concise description of the code with usage.

I have

Written and run all necessary tests with CLIMA by including tests/runtests.jl
Followed all necessary style guidelines and run julia .dev/climaformat.jl .
Updated the documentation to reflect changes from this PR.

For review by CLIMA developers

There are no open pull requests for this already
CLIMA developers with relevant expertise have been assigned to review this submission
The code conforms to the style guidelines and has consistent naming conventions. julia .dev/format.jl has been run in a separate commit.
This code does what it is technically intended to do (all numerics make sense physically and/or computationally)

charleskawczynski · 2020-06-11T00:35:15Z

Looks awesome. We still need to configure bors dependence, right?

slurmci-test.toml

.github/workflows/Linux-UnitTests.yml

jkozdon · 2020-06-11T02:42:15Z

Looks awesome. We still need to configure bors dependence, right?

Do we need each github action job to have it's own name to work with bors?

See this discussion

Alternatively we can rename the jobs this way

jakebolewski · 2020-06-11T16:43:44Z

I haven't configured bors yet as I didn't know how it worked with the status indicator. It seem like you can reference the job name so we'll just have to make those two unique instead of both being named test?

jakebolewski · 2020-06-11T16:44:38Z

The other behavior would be to decide if we only want to run the tests for PR's and not for every push. I think this might be better default, if you need to test some behavior out you can create a draft / WIP PR.

charleskawczynski · 2020-06-11T16:57:02Z

I haven't configured bors yet as I didn't know how it worked with the status indicator.

From what I tested here, the syntax of the status name depends on whether we're using a matrix job or not.

The other behavior would be to decide if we only want to run the tests for PR's and not for every push. I think this might be better default, if you need to test some behavior out you can create a draft / WIP PR.

IMO, every push seems unnecessary. Often I find myself doing what @kpamnany does (canceling docs/azure after pushing to not waste resources). It'd be really nice to have a way to trigger incremental tests. Could mergify somehow help?

jakebolewski · 2020-06-11T17:01:27Z

if you push now, then its configured to cancel any previous jobs on that branch.

jakebolewski · 2020-06-11T17:04:48Z

I'm guessing that is the desired behavior, automatically kill stale jobs only build the latest commit on every branch.

I guess my question is if these jobs should only be triggered for PR's and not every branch that get's pushed.

jakebolewski · 2020-06-11T17:05:18Z

bors try

bors · 2020-06-11T17:06:15Z

try

Build failed:

docs-build

jakebolewski · 2020-06-11T17:13:20Z

bors try

jakebolewski · 2020-06-11T17:38:21Z

bors try-

kpamnany · 2020-06-11T17:43:46Z

I guess my question is if these jobs should only be triggered for PR's and not every branch that get's pushed.

Currently, CI is not triggered for branch pushes and only runs when you open a PR for that branch. I think this behavior is fine.

kpamnany · 2020-06-11T17:45:10Z

I think what Charlie was getting at was if pushes to PRs should always trigger full CI. If we can just do a Linux CI build for pushes to a PR branch and do the full CI on bors try or bors r+ that would perhaps be a good thing?

jakebolewski · 2020-06-11T18:15:32Z

bors try

bors · 2020-06-11T18:46:07Z

try

Build failed:

ci/slurmci

jakebolewski · 2020-06-11T19:21:54Z

bors try

jakebolewski · 2020-06-12T13:19:58Z

I think this is ready to go, it has / had the behavior that @kpamnany that you were describing :). I put a reasonable time limit on the jobs as well.

jakebolewski · 2020-06-12T14:56:21Z

@charleskawczynski I can add some documentation, where would be the best place to do so? (wiki, docs, etc.?)

kpamnany

So excited for this!

charleskawczynski · 2020-06-12T15:16:02Z

@charleskawczynski I can add some documentation, where would be the best place to do so? (wiki, docs, etc.?)

Good question. The badge you can add to the README, this is what I've done in the past for a GHA badge. We could also add a blurb of how this works to the continuous integration section of the contributing docs. But wiki sounds good too. But I don't have strong feelings as to where it belongs (I do like the idea of the badges on the front page though!).

jakebolewski · 2020-06-12T15:30:29Z

Ok I added some docs describing the current CI checks to Contributing / continuous_integration section.

jakebolewski · 2020-06-12T16:32:20Z

bors r+

jakebolewski · 2020-06-12T16:33:50Z

lets see how this goes, I'm sure with some experience we'll need to make some tweaks to the config so don't hesitate to raise issues

1234: Switch CI to use Github Actions, run tests in parallel and fast fail when possible r=jakebolewski a=jakebolewski * Run subset on every push in parallel * Kill jobs that fail quickly * Kill stale jobs quickly upon new commits to a branch * Run all OS unit tests only on bors try / bors merge * Remove Azure Devops Pipelines # Description A clear and concise description of the code with usage. Co-authored-by: jakebolewski <[email protected]>

simonbyrne · 2020-06-12T18:03:28Z

.github/workflows/Linux-UnitTests.yml

+          test-modules: ["Arrays,Atmos,Common,Diagnostics,Driver,InputOutput,Utilities",
+                         "Numerics/DGMethods",
+                         "Numerics/Mesh",
+                         "Numerics/ODESolvers",
+                         "Numerics/SystemSolvers",
+                         "Ocean"]


this is super neat!

bors · 2020-06-12T19:22:43Z

try

Timed out.

jakebolewski · 2020-06-12T19:28:22Z

hmm all checks passed but ittlooks like bors was misconfigured to check the status of the github action matrix? It seems stuck in a running state.

@simonbyrne / @charleskawczynski any ideas?

jkozdon · 2020-06-12T19:36:10Z

bors.toml

@@ -1,5 +1,5 @@
 status = [
-  "azure-ci",
+  "test-os",


Suggested change

"test-os",

"test-os (ubuntu-latest)",

"test-os (windows-latest)",

"test-os (macos-latest)",

I think it reads the names list on the left side of https://github.com/CliMA/ClimateMachine.jl/runs/766357298

bors · 2020-06-12T19:41:17Z

Canceled.

jakebolewski · 2020-06-12T19:41:35Z

bors r+

1199: Added single stack horizontal statistics and test r=charleskawczynski a=ilopezgp Added horizontal statistics diagnostics for single stack driver and a single stack tutorial of the 3D Burgers equation. Modified plothelpers.jl to admit the xlabel as a keyword argument. # Description A clear and concise description of the code with usage. 1232: Add simple hydrostatic spin-down test for Ocean Model r=blallen a=blallen # Description This PR adds a simple test of the hydrostatic dynamics of the Ocean model. It is intended to be used as the basic test case when implementing the split-explicit time stepper. The analytic solution (and more) can be found in this overleaf: https://www.overleaf.com/project/5e9dfd2716ba2600018de266 Currently, I'm just testing that the solution is within 0.5% of the analytic solution after a day. If anyone has better ideas please suggest them. I'd also like to turn this into a Literature tutorial, so I don't think I want to go the full convergence testing route with this, altho it is possible. 1234: Switch CI to use Github Actions, run tests in parallel and fast fail when possible r=jakebolewski a=jakebolewski * Run subset on every push in parallel * Kill jobs that fail quickly * Kill stale jobs quickly upon new commits to a branch * Run all OS unit tests only on bors try / bors merge * Remove Azure Devops Pipelines # Description A clear and concise description of the code with usage. Co-authored-by: Ignacio <[email protected]> Co-authored-by: Brandon Allen <[email protected]> Co-authored-by: jakebolewski <[email protected]>

bors · 2020-06-12T19:56:08Z

Build failed (retrying...):

test-os (ubuntu-latest)
test-os (windows-latest)
test-os (macos-latest)

1232: Add simple hydrostatic spin-down test for Ocean Model r=blallen a=blallen # Description This PR adds a simple test of the hydrostatic dynamics of the Ocean model. It is intended to be used as the basic test case when implementing the split-explicit time stepper. The analytic solution (and more) can be found in this overleaf: https://www.overleaf.com/project/5e9dfd2716ba2600018de266 Currently, I'm just testing that the solution is within 0.5% of the analytic solution after a day. If anyone has better ideas please suggest them. I'd also like to turn this into a Literature tutorial, so I don't think I want to go the full convergence testing route with this, altho it is possible. 1234: Switch CI to use Github Actions, run tests in parallel and fast fail when possible r=jakebolewski a=jakebolewski * Run subset on every push in parallel * Kill jobs that fail quickly * Kill stale jobs quickly upon new commits to a branch * Run all OS unit tests only on bors try / bors merge * Remove Azure Devops Pipelines # Description A clear and concise description of the code with usage. Co-authored-by: Brandon Allen <[email protected]> Co-authored-by: jakebolewski <[email protected]>

bors · 2020-06-13T00:47:24Z

Build failed (retrying...):

ci/slurmci

* Run subset on every push in parallel * Kill jobs that fail quickly * Kill stale jobs quickly upon new commits to a branch * Run all OS unit tests only on bors try / bors merge * Remove Azure Devops Pipelines

bors · 2020-06-13T00:54:15Z

Canceled.

jakebolewski · 2020-06-13T00:54:31Z

bors r+

bors · 2020-06-13T02:38:29Z

Build succeeded:

jkozdon reviewed Jun 11, 2020

View reviewed changes

slurmci-test.toml Show resolved Hide resolved

jkozdon reviewed Jun 11, 2020

View reviewed changes

slurmci-test.toml Outdated Show resolved Hide resolved

jkozdon reviewed Jun 11, 2020

View reviewed changes

.github/workflows/Linux-UnitTests.yml Outdated Show resolved Hide resolved

bors bot added a commit that referenced this pull request Jun 11, 2020

Try #1234:

51b69db

bors bot added a commit that referenced this pull request Jun 11, 2020

Try #1234:

18517dd

bors bot added a commit that referenced this pull request Jun 11, 2020

Try #1234:

9ad6cc0

jakebolewski changed the title ~~WIP: Switch CI to use Github Actions, run tests in parallel and fast fail when possible~~ Switch CI to use Github Actions, run tests in parallel and fast fail when possible Jun 11, 2020

bors bot added a commit that referenced this pull request Jun 11, 2020

Try #1234:

952c89a

jakebolewski force-pushed the jcb/github_actions branch 3 times, most recently from b5686a8 to 06a54f9 Compare June 12, 2020 13:18

kpamnany added the CI Continuous Integration label Jun 12, 2020

jakebolewski requested a review from charleskawczynski June 12, 2020 14:24

kpamnany approved these changes Jun 12, 2020

View reviewed changes

jakebolewski force-pushed the jcb/github_actions branch 2 times, most recently from 5e34baf to b979b49 Compare June 12, 2020 15:26

simonbyrne reviewed Jun 12, 2020

View reviewed changes

jkozdon reviewed Jun 12, 2020

View reviewed changes

jakebolewski force-pushed the jcb/github_actions branch from b979b49 to d3cb2da Compare June 12, 2020 19:41

Switch CI to use Github Actions

c69ff9f

* Run subset on every push in parallel * Kill jobs that fail quickly * Kill stale jobs quickly upon new commits to a branch * Run all OS unit tests only on bors try / bors merge * Remove Azure Devops Pipelines

jakebolewski force-pushed the jcb/github_actions branch from d3cb2da to c69ff9f Compare June 13, 2020 00:54

bors bot merged commit cef6389 into master Jun 13, 2020

bors bot deleted the jcb/github_actions branch June 13, 2020 02:38

This was referenced Jun 13, 2020

Revamp test suite, version 2 #888

Closed

Prototype of test suite revamp #858

Closed

charleskawczynski mentioned this pull request Jun 15, 2020

Create RunTests action #848

Closed

4 tasks

-  "test-os",
+  "test-os (ubuntu-latest)",
+  "test-os (windows-latest)",
+  "test-os (macos-latest)",

Switch CI to use Github Actions, run tests in parallel and fast fail when possible #1234

Switch CI to use Github Actions, run tests in parallel and fast fail when possible #1234

Conversation

jakebolewski commented Jun 11, 2020

Description

For review by CLIMA developers

charleskawczynski commented Jun 11, 2020

jkozdon commented Jun 11, 2020

jakebolewski commented Jun 11, 2020

jakebolewski commented Jun 11, 2020

charleskawczynski commented Jun 11, 2020

jakebolewski commented Jun 11, 2020

jakebolewski commented Jun 11, 2020

jakebolewski commented Jun 11, 2020

bors bot commented Jun 11, 2020

try

jakebolewski commented Jun 11, 2020

jakebolewski commented Jun 11, 2020

kpamnany commented Jun 11, 2020

kpamnany commented Jun 11, 2020

jakebolewski commented Jun 11, 2020

bors bot commented Jun 11, 2020

try

jakebolewski commented Jun 11, 2020

jakebolewski commented Jun 12, 2020

jakebolewski commented Jun 12, 2020

kpamnany left a comment

Choose a reason for hiding this comment

charleskawczynski commented Jun 12, 2020

jakebolewski commented Jun 12, 2020

jakebolewski commented Jun 12, 2020

jakebolewski commented Jun 12, 2020

simonbyrne Jun 12, 2020

Choose a reason for hiding this comment

bors bot commented Jun 12, 2020

try

jakebolewski commented Jun 12, 2020 • edited Loading

jkozdon Jun 12, 2020

Choose a reason for hiding this comment

jkozdon Jun 12, 2020

Choose a reason for hiding this comment

bors bot commented Jun 12, 2020

jakebolewski commented Jun 12, 2020

bors bot commented Jun 12, 2020

bors bot commented Jun 13, 2020

bors bot commented Jun 13, 2020

jakebolewski commented Jun 13, 2020

bors bot commented Jun 13, 2020

jakebolewski commented Jun 12, 2020 •

edited

Loading