Integrate GitHub Actions CI/CD tests

[webknjaz]

Demo: https://github.com/webknjaz/setuptools/runs/209318165 (in fork because it won't show up in upstream w/o merge)

[techtonik]

Does it work offline? Or it is solely for vendor lock-in?

[pganssle]

@techtonik That seems a bit of an unfair standard to judge this contribution by, considering none of our other CI providers have configuration files that work offline or are open source.

Also, I may be misunderstanding your concern - are you suggesting that @webknjaz is adding this to try and lock us in to a specific vendor, or that he's adding it to help us avoid vendor lock-in?

[techtonik]

It is possible to Travis CI tests this way - https://stackoverflow.com/questions/21053657/how-to-run-travis-ci-locally and I guess GitLab CI can also be self-hosted, so I just wonder if there is some advancement in defining CI tasks with this new technology.

I'd prefer open source projects promote open source development practices that don't depend on proprietary platforms, but that's just my opinion.

[webknjaz]

it is solely for vendor lock-in

GitHub has promised to release an open-source runner (I guess that'll happen after Nov 13) for their workflows which you'd be able to plug into your repos BYON-style.

It is possible to Travis CI tests this way

Those images are out of date by a few years. And Travis CI doesn't use docker anymore. The only way to truly debug in their env today is to email them and request SSH feature.

This contribution is to (1) demo the new thing and (2) probably spread the load between CIs + the result statuses are more precisely shown in PRs. Currently, I'm thinking that it should be quite good for fast linters in particular.
Also, it's useful to test against multiple platforms to hit more corner cases. I've tried a number of projects and in most of them, I've found failing tests (hence windows factor in the matrix commented out). So it's useful to cover more cases.
I'm not advocating to completely replacing existing stuff, just showing additional resources.

It's also useful for doing GitHub-related automations (you can react to any event, post comments back etc.).

[techtonik]

FWIW, Travis CI images are updated regularly - check this out - https://hub.docker.com/u/travisci

[webknjaz]

Maybe smth's changed but those aren't VMs they are actually running.

[webknjaz]

@pganssle you may have to hit "Enable" in the pop-up @ https://github.com/pypa/setuptools/actions

[techtonik]

Maybe smth's changed but those aren't VMs they are actually running.

Right, containers are not VMs.

[webknjaz]

@pganssle ready to give it a try?

[pganssle]

I don't really have time to look into this right now, maybe one of the other maintainers can take a look? @pypa/setuptools-developers

I am not 100% clear on whether this adds more CI runs to each PR or if it's running on a schedule, like a cron job or something. I remember last time I tried to hook up travis or something as a github app instead of its built-in version, I didn't really care for the UI.

Do you have any examples of repos that already set up something like this so we can see how it will change the PR workflow?

[jaraco]

I'm pleased that the recipe isn't too complicated and is fairly intuitively read. If @webknjaz can assert that this change doesn't affect the PR workflow or at least doesn't block the current PR workflow, then I'm happy to proceed. If it does affect the PR workflow, I'd like to ensure that it doesn't dramatically impact either the reliability (flakiness) of the tests or the total run time. I'd be okay to accept it if @webknjaz could provide a snapshot analysis of the total run time of a PR check run before and after accepting this change.

[webknjaz]

@pganssle

I am not 100% clear on whether this adds more CI runs to each PR or if it's running on a schedule

All 3: PR, push and cron.

Do you have any examples of repos that already set up something like this so we can see how it will change the PR workflow?

Scroll to the Checks section on this page: aio-libs/aiohttp#4056.
As for the workflow: no change. You may opt-in to requiring checks in the branch protection settings, but it doesn't block anything by default. It's just another status report in the UI for you.

@jaraco unless you enforce this, it won't block PRs. Also, GitHub runs up to 20 jobs in parallel and in this demo in my fork you can see that each job takes 3-4 mins to complete: https://github.com/webknjaz/setuptools/runs/209318165. So it's reasonable to expect that all jobs from a workflow would be finished in 4 minutes after pushing the code.

[webknjaz]

It's important to note that by default any job failure will cause all other jobs in the workflow to be canceled. If you don't want that, apply this patch:

Suggested change

	strategy:
	strategy:
	fail-fast: false

[hugovk]

I've found this useful, especially if there's a network-related timeout in one job.

[webknjaz]

@jaraco is this enough?

[jaraco]

Yes, let's give it a go. We can dial it back or roll it back entirely if it causes too much friction.

[webknjaz]

@jaraco mind merging then?

[webknjaz]

@techtonik as per your earlier question — here's some refs about the open-source runner:

• https://github.com/actions/runner
• https://help.github.com/en/actions/automating-your-workflow-with-github-actions/hosting-your-own-runners

[jaraco]

I'm not sure why I didn't merge this before. Here it goes.