Add tests to GitHub Actions CI/CD workflow

[webknjaz]

Trying to figure out how hard it is to make actions run tests...

[chrahunt]

Thank you!

Related to #7279. A few comments/questions about this change specifically:

1. What CI providers are we trying to displace with these tests?
2. We should understand what impact this has for running tests on multiple PRs. From here it looks like we get up to 20 concurrent jobs and 5 macOS jobs for free as an open-source project. Compared to 30 on Azure Pipelines and I'm not sure about Travis
3. Is there a way to publish and query our test run info? One benefit of Azure Pipelines is the test analysis tools and API available for querying test information for each run, see here for example.

[webknjaz]

@chrahunt

1. This is not trying to displace CIs but rather extend envs coverage. Each CI provider has its own way of setting up VMs which may result in discovering some weird cases reproducible only under one env, for example.
2. This question makes sense if the checks are required by the branch protection rules. Otherwise, it's yet another set of advisory checks.
3. I don't think so. I think it should be possible to emulate test report tables via Checks API but this would require a bit of extra coding.

[pradyunsg]

This question makes sense if the checks are required by the branch protection rules. Otherwise, it's yet another set of advisory checks.

I think @chrahunt meant, what is the effect if multiple PRs are opened at the same time -- how much of a slowdown would this entail?

[webknjaz]

I think @chrahunt meant, what is the effect if multiple PRs are opened at the same time -- how much of a slowdown would this entail?

Ah, not much I think.

Compare these:

• GitHub Actions CI/CD — in this PR (only Test Suite workflow) — 21m 46s — https://github.com/pypa/pip/commit/dc935c48009eeb77e73dc607a615bcef32889f3e/checks?check_suite_id=407979541 (visible via https://github.com/pypa/pip/actions?query=ea96d0a+event%3Apull_request+actor%3Awebknjaz+workflow%3A%22Test+suite%22+branch%3Afeatures/integrate-github-actions-ci-cd--tests)
• Azure Pipelines (Linux) — 18m 50s
• Azure Pipelines (macOS) — 16m 15s
• Azure Pipelines (Windows) — 31m 41s

Since GH workflows run in parallel with Azure, I'd say that they'll finish faster, hence no slow-down. Also, I've set fail-fast: false because it's easier to debug stuff this way. If we remove it, failed tests will abort the whole workflow faster.

[webknjaz]

@chrahunt @pradyunsg how do we advance this now?

[pradyunsg]

@chrahunt @pradyunsg how do we advance this now?

Let's have a discussion in #7279 about what we want our CI situation to look like.

[pradyunsg]

I'm gonna say this is blocked on #7279, since once we know how we want to do CI, we'll pick the relevant parts from this PR. :)

[pradyunsg]

Closing since:

• this PR would need to be updated anyway, whenever we have a resolution in Clarify & document our CI strategy #7279
• this PR's code isn't gonna vanish
• this PR has had no activity since the "before times"