OIDC exchange support

[woodruffw]

This changeset adds support for OIDC "token exchange," an authentication technique that PyPI (and TestPyPI, and maybe some future others) support as an alternative to long-lived username/password combinations or API tokens.

PyPI's documentation: https://pypi.org/help/#openid-connect

OIDC token exchange boils down to the following set of steps:

1. A user (currently only someone in the OIDC beta on PyPI) configured a particular GitHub Actions workflow in their repository as a trusted OIDC publisher;
2. That workflow uses this action to mint an OIDC token;
3. That OIDC token is sent to PyPI (or another index), which exchanges it for a temporary API token;
4. That API token is used as normal.

---

WIP.

This is non-functional in its current state, since the corresponding functionality on PyPI's side hasn't been enabled yet.

Some TODOs:

• Handle repository_url correctly;
• Figure out a better error message propagation technique: the exchange script current just spits things out to stderr;

[webknjaz]

@woodruffw FYI I may merge #122 at some point which would cause conflicts. I hope you don't mind.

[woodruffw]

@woodruffw FYI I may merge #122 at some point which would cause conflicts. I hope you don't mind.

Not at all, thanks for the heads up!

[woodruffw]

For repro purposes, here's how I generated the hashed requirements file:

docker build -t gh-action-pypi-publish .
docker run --rm -it -v $(pwd):/app --entrypoint /bin/bash gh-action-pypi-publish 

Then, in the container:

pip install pip-tools
pip-compile --allow-unsafe --output-file=requirements/runtime.txt --resolver=backtracking --strip-extras requirements/runtime.in

[woodruffw]

I think this is ready for a full review!

[webknjaz]

@woodruffw here are some thoughts. Also, could you rebase the PR to pull in the latest code? It'll also help to clear the pre-commit failure.

P.S. Most of the time, I use true merge for PRs, so this is your chance to clean up the commits if you want.

[webknjaz]

Question: should this update the README too? If we don't want to expose the OIDC details, we could at least link the beta repo for folks who enroll to get back to, right?

[woodruffw]

Question: should this update the README too? If we don't want to expose the OIDC details, we could at least link the beta repo for folks who enroll to get back to, right?

I'm happy to update the README too! IMO there's no problem with doing so, as long as we accurately note that this is currently in beta and won't work for ordinary users yet.

[webknjaz]

Note to myself: a part of me wants to showcase using something like step-security/harden-runner@v2 here, but I understand that this would make the example overloaded and would distract the readers. This probably deserves its own README section with a few snippets...

[webknjaz]

@woodruffw I think this is ready. Do you want to shuffle the commits around before the merge?
Also, could you update the PR description with some pointers for future code archeologists? This would help me write a nice changelog message too.

[woodruffw]

Do you want to shuffle the commits around before the merge?

Yep, I'll squash down again.

Also, could you update the PR description with some pointers for future code archeologists? This would help me write a nice changelog message too.

Sure thing!

Edit: Done.

[webknjaz]

@woodruffw we'll probably need to contribute an article next to https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services when this is out of beta.

[woodruffw]

@woodruffw we'll probably need to contribute an article next to https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services when this is out of beta.

Agreed! I'll chat with some of the GitHub folks about adding that.

[webknjaz]

Agreed! I'll chat with some of the GitHub folks about adding that.

Here's where the content source is located: https://github.com/github/docs/tree/main/content/actions/deployment/security-hardening-your-deployments.

Other things to update:

• Python CI/CD doc: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python#publishing-to-package-registries
• Publishing starter workflow https://github.com/actions/starter-workflows/blob/main/ci/python-publish.yml (selectable and able to be integrated in repos in a single click)

[webknjaz]

@woodruffw apparently, this caused a regression because PyPI itself has a special-cased upload URL which I forgot about — #130.

[webknjaz]

(fixed in v1.8.1)

[woodruffw]

Good to know, thanks for fixing and sorry for the regression 🙂

[webknjaz]

No worries, it was my oversight...

[di]

Other things to update:

• Python CI/CD doc: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python#publishing-to-package-registries
• Publishing starter workflow https://github.com/actions/starter-workflows/blob/main/ci/python-publish.yml (selectable and able to be integrated in repos in a single click)

Doesn't look like either of these have happened yet! Should we open a new issue to capture that here? (cc @jhutchings1 as well)

[webknjaz]

Should we open a new issue to capture that here?

Yes, please! I actually have a browser tab somewhere in hopes to get to the starter workflows. Best to make an issue, you're right! And I totally forgot about that doc. I know that William managed to push updates to some other page @ GH docs but not this one..

[webknjaz]

And it's a bit annoying as there's a ton of repos pinning the action to a very old version because the starter workflows suggest that hash and they don't couple that with dependabot: https://github.com/pypa/gh-action-pypi-publish/network/dependents?package_id=UGFja2FnZS0yOTQyNTU2OTQw

[jhutchings1]

And it's a bit annoying as there's a ton of repos pinning the action to a very old version because the starter workflows suggest that hash and they don't couple that with dependabot: https://github.com/pypa/gh-action-pypi-publish/network/dependents?package_id=UGFja2FnZS0yOTQyNTU2OTQw

Dependabot version updates can be configured to update these, but that's something the repository owners would need to opt-in to receiving by checking in a dependabot.yml file, unfortunately. I'm not in love with hash pinning either, but I know it's our recommended best practice

[webknjaz]

@jhutchings1 I know. The problem is that when people click on "create me a workflow from that template" GitHub doesn't suggest them to also enable dependabot so they have no idea, unless they have pre-existing experience with how all of this works.

The hashes aren't exactly a problem here. The starter workflow still suggests people an old way with a single job and API tokens instead of a more seamless mechanism that is OIDC.

[jhutchings1]

@webknjaz Agreed, @di and I were discussing how to get the starter workflow updated to support OIDC today. I'm here to help, so just let me know if you run into any trouble there that I can help with.

[webknjaz]

@jhutchings1 I'd be happy with https://packaging.python.org/en/latest/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/ copy-pasted there. There's an entire workflow collapsed at the end for copying.