Single-use API token to create new package

[Zac-HD]

I'm currently at the PyCon Australia sprints, in the 2FA beta, and a big fan of narrowly scoped API tokens. But when I went to make the first ever release of hypothesmith I had a problem: there's no way to scope an API token when the package doesn't exist yet!

My workaround was to create a token that works for all repos, upload the package, then revoke the token (and finally create a scoped token).

I and others here would prefer to have an option for a single-use and time limited token that can only be used to upload a new package, and is automatically revoked immediately on first use.

[brainwane]

Thanks for the feature request, @Zac-HD! This is a great idea in my opinion. Unfortunately I believe it is out of scope for the current grant-funded project (for more on our current grants, see the Packaging WG wiki page), as are other caveat limitations (see #6255).

So I'll be talking with @ewdurbin and other maintainers to figure out whether (a) I'm wrong about it being a great idea :) and (b) whether we can find funding to design and implement this in the future.

Thanks again!

[brainwane]

This use case is specific and makes a lot of sense! But. Project creation is a rare event in general for PyPI, so we want to be careful about what we engineer for and what common vs. uncommon cases we make easier in terms of UX.

A few different ways we could implement this:

• as a caveat, perhaps along with the other caveats suggested in Add caveats to macaroons for expiration (time) and version #6255 -- a one-time user-scoped token
• via package preview, Draft release feature on main archive to allow testing a release before it goes live #726, so the user can stage a project, then scope a token to it
• allowing users to create a token scoped arbitrarily - type any project name! (This would require that we ensure an auth check, so using such a project name doesn't actually allow the user to upload to an existing project they don't have permissions for.) And, ideally, make it a one-time token so that using it trashes the token along the way (maybe using caveats)

[rachelcipkins]

This would probably be best to implement as a caveat. I can work on adding it.

[brainwane]

@woodruffw mentions some questions for discussion ("The semantics around one-time tokens and project version tokens") - @Zac-HD and @di you may be interested in commenting there.

[webknjaz]

@brainwane may I suggest another idea:

• Create a token that has the only privilege: create a new project (i.e. for the first upload). It'd create a project under the user it's bound to.
• On the back-end, PyPI would re-purpose the token to be bound to the project created. And it'd be impossible to create any new projects with this token, just use it as a normal project-scoped token.
• This would emulate the current use of passwords and re-enable the CI flows that allowed putting in creds for the PyPI upload into the CI/CD system before the project is created and then the project creation is done by the CI, not a human (hence no manual step on the dev machine).

[Zac-HD]

Ooh, that's really nice!

[ewjoachim]

I'm a bit late to the party, but just wanted you to know: it's actually already possible to generate a token scoped to any project, including one that doesn't exist yet, and the easiest way of doing that is using the pypitoken library.

import pypitoken
token = pypitoken.Token.load("pypi-foo")  # a token with "user" privileges
token.restrict(projects=["your-new-project-normalized-name"])
token.dump()  # returns your new token

[ewjoachim]

Does this mean the ticket should be closed ? Or those this ticket relate to implementing this in the Warehouse UI so that you don't have to use a python lib to do it ?

[di]

I think we may want to leave this open to think about a more user-friendly way of doing this. E.g., should this be baked into twine?

[webknjaz]

I don't think that workaround solves the issue fully. While it's possible to generate a restricted token this way, I'd rather see it in the UI among other tokens that are bound to the projects. My tokens have the location of where they are used in their names (CI name + project name etc) so I can easily revoke them. The derivative tokens add a lot of mental overhead of tracking where they come from, what is the "parent token", I should always remember that revoking a specific parent token will affect all the derived ones and there's no easy way of tracking this and preventing accidental revocations that only come up on use in surprising places.
This is why I think that having a Warehouse-native feature with good traceability is imperative.

[ewjoachim]

This makes sense, but as an individual user, you can commit to throw away the parent token when you generate a restricted one and to never use a token for two different things. The description field provides space for you to be explicit on how the token will be used.

I agree though that warehouse currently stores the caveat it applied and displays the information, and this is something you won't have if you apply the caveat yourself.

(I've added a paragraph in the docs that summarize your ideas, which I find very interesting)

I think we may want to leave this open to think about a more user-friendly way of doing this. E.g., should this be baked into twine?

Well putting this in twine would be great, but then it's not really linked to new packages anymore, it would work the same for new and existing packages. But yeah, I'd love it if we could restrict all aspects of a token just before uploading (project, version, filename, hash), so that even a MITM wouldn't be able to do much.

[ewjoachim]

Looks like twine integration is not likely to happen soon: pypa/twine#749 (well the issue could turn around at some point, but I don't think it will)

[woodruffw]

xref #11296, which I think should address this use case: users will be able to request a temporary project reservation on PyPI, which in turn will allow them to create an API token that'll only be valid for as long as that reservation unless they upload a package within the reservation window (at which point it becomes an ordinary project).

[woodruffw]

Circling back: the OIDC publishing functionality has been merged, and is currently in a closed beta: #12965

[di]

I think given our upcoming support for this via OIDC-based publishing, and upcoming support for that feature in https://github.com/pypa/gh-action-pypi-publish, I think this is likely the way we are going to recommend creating new projects going forward for users that want to avoid username/password -based auth (with the known caveat that this is only applicable to projects that release from GitHub).

Given that, I'll probably close this once that feature is launched, but happy to have any discussion here if anyone is opposed.

[woodruffw]

I believe this is fully covered, now that trusted publishing is fully released and includes support for "pending" publishers!

[di]

Agreed -- and we're actively working on adding other identity providers, e.g. #13551.

Closing!

[webknjaz]

FWIW I think that this feature would still be useful for less popular CIs, or those that are custom, running behind a firework, on premise etc.