Not obvious how to use multiple project API tokens with keyring

[bhrutledge]

Your Environment

1. Your operating system: macOS

2. Version of python you are running: 3.7.6

3. How did you install twine? pipx

4. Version of twine you have installed (include complete output of):

twine version 3.1.1 (pkginfo: 1.5.0.1, requests: 2.22.0, setuptools: 45.1.0,
requests-toolbelt: 0.9.1, tqdm: 4.42.0)

5. Which package repository are you targeting? pypi and testpypi

The Issue

Twine only uses the repository URL and username to retrieve the credentials, so the standard use of keyring doesn't seem support multiple project API tokens.

$ keyring set https://upload.pypi.org/legacy/ __token__
Password for '__token__' in 'https://upload.pypi.org/legacy/': 

Possible Workaround

Adding the package name as a query parameter seems to work, but feels like a hack:

$ keyring set https://upload.pypi.org/legacy/?example-pkg __token__
Password for '__token__' in 'https://upload.pypi.org/legacy/?example-pkg': 

$ twine upload \
    --repository-url https://upload.pypi.org/legacy/?example-pkg \
    --username __token__ \
    dist/*

Or:

$ cat ~/.pypirc
[distutils]
index-servers =
    pypi
    example-pkg

[pypi]
username = __token__

[example-pkg]
repository = https://upload.pypi.org/legacy/?example-pkg
username = __token__

$ twine upload --repository example-pkg dist/*

Possible Solutions

Off the top of my head, without adding an additional argument to keyring:

• Add the repository name to be part of the keyring USERNAME argument, e.g.:

  keyring set https://upload.pypi.org/legacy/ __token__:example-pkg
  

  I think this might be relatively quick to implement, but feels clunky to document and use.

• Use the repository name as the keyring SERVICE argument e.g.:

  keyring set example-pkg __token__
  keyring set pypi __token__
  

  This feels friendlier to users. However, I'm guessing it requires more substantial changes in twine's configuration handling.

[bhrutledge]

This feels related to the proposals in #216 and #324 to add a command to streamline twine's handling of credentials.

[bhrutledge]

The discussion around this issue seems to be evolving from pypa/packaging.python.org#297 (comment). I think this issue is blocked until that's resolved.

[8day]

Judging by the message shown after user have created project-scoped API token

To use this API token:

- Set your username to __token__
- Set your password to the token value, including the pypi- prefix

For example, if you are using Twine to upload multiple projects to PyPI, you can set up your $HOME/.pypirc file like this:

	[distutils]
	  index-servers =
		testpypi
		PROJECT_NAME

	[testpypi]
	  username = __token__
	  password = # either a user-scoped token or a project-scoped token you want to set as the default

	[PROJECT_NAME]
	  repository = https://test.pypi.org/legacy/
	  username = __token__
	  password = # a project token 

You can then use twine --repository PROJECT_NAME to switch to the correct token when uploading to PyPI.

For further instructions on how to use this token, visit the PyPI help page.

it looks like this issue was kind of solved behind the scenes, but it seems to be broken (pure .pypirc-based solution, w/o CLI, seems to work though): if password is missing from .pypirc, twine will use keyring get {repository} {username}, instead of keyring get {repository} {PROJECT_NAME} (if there are multiple index servers with same URL and they must be accessed using API token, then it doesn't matter how many of them there are -- both URL and username will be identical for all of them). If this were to be carried to CLI, things would become unintelligable: twine --repository {id_for_keyring} --repository-url {repository_url} --user __token__. Also this would require lots of modifications so that twine.repository.Repository() stored name of the repository.

The following are my thoughts on the issue (tl;dr: use __token__:{some_pc_wide_unique_id}, as shown in first post).

To upload package to index server, using classic case, we must know its URL, username and password. To store password in a secure storage, we can map URL and username to password: both index servers and their usernames are unique, therefore they will produce reliable mapping. When we want to upload package to index server using API token, here too we must know its URL, username and password. Because in case of login through API token username is always constant, namely __token__, we can't map URL and a username to password, unless index server has only one user. Thus, we must provide some variable, alternative to username. .pypirc format dictates that the name of index server, a.k.a. repository, was to be used as such "alternative variable", but ATM twine uses URL&username for password lookup no matter what, therefore, despite the amount of defined index servers, it will use keyring get {repository_url} __token__ to retrieve passwords. Another problem is that if twine were to use name of index server (PROJECT_NAME in .pypirc documentation) as an alternative to username to retrieve password, then CLI commands would look unintelligible: twine --repository {replacement_for_username} --repository-url https://some.page --user __token__ (.pypirc config doesn't look much better; note that value passed to --repository would have to be passed around in some weird, confusing way, decreasing quality of the codebase).

A better solution could be to add extra key to .pypirc index servers -- token_name, which will store name of the token, which will then be used along with URL (repository) to retrieve actual token from some secure storage (keyring). Because there can be only one unique API token name per index server, like in case with username, this solution is quite intuitive, as well as it makes sense to use actual name of API token stored on [Test]PyPI, which will make things even more intuitive. As for CLI, there will have to be a new key to accept such token name -- -t/--token-name. When -t/--token-name will be set, it will be required that username was set either manually or by default only to __token__, otherwise exception must be raised.

The thing is, because -u and -t will be effectively mutually exclusive (-u/-t __token__ is the only case where it's not true and they act identical), it may seem that there is some way to reuse -u/username, and there is -- use specially formatted value like {token_prefix}{separator}{token_name} (separator should be used for clarity). While not necessary, __token__ used as a token_prefix will allow to make separator and token_name optional in case of mono-token use, make .pypirc less complicated, with index server configs used as intended, as well as preserve backward compatibility. As a cherry on top, all it'd take to implement this feature is to change one line in twine.settings.Settings.create_repository()

repo = repository.Repository(
	cast(str, self.repository_config["repository"]),
	self.username,
	self.password,
	self.disable_progress_bar,
)

to

repo = repository.Repository(
	cast(str, self.repository_config["repository"]),
	"__token__" if self.username.startswith("__token__:") else self.username,
	self.password,
	self.disable_progress_bar,
)

And of course documentation will have to be updated as well, not to mention doc for .pypirc and info at https://test.pypi.org/manage/account/token/.

BTW, apart from twine upload, twine.settings.Settings.create_repository() is used only by twine register, which requires proper, "classic" credentials anyway, therefore it's OK to use this solution. Of course, it's possible that it may be better to modify twine.settings.Settings.create_repository() to accept custom username, which will be passed to it by twine.upload.upload()...

---

Edit: From the looks of it, someone used one of the solutions from the first post w/o making sure that it was possible to have project-specific URLs, thus breaking current multi-token solution.

[bhrutledge]

@8day thanks for sharing your thoughts. I haven't read them closely, but a quick scan suggests that you and I have arrived at similar conclusions. A few quick notes (which you may have covered already):

• I'm reluctant to add additional configuration options to Twine; the precedent of __token__ being a valid username is well-established at this point.
• I don't want to ask for changes to keyring, because it feels unnecessary and complex to manage.
• As noted in a previous comment, there's a lot of discussion around this at Update distributing guide to prefer twine's keyring support, and discourage use of pypirc packaging.python.org#297 (comment), though it stalled.

Since opening this issue over 2 years ago, I haven't seen any demand for a fix. So, it's not a high priority, esp. given the unresolved design decisons.

[peterjc]

There is going to be demand now, with PyPI pushing 2FA, then sending emails like this:

---------- Forwarded message ---------
From: PyPI [email protected]
Date: Fri, Jul 15, 2022 at 4:15 PM
Subject: [PyPI] Migrate to API tokens for uploading to PyPI
To: ...

What?
During your recent upload or upload attempt to PyPI, we noticed you used basic authentication (username & password). However, your account has two-factor authentication (2FA) enabled.

In the near future, PyPI will begin prohibiting uploads using basic authentication for accounts with two-factor authentication enabled. Instead, we will require API tokens to be used.

What should I do?
First, generate an API token for your account or project at https://pypi.org/manage/account/token/. Then, use this token when publishing instead of your username and password. See https://pypi.org/help/#apitoken for help using API tokens to publish.

[sigmavirus24]

Maybe that demand will help generate solutions instead of comments with subtext guilting maintainers for not prioritizing something that hasn't been necessary that contribute little to the discussion

[peterjc]

You're reading too much into it Ian, no slight was intended. My comment was intended as a direct reply to Brian's final paragraph:

Since opening this issue over 2 years ago, I haven't seen any demand for a fix. So, it's not a high priority, esp. given the unresolved design decisons.

Out of pragmatism, in the short term I'll just use a single token for all my PyPI projects, since that is documented and clear. Thank you.

[jaraco]

Does the existence of trusted publishing reduce the demand/desire for this feature?

[sigmavirus24]

Does the existence of trusted publishing reduce the demand/desire for this feature?

To some degree, sure it probably does. That said, PyPI isn't the only target people try to upload to (and I've heard of some other corporate indexes trying to mimic this behavior for some bizarre reason). For another, I'm certain there are projects which can not use GitHub/GitHub Actions (or GitLab which I think is one of the other trusted publishers) to publish things. Some folks self-host Jenkins, etc but still release to PyPI and likely need some way to securely manage tokens (although I doubt they want keyring for that).

Beyond that, I'd expect some people just don't trust GitHub to publish things and would feel better doing so themselves with a token. I don't blame them as people put a lot of trust in a company that's all too often not done what's best for the community since it works to serve shareholders goals.

So, yes, I think the demand is lessened but not by a measurable quantity (only because the original demand was not measurable).