Python package index upload API spec

[brainwane]

As a new twine maintainer I've been running into questions like:

• Now that Warehouse doesn't use register anymore, can we deprecate it from distutils, setuptools, and twine? Are any other package indexes or upload tools using it? (make register error (with PyPI) more informative twine#311)
• it would be nice if twine could depend on a package index providing an HTTP 201 response in response to a successful upload, and fail on 200 (a response some non-package index servers will give to an arbitrary POST request).

I do not see specifications to guide me here, e.g., in the official guidance on hosting one's own package index. PEP 301 was long enough ago that it's due an update, and PEP 503 only concerns browsing and download, not upload.

I suggest that I write a PEP specifying an API for uploading to a Python package index. This PEP would partially supersede PEP 301 and would document the Warehouse reference implementation. I would write it in collaboration with the Warehouse maintainers who will develop the reference implementation per pypi/warehouse/issues/284 , and consulting with the maintainers of packaging and distribution tools such as zest.releaser, flit, devpi, pypiserver, etc.

I believe my steps would be:

1. gather feedback here for about a week or until conversation dies down
2. run this idea past distutils-sig and maintainers of related tools for about a week or until conversation dies down
3. start a PEP draft, submit as a PR to the python/peps repo and circulate on distutils-sig for comment
4. discuss with others at the packaging sprints in May
5. finalize PEP and get PEP accepted by BDFL-Delegate
6. coordinate with PyPA, maintainers of distutils, maintainers of packaging and distribution tools, and documentation maintainers to implement PEP compliance

Thoughts are welcome.

[hpk42]

sounds good -- maybe it makes sense to also have a twine-branch and a warehouse branch or so, to showcase the interactions? We could then maybe also add a devpi-server/devpi-client branch. I guess that "compliant" uploaders could set a special header, requesting the more refined behaviour. Not sure it's needed but throwing around the idea.

[fschulze]

Currently devpi still needs the register call

[di]

I would love it if the upload API was well-specified.

[ncoghlan]

I think there are two slightly different questions to consider here:

1. Documenting what the current upload API between twine & warehouse actually is, similar to the way PEP 503 focused on describing the status quo, without making any changes to it. That way, other servers (like devpi) and other upload clients have the info they need to help ensure interoperability.
2. Adding some backwards compatible refinements to the API to help improve aspects of the user experience (e.g. the PEP 503 amendment to add the data-requires-python link attribute).

While I'm +1 on both of those ideas, I think it's worth separating the steps such that documentation of the existing API doesn't get held up by discussion of possible changes to that API.

[brainwane]

Conversation died down but I was busy :) so I will start a distutils-sig thread.

(I should note that there are a ton of codebases available out there for private package indices, cf. https://stackoverflow.com/questions/1235331/how-to-roll-my-own-pypi . I'll make an effort to reach out to the ones I can find.)

[brainwane]

Posted: https://mail.python.org/mm3/archives/list/[email protected]/thread/WEPTF7Q7475UA7VVULDLIG3A445WOCLI/

[brainwane]

I don't have time to work on this and hope someone else picks it up.

[brainwane]

@dstufft laid out some ideas in pypi/warehouse#7730 that would interest anyone who's also interested in this issue.