twine should fail more gracefully when encountering a metadata version it doesn't support

[Julian]

(It's hard to provide a full reproduce here, so if any of the below isn't clear enough, apologies, but it's a first shot:)

I'm uploading a package (a wheel specifically) generated in the "standard way" with new setuptools and using twine upload, but twine dies by simply saying:

⊙  /Users/Julian/Development/Magnetic/Floodgate/.tox/pypy/bin/pypy -m twine upload --repository-url http://127.0.0.1:53385 --username floodgate-unittest-user --password 'some password' /var/folders/kh/mscfrt910qb_qb8wqrz7t_7w0000gn/T/tmp9TRY2p/floodgate_unittest-0.1.0-py2-none-any.whl                                                               Julian@Macnetic
Uploading distributions to http://127.0.0.1:53385
TypeError: must be convertible to a buffer, not None

Stepping through with a debugger, it looks like what's failing is that metadata.name is None in PackageFile.from_filename, which pkg_resources.safe_str can't cope with (because it regexes it for some reason).

(So that sounds like possibly one improvement, that None shouldn't make it all the way down there)

The way it gets to be None seems to be that Distribution.parse calls _getHeaderAttrs to look up a callback to use for extracting metadata for a particular setuptools version, but if the version isn't present, that function returns an empty list, and no metadata is parsed (so name is None).

Seems like another nice improvement would be failing loudly if the setuptools metadata version isn't supported.

[Julian]

Er, and to be clear, I've found 7f86514, so gonna try to see whether this works with that commit.

[di]

Yes, it seems like you've built a Metadata 2.1 distribution, but are using an old version of twine/pkginfo that doesn't know about Metadata 2.1.

[sigmavirus24]

@di which versions of pkginfo support Metadata 2.1? We can do a fast release raising the minimum version requirement for that dependency.

[di]

@sigmavirus24

twine/setup.py

Line 23 in fa55c0c

"pkginfo >= 1.4.2",

[goodtune]

Also note that some package index implementations do not support receiving Metadata 2.1 packaged uploads - an example is the current version of Artifactory (see https://www.jfrog.com/jira/browse/RTFACT-16189 tracking the issue).

In this case there is a response from the server but all the user gets as notification is the HTTP response code of 400, despite there being a useful server response revealing the issue.

Current output...

HTTPError: 400 Client Error: Bad Request for url: https://artifactory.example.com/artifactory/api/pypi/internal-repo

The useful information that is discarded is...

{
  "errors" : [ {
    "status" : 400,
    "message" : "No enum constant org.jfrog.repomd.pypi.model.PypiMetadata.MetadataVersion.v2_1"
  } ]
}

I'm running a forked version which does a resp.raise_for_status() and outputs the body when it encounters an issue as a work-around until it's fixed upstream, but I thought I'd mention it here for anyone else who may be encountering it.

[djpugh]

the resp.raise_for_status() and output body is really useful - and I see it has been updated in master, when is this likely to be released?

[ljohnston]

It seems recent versions of Artifactory no longer choke on 2.1 metadata packaging (though a warning is still issued in the artifactory logs). That said, I can exactly reproduce this issue as described when using twine to upload to our Artifactory:

HTTPError: 400 Client Error: Bad Request for url:

I CAN, however, upload the exact same package using curl without any issues.

[gatesn]

@ljohnston I hit the same thing. If you build your sdist / wheel using setuptools < 38.6.0 it should work fine.

http://setuptools.readthedocs.io/en/latest/history.html#v38-6-0

[pstephenson02]

@gatesn I'm experiencing the same issue. I tried downgrading to setuptools 38.5.2, rebuilding my sdist and wheels and for some reason I still see the same error. I verified I'm getting the exact same response from Aritfactory by pulling master on this repo and using the --verbose flag. Any ideas?

[gatesn]

If you use —verbose you should see more than just the 400 error. What’s the reason that’s printed out?

[pstephenson02]

I'm seeing the exact same response that @goodtune mentioned above:

{
  "errors" : [ {
    "status" : 400,
    "message" : "No enum constant org.jfrog.repomd.pypi.model.PypiMetadata.MetadataVersion.v2_1"
  } ]
}

[minghuijiang]

You will need to pin setuptools==35.5.2 and wheel==0.30.0 for the workaround until artifactory support 2.1.

[CCardosoDev]

For me I only had to pin wheel==0.30.0 the latest setuptools (which is 40.0.0 at this moment) worked just fine.

[mat013]

For those who needs to verify whether it is the same issue you need to patch utils.py in the method check_status_code with something similar like this

print(response._content)
response.raise_for_status()

[CNG]

wheel==0.30.0 was also all I needed for Artifactory. Thanks!

[florent-vial]

You will need to pin setuptools==35.5.2 and wheel==0.30.0 for the workaround until artifactory support 2.1.

At the time of writing 35.5.2 is not available. In the proposed list i see 35.0.2 is working for me (together with wheels==0.30.0)

[Rocking80]

This issue is not resolved yet?

[stevenmanton]

This worked for me: pip install setuptools==35.0.2 wheel==0.30.0