long_description_content_type flag in setup.py seems to be ignored on upload.

[fake-name]

Describe the bug
I have a python package on PyPi that just (as of the last ~4-5 days) now fails to upload, with a new error:

durr@rwpscrape /m/S/S/WebRequest> python3 setup.py sdist upload

[snip build process]

Submitting dist/WebRequest-0.0.28.tar.gz to https://upload.pypi.org/legacy/
Upload failed (400): The description failed to render in the default format of reStructuredText. See https://pypi.org/help/#description-content-type for more information.
error: Upload failed (400): The description failed to render in the default format of reStructuredText. See https://pypi.org/help/#description-content-type for more information.

Ok, that's clear enough. My readme is in markdown, let's see how to fix this.

Going to https://packaging.python.org/tutorials/packaging-projects/#description, it states you can add long_description_content_type="text/markdown", to your setup.py file, and it'll be processed as markdown.

<does edits>....
<submits to PyPi again

Submitting dist/WebRequest-0.0.28.tar.gz to https://upload.pypi.org/legacy/

[snip build process]

Upload failed (400): The description failed to render in the default format of reStructuredText. See https://pypi.org/help/#description-content-type for more information.
error: Upload failed (400): The description failed to render in the default format of reStructuredText. See https://pypi.org/help/#description-content-type for more information.

What?

Expected behavior
I expect if I specify the readme is markdown, as the documentation says to do so, the readme should be treated as markdown.

To Reproduce
Attempt to upload a package with a readme that is valid markdown, but not valid reStructuredText.

My Platform

• Python 3.5.2 x64
• Ubuntu 16.04 LTS
• setuptools-39.2.0 (I specifically checked that it's the most current version)
• readme_renderer 20.0

Actual project in question is here. The readme.md in question is here, and the setup.py is here

[fake-name]

Note. https://pypi.org/help/#description-content-type also claims you can use other formats (and specifically mentions markdown!), but after installing readme_renderer, local checking appears to fail (?):

durr@rwpscrape /m/S/S/WebRequest> python3 setup.py check -r -s
running check
warning: Check: Not checking long description content type 'text/markdown', this command only checks 'text/x-rst'.

error: Please correct your package.
durr@rwpscrape /m/S/S/WebRequest>

Is markdown support a feature that's not actually been implemented yet? Did the documentation get released too early or something?

[jamadden]

You might try using the latest version of twine to handle the upload. Also, more metadata is available when you upload a wheel first, IIRC.

$ python setup.py sdist
$ pip install -U wheel
$ python setup.py bdist_wheel
$ twine upload dist/*whl dist/*gz

[jstaf]

@jamadden Nope, it's definitely broken with the latest twine/uploading a wheel first.

I think the issue is caused by a bug in https://github.com/pypa/readme_renderer - it's exiting with an error when it checks packages with a markdown readme (when it should not):

$ python setup.py check -s -r
running check
warning: Check: Not checking long description content type 'text/markdown', this command only checks 'text/x-rst'.

error: Please correct your package.

[theacodes]

I actually think this is due to #3980, I'll investigate now. Thank you for finding this!

[jstaf]

@theacodes - No worries! And yeah, that does seem more likely: it matches the timeframe of @fake-name 's original issue much more closely.

[theacodes]

@fake-name I can't reproduce this. See https://test.pypi.org/project/WebRequest/. I'm using setuptools 39.2.0 and twine 1.11.0.

If you wouldn't mind uploading your wheel and sdist somewhere else, I'd love to take a look.

[theacodes]

Also I see you tried using python3 setup.py sdist upload, please try with Twine.

[jstaf]

I think I found the issue - it's actually a problem with wheel. Spent a lot of time just now trying to make a reproducible example with one of my own repositories, only to find that my issues magically disappeared on creating a new Python environment and updating all my packages. It turns out that updating wheel from 0.30.0 to 0.31.0 fixes things (setuptools was 39.2.0 and twine was 1.11.0 in both cases).

[theacodes]

Gotcha. Makes sense. Shall we close this?

…

On Mon, May 28, 2018, 8:23 PM Jeff Stafford ***@***.***> wrote: I think I found the issue - it's actually a problem with wheel. Spent a lot of time just now trying to make a reproducible example with one of my own repositories, only to find that my issues magically disappeared on creating a new Python environment and updating all my packages. It turns out that updating wheel from 0.30.0 to 0.31.0 fixes things (setuptools was 39.2.0 and twine was 1.11.0 in both cases). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4079 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAPUc0qjc8sbdeqtJImtSwP8C8FHv90Qks5t3L87gaJpZM4UNuAe> .

[jstaf]

Go for it! (also thanks for looking into this!)

[fake-name]

Submitting dist/WebRequest-0.0.29.tar.gz to https://upload.pypi.org/legacy/
Upload failed (400): The description failed to render in the default format of reStructuredText. See https://pypi.org/help/#description-content-type for more information.
error: Upload failed (400): The description failed to render in the default format of reStructuredText. See https://pypi.org/help/#description-content-type for more information.

Still fails at my end. I manually updated wheel, setuptools was already up-to-date (Setuptools 39.2.0, wheel 0.31.1).

Twine is failing:

durr@rwpscrape /m/S/S/WebRequest> twine upload dist/WebRequest-0.0.29.tar.gz
Uploading distributions to https://pypi.python.org/pypi
UploadToDeprecatedPyPIDetected: You're trying to upload to the legacy PyPI site 'https://pypi.python.org/pypi'. Uploading to those sites is deprecated.
 The new sites are pypi.org and test.pypi.org. Try using https://upload.pypi.org/legacy/ (or https://test.pypi.org/legacy/) to upload your packages instead. These are the default URLs for Twine now.
 More at https://packaging.python.org/guides/migrating-to-pypi-org/ .

let me fix that, and see if it was relevant.

Ok, uploading with twine works, using setup.py does not. I've checked that setuptools, wheel, twine, and readme_renderer are all up to date, is there something I'm missing?

Some built packages:

SDist: WebRequest-0.0.30.tar.gz

BDist: WebRequest-0.0.30.linux-x86_64.tar.gz

[greyli]

Having the same issue. I have all requirements updated:

$ pip install --upgrade setuptools wheel twine
Requirement already up-to-date: setuptools in c:\python27\lib\site-packages (39.2.0)
Requirement already up-to-date: wheel in c:\python27\lib\site-packages (0.31.1)
Requirement already up-to-date: twine in c:\python27\lib\site-packages (1.11.0)

setup.py:

from setuptools import setup

basedir = path.abspath(path.dirname(__file__))
with open(path.join(basedir, 'README.md'), encoding='utf-8') as f:
    long_description = f.read()

setup(
    ...
    long_description=long_description,
    long_description_content_type='text/markdown',
    ...
)

PKG_INFO:

Metadata-Version: 2.1
...
Description: # Flask-CKEditor
...
Description-Content-Type: text/markdown

Here is the error output when upload:

$ twine upload dist/*
Uploading distributions to https://upload.pypi.org/legacy/
Uploading Flask_CKEditor-0.4-py2.py3-none-any.whl
100%|##########################################################################################################################################| 4.95M/4.95M [01:15<00:00, 22.7KB/s]
Uploading Flask-CKEditor-0.4.tar.gz
100%|##########################################################################################################################################| 3.70M/3.70M [00:55<00:00, 33.8KB/s]
HTTPError: 400 Client Error: The description failed to render in the default format of reStructuredText. See https://pypi.org/help/#description-content-type for more information. fo
r url: https://upload.pypi.org/legacy/

Any idea? It seems that only the wheel file (.whl) can be uploaded successful.

[theacodes]

Can you send over your sdist? I'd like to take a look at it.

[ocefpaf]

Just to provide another data point, Travis-CI auto deployment is also broken when using Markdown READMEs.

Here is the setup.py and as Travis-CI is using twine-1.11.0 and wheel-0.31.1, see https://travis-ci.org/pyoceans/erddapy/jobs/385360404#L1666.

The error message:

HTTPError: 400 Client Error: The description failed to render in the default format of
reStructuredText. See https://pypi.org/help/#description-content-type for more information.
for url: https://upload.pypi.org/legacy/ PyPI upload failed.

makes me think that something somewhere is ignoring that this is a Markdown file even though it is explicitly declared in setup.py as recommended in https://pypi.org/help/#description-content-type

[rpkilby]

long_description_content_type works with twine, but seems to have no effect when using setuptools.

Works:

• setup.py contains long_description_content_type='text/markdown',
• setup.py bdist_wheel (sdist also works)
• twine upload -r test dist/*

Does not work:

• setup.py contains long_description_content_type='text/markdown',
• setup.py bdist_wheel upload -r test

Pacakges:

pip==10.0.1
setuptools==39.2.0
twine==1.11.0
wheel==0.31.1

[ocefpaf]

@rpkilby based on your message I guess that Travis-CI is using setup.py bdist_wheel upload :-/

I'm trying to find the proper channel there to file an issue but it would be nice if setup.py sdist/bdist_wheel upload worked too until we all make the transition to twine.

[theacodes]

We're reverting this behavior in #4099, hang tight.

[frmdstryr]

A package I originally uploaded with no content type set, and then attempted to change to markdown by adding long_description_content_type="text/markdown" and it's also not working.

Is it possible to change the content type for an existing package?

[di]

Hi @frmdstryr, it's not possible to change anything about a published package -- they are essentially immutable.

It's likely that you have an out-of-date version of either twine, wheel or setuptools which cannot interpret this new metadata field. You can upgrade them all with:

$ pip install -U twine wheel setuptools

Once you do that, you can either increment the version number and make a new release, or make a post release to fix your description.

[frmdstryr]

Sorry, by existing I meant the package was originally uploaded without a content type set, it was a new release with an incremented version number.

Also setuptools and twine are already up to date

$ pip install --upgrade twine
Requirement already up-to-date: twine in /home/jrm/.local/lib/python2.7/site-packages
Requirement already up-to-date: requests!=2.15,!=2.16,>=2.5.0 in /usr/local/lib/python2.7/dist-packages (from twine)
Requirement already up-to-date: setuptools>=0.7.0 in /usr/local/lib/python2.7/dist-packages (from twine)

[di]

@frmdstryr Can you share a link to the release in question?

[tuanluu-agilityio]

Please update setuptools to latest version. It work fine for me.
Thanks

[tidely]

I have setuptools upgraded to version 41.0.1
twine version 1.13.0
wheel version 0.33.4
and i am still getting this issue

[devfans]

Still have the issue with latest packages installed:
setuptools 41.0.1
twine 1.13.0
wheel 0.33.4

HTTPError: 400 Client Error: The description failed to render in the default format of reStructuredText. See https://pypi.org/help/#description-content-type for more information. for url: https://upload.pypi.org/legacy/

[devfans]

Seems I can make it working by modifying the README.md a bit, which is change
```python
Some code
```

to
```

some code
```

[jamadden]

PyPI only renders rst(reStructuredText) file so if you have README.md then you have to convert into README.rst file after that you will be able to upload the file on PyPI.

PyPI has supported markdown long descriptions since March 2018, so there's no need to convert.

[jagadeesh-l]

twine check dist/*

Checking dist\xx-1.1-py2.py3-none-any.whl: FAILED
long_description has syntax errors in markup and would not be rendered on PyPI.
line 1: Error: Document or section may not begin with a transition.
warning: long_description_content_type missing. defaulting to text/x-rst.

[hlovatt]

I have same issue, e.g. like @jagadeesh-l describes, on version 3.2.0 of twine.

[stef]

i have the same issue. even after updating using pip install -U twine wheel setuptools and removing any language indicators after ```

[Syazvinski]

Im still having this issue as well. Anyone found a fix?

[phseiff]

I ran

sudo pip3 install --upgrade pip
sudo pip3 install --upgrade setuptools
sudo pip3 install --upgrade wheel
sudo pip3 install --upgrade twine
sudo pip3 install --upgrade setuptools_scm

and then, I ran

python3 setup.py sdist
twine upload -r testpypi -u <myusername> -p <mypassword> dist/*

and I still get

The description failed to render in the default format of reStructuredText. See https://test.pypi.org/help/#description-content-type for more information.

even though I correctly specified

    long_description=open('README.md', "r").read(),
    long_description_content_type="text/markdown",

in my setup.py.

It's also worth noting that uploading works just fine from my own machine, and used to work in GitHub actions too for the previous release, but I did not figure out where this difference in behavior comes from (probably because of different versions of Python modules).

The code in question is this GitHub actions pipeline (link points to the exact lines where the package is supposed to be pushed to PyPi for convenience).

Thanks in advance!

[di]

Anyone still having an issue with Description-Content-Type should file a new issue.