Add markdown support

[nlhkabu]

I've been working styling the information panel (issue #801).

One thing that I've noticed is that a number of package long descriptions are not coming through with the correct formatting:

The first is clearly markdown, the second, I am not sure, but I have noticed a number of other packages using the same syntax.

Obviously, it would be great to render these out to HTML so the new styles will work.

[nchammas]

The first is Markdown (as you pointed out) and the second is reStructuredText.

The current PyPI does not render long descriptions in Markdown. For example, here is the project from your first screenshot on the current PyPI.

There is a related discussion over on the PyPI repo that you should probably look at. It seems there is some stuff that needs to be figured out in order for PyPI to be able to support multiple formats for the long description. Currently, only RST is supported.

Here is the project from your second screenshot on the current PyPI. The long description renders correctly since it's in RST.

[nlhkabu]

Thanks @nchammas - looks like I walked into a minefield on that one :P

So from that thread I take it that Markdown support is unlikely in the short term.

On the RST side - do we have plans to support this (ping @dstufft)? I assume we will want to do this to provide parity with the current site?

[dstufft]

Hmm, we should be using the exact same code to render descriptions as PyPI is. Sounds like a bug of some sort.

[dstufft]

Looking at this again, it appears RST rendering is working the same as it does in the legacy site. I think the confusion comes from the fact that our test data comes from TestPyPI not from PyPI so you need to compare to https://testpypi.python.org/pypi/django-excel-response3/ which also has broken display. My guess is they fixed some broken RST and didn't update TestPyPI with the fix.

[dstufft]

I'm going to drop this from "Become PyPI". We don't currently support Markdown and actually doing so properly is more effort than it's worth to try and get it into our initial launch.

[nlhkabu]

Ok, this makes sense... thanks for the follow up @dstufft

[boxed]

I've recently had a lot of problems getting the rST to work on pypi. Some problems were fixed by running the check command in setup.py, but one error I had didn't show up there: I had the string 'foo__' which is a reference in rST. This error didn't show up when running the check command so I had to do a binary search of the description to find it :P

Clearly some work on improving the error checking on this would be nice, but isn't strictly a part of warehouse although warehouse suffers from it.

[hadim]

Could you elaborate a bit more about why it's a lot of effort to support Markdown ?

A lot of pypi packages are hosted on github where they have a Markdown README file. Maintaining an alternative README file in .rst for pypi seems to be a lot of work for nothing... That is why most of developers (me included) don't bother converting the description field and let it as Markdown.

It follows that a lot of packages does not have a nicely formated page on pypi.

As the future platform for Python packages, don't you think it's worth trying to put some effort in it ?

[hadim]

I just see that some work is undergoing there pypa/readme_renderer#3

[hadim]

Please see pypa/packaging.python.org#258 for a more complete overview.

[Lucas-C]

By reading this thread, it seems to me this feature is pending on the completion of the PR mentioned by @hadim ?
pypa/readme_renderer#3
as this README rendereding lib is used by warehouse: https://github.com/pypa/warehouse/blob/master/requirements/main.in#L37

Is that correct ?

[alonisser]

+1 fo markdown.. I would like to help with coding if supporting markdown is an option

[9999years]

I’ll also help implement support for Markdown or other languages! I think one of Github’s strengths is strong markup-language support. (Github currently supports nine different languages!)

I think it would make sense to implement the documentation as a subset of HTML rather than any one markup language; one of RST’s big problems is that there’s no support for nested inline styles, such as code links or bold italics, and we could avoid the inflexibilities of any particular markup language by supporting several or a superset of all of them.

[pradyunsg]

pypa/readme_renderer#51 implements Markdown Support and making it easier to add support for more languages in the future (I guess?).

It still needs to be reviewed AFAICT.

[gemerden]

Ehm, while understanding that feature requests can lead to discussion which can lead to better solutions, i am a bit lost on why this is complicated, especially compared to the value it adds. Seems that find a way to indicate what format to render and render it that way does not have to take 3 years.

I have seen loads of suggestions for solutions (e.g. add long_description_format to setup) that might for some reason or other not be perfect, but all are better than the current state.

My personal experience:

• oh, pypi does not render markdown,
• hm, google, maybe there is an automatic converter,
• read about pypandoc,
• install pypandoc,
• change an run setup.py,
• re-upload to pypi,
• check doc, hm, mostly ok, but conversion is not perfect,
• change markdown,
• 2 or 3 versions further: all good (i think, getting tired of rereading the readme).

And i am pretty sure that pypandoc is the easiest way to go and this took me about 3 hours for a simple project. Multiply by number of uploaders from github ...

Anyway, bit ranty, sorry about that, is there an eta yet?

[dstufft]

It's Open Source Software, the ETA is that it'll get done whenever someone has to motivation to do it. Someone took the effort to go through and get agreement on the mechanism to indicate what format to render things in, now someone needs to go through and add support for that to setuptools, twine, and Warehouse.

[ghost]

@dstufft What about pypa/readme_renderer#51?

[di]

With #3296 merged and a pre-release of twine available, it is now technically possible to publish descriptions on PyPI using Markdown. 🎉

For example, see https://pypi.org/p/markdown-description-example/

Here's a step-by-step guide with everything necessary to make this work for your new or existing project: https://dustingram.com/articles/2018/03/16/markdown-descriptions-on-pypi

I'll leave this issue open until @brainwane makes the official twine release is made on Monday. Please feel free to comment here if you try this and are having issues.

[brainwane]

I'd love for people to start preparing PRs for documentation improvements for the sample project used in the official package distribution guide, and for that guide itself.

[brainwane]

Also, per Dustin's guide, if you're testing this, make a source distribution (sdist), not a wheel. Wheel support may depend on pypa/wheel#231 being resolved.

[sfdye]

@di Will this work now if we deploy to pypi from travis?
https://github.com/PyGithub/PyGithub/blob/master/.travis.yml

[9999years]

I’ve created a PR for the sample project (pypa/sampleproject#66) and I’m working on one for the guide as well. For everyone’s reference (from @di’s blog post) twine>=1.11.0 and setuptools>=38.6.0 are required for Markdown support.

[brainwane]

Thanks for your PR, @9999years! Versions: pkginfo 1.4.2, Twine 1.11.0 (right now 1.11.0rc1 will work), setuptools 38.6.0, and I'm not entirely clear regarding wheel since wheel support does not work yet as of wheel 0.30.0. Dustin's blog post mentions some of those. I am not sure about Python versions (distutils being part of the Python standard library). Twine 1.11.0 requires the relevant version of pkginfo.

There's an open issue for the packaging docs.

@sfdye I see you're creating both a wheel and a source distribution but it looks like you only upload an sdist so I think you'll be fine, but of course I defer to @di here. (I'm experimenting with Markdown support with this test project and Markdown support works if I upload only an sdist, but if I upload a wheel, or an sdist and a wheel, then Markdown support does not yet work.)

Those of you who want to use GitHub-flavored Markdown (beyond what's in the CommonMark specification), please go over to pypa/packaging-problems#126 to find out what's necessary, again, in a few different codebases.

[9999years]

Okay, the guide PR is at pypa/packaging.python.org#457.

[ncoghlan]

For the time being, we don't have any plans for distutils itself to support metadata versions beyond 1.1 - the guidance for a number of years has been to use setuptools instead (regardless of Python version), and this is just one more reason to continue that recommendation.

However, I think the existence of the standard library distutils API implementation is a genuine problem given that we don't really want people to be using it, so I filed pypa/packaging-problems#127 to discuss how we might manage the process of making setuptools the reference implementation of the traditional distutils API.

[di]

and I'm not entirely clear regarding wheel since wheel support does not work yet as of wheel 0.30.0

Yes, per pypa/wheel#231 we will need to instruct users wanting to build wheels to use an as-of-yet-unreleased version of wheel (working on this now).

[brainwane]

Twine 1.11.0 is now out and includes some fixes that'll stop users from trying stuff that won't work anymore.

We're leaving this issue open till @di's pypa/wheel#232 is merged & released, so users can upload Markdown READMEs in their wheels.

(You can thank Dustin in person at PyCon in May, where he's speaking and sprinting.)

[brainwane]

@di does pypa/wheel@0.30.0...0.31.0 and https://pypi.org/project/wheel/0.31.0/ mean we can close this issue?

[di]

Yep! Thanks everyone for your patience w/ this issue.