Use the mkdocs package to generate documentation

[waylan]

We currently use a homegrown solution to generate our documentation. While it works fine, it is a little inflexible and requires some ridiculous hoops to add new pages: (1) create the new page, (2) add the page to the siteindex, and (3) update the meta data of both the page before and page after. I'd like to eliminate step 3 which is redundant when we have the siteindex. But the current implementation renders each page individually so info about next and previous pages isn't available unless it is stored in the given page. The existing siteindex just serves as a dumb, static list of all pages. If you forget to add or delete a page from the siteindex, everything still works except that the siteindex is wrong. Most site generators use the siteindex to build a global context which provides each page with its next/previous page links, etc.

Rather than "reinventing the wheel" we should just use an existing solution. mkdocs looks like the most reasonable solution to me. It appears to be under active development (unlike a few of my false starts), it has a set of goals which align with our needs, it uses Python-Markdown under the hood, and has configurable support for Python-Markdown extensions. In fact, with an updated template it would almost be a drop-in replacement. It is also supported out-of-the-box by readthedocs.

There are a few problems though:

1. mkdocs does not support the latest version of Python-Markdown as it still supports Python 2.6 (we dropped support in PM v 2.5). So we need to wait for mkdocs' issue 165 to be resolved.
2. mkdocs is using the old (special) treatment of shortened extension names which is currently pending deprecation (see 2.5 release notes). A minor refactor of its convert_markdown function is in order (see below).
3. A way to configure extensions should be implemented as the old way (appending to extension name wrapped in parentheses) is also pending deprecation. Perhaps something similar to the new --extensions-config command line option would serve well if implemented as part of mkdocs config file.

A refactor of the convert_markdown function might include the following:

1. As previously mentioned 'toc' => 'markdown.extensions.toc', etc.
2. Account for duplicate extensions (the user may have specified an extension in their config which is also in the default list -- perhaps use a set?) This is not an issue as long as the default list is loaded first. the last instance to be loaded wins. Should probably add some tests to PM for this (and check for what happens when an extension deletes an existing processor when the previously loaded instance already deleted it).
3. Use md.toc rather than doing its own thing - but that requires the TOC extension to always assign the toc to md.toc, not just when no TOC Marker is found in the document (remember, a doc author may have placed their own toc marker in the doc also). Also TOC extension should register and 'reset' the md.toc. A refactor of the TOC extension needs to happen here first. TOC refactor has been done and will be released in 2.6. So this change depends on markdown>=2.6

[d0ugal]

I'm part of the MkDocs team and I just stumbled on this. I'd love to help make this happen. It looks like the changes to MkDocs are mostly simple or just require some process stuff on our end (like dropping python 2.6). For now I'll just track this but after I get the next release out I'll try to give this some attention on the MkDocs end. If nothing else, this provides us with some great tips about what we are doing wrong.

[mitya57]

It will be also nice if mkdocs (officially) provided some Python API to make it easy to use from setup.py (so far the documentation only mentions calling it from shell).

[d0ugal]

@mitya57 Yeah, that is something we don't want to support at the moment. I wonder if these is another way we can make it easier.

[erez-o]

Old issue, but it's still open so I'll suggest another:

I created a new website for hosting and creating documentation of open source projects (100% free of course). It's now on Beta. It also supports automatic code documentation for python out of the box.

You can see how python-markdown documentation looks at http://docsforge.com/2/Python-Markdown/docs/latest/

I'm currently the default owner since I uploaded it, but here's a screenshot of how it looks for the maintainer if you're interested.

Btw, I'm using python-markdown for rendering markdown in my website.

[waylan]

This was addressed in #602.