Converting the standard library to 100% Markdown

[mbauman]

There are currently 332 282 223 218 200 docstrings that haven't made the conversion to Markdown. While they still display nicely in the online manual, they are simply output as plaintext with RST markup at the terminal. Since the end goal is to have the entire manual as markdown (#12573), we will need to figure out ways of spelling the features we want in Markdown sooner or later. Here are the syntaxes that are currently used, roughly sorted by frequency of use:

• Function references (:func:foo and sometimes `:ref:`~ operator <~> or :func:f(x+y) ``)
• Object references (:obj:Bar and sometimes `:class:`Quux or :const:BAZ``)
• Error references (:exc:ErrorException``)
• Links to sections in the manual (:ref:Link name or auto-populated with anchor title by `:ref:`man-link-anchor)
• Doctests (RFC: Allow doctests in Markdown code #13045)
• Inline latex :math:\latex`` (Convert docstrings with inline latex to Markdown #13058) — Just need to teach Markdown how to export inline $x^2$ to RST (72 uses, very few blocked by other syntaxes).
• Superscripts :sup:T`` (Convert Ac_ldiv_B etc. docs to Markdown #13108) — could use unicode since its use is limited to Aᵀ and `Bᴴ` (18 uses, none blocked by other syntaxes).
• Bullet lists (Manually convert bulleted lists to Markdown #13046)
• **Example**s (Convert some docs with **Example** to Markdown #13102)
• Notes (.. note::) (example: gcdx)
• Citations, sometimes with inline references to things cited elsewhere (logm, pinv, sqrtm, expm, bkfact, qrfact, full)
• Tables (MultiMarkdown has a possible solution) Markdown just needs to be taught how to export them to RST.
• Literal inline code blocks (:literal:has` in it``)?

We probably don't really need to preserve all of Python's different ways of referencing functions/objects/classes/constants/exceptions/etc. But that is far and away the biggest reason why most RST blocks haven't been converted. If we can figure out a way to do references, we'll be 90% of the way there.

Note that if one method's docstring failed to auto-convert to markdown, none of the functions methods were converted to markdown. So when addressing one of the items above, search to see if addressing the docstring of one method frees all the other methods, too.

[jakebolewski]

This is a great list. The complexity of implementing any solution wrt to crosslinking will be dramatically reduced if the rest of the manual is converted to Markdown. Most of the reasons why so many features are used is that they link back into the manual and vice versa.

[mbauman]

Yeah, but I think that, regardless of the manual solution, we'll want some shorthand way to specify manual/stdlib anchors that aren't necessarily on the same page.

Perhaps we could just use regular anchor syntax, and then the RST/mythical-new-manual output will reinterpret them as manual/stdlib cross references? E.g., [foo](#foo) or [bar()](#bar) or [the baz operator](#baz) or [random numbers section](#man-random-numbers). They'd all just output to plain old :ref:title ``s for now.

[mbauman]

Perhaps the solution for function references is to use $(link(foo)). For now, it could just splice in "[foo()](#foo)", which would become :ref:foo() ``, but it would allow us to use multiple dispatch to format functions/objects/types/etc differently in the future without mucking with the docs again.

[hayd]

• inline latex should just work, it just needs to be rendered to rst. (This is another case where refactor would help (to make it clearer what methods are missing). Move markdown types into their own file. #11421 Markdown code refactor #12740.)
• @MichaelHatherly and I were hashing out in Docile a possible syntax for notes and citations etc...
• What's wrong with using the current (github) table syntax? Is it just we need the julia markdown -> rst?
• Note for literals you can use the `` syntax:

julia> Markdown.parse("``has` in it``")
  has` in it

• For references could a syntax like:

  """text with $(Ref("`f(x+y)`", f))"""
  

(Perhaps would need a trick to work with doc"""...""".) I'm sure @one-more-minute had a solution for this in the past...

[mbauman]

inline latex should just work, it just needs to be rendered to rst.

Yup, I got that working last night and should have a PR for those guys this evening.

What's wrong with using the current (github) table syntax

I wasn't aware it was supported. If it is: awesome! 🍰 There's only about four or five tables, so it's not high on the priority list right now.

Note for literals you can use the `` syntax

Yeah, there's literally one use of the :literal: directive. The question mark was more for its purpose in RST. It's an easy fix.

references

Splicing always inserts a string, even into string macros, no? So we'll have to figure out what the Markdown surface syntax is in any case. Duh, right, it's the non-evaluated string. The macro decides what to do with the plaintext. So it could be a special pseudo-interpolated syntax.

[hayd]

The ref I was thinking of is in the markdown tests. I don't see any reason we'd need the type func/obj/class/const, that can be grokked from the type. This would also solve ALL the links as we'd do:

:ref:`Link name <man-link-anchor>`
[Link name]($(url(ref(f)))  # or something similar but nicer

Is there a method to get the docs url from the type/func/const etc. ?

[rubydesign]

Hi
I'm spending my time reading at the moment, so if there is some clear cut work on these lines, i could help.
Torsten

[MichaelHatherly]

Just an update on this: I'm currently translating the rest of the rst docstrings to markdown, tidying up and reformatting others, adding @StefanKarpinski's suggestion (#14310 (comment)) for inline LaTeX using double backticks, and removing all usage of @doc_str from Base. I'll probably have something worth submitting either tomorrow or Saturday.

(The diff is currently quite large, ~13000 lines, but spread over quite a few small commits to hopefully make reviewing easier.)

[hayd]

+1. Do you have a plan for refs etc.?

[StefanKarpinski]

Please see #13308 for my work on converting RST to a variation of Markdown that I had to cobble together from conventions in GitHub Markdown, Scholarly Markdown and Pandoc Markdown. Definitely not final and I still need to modify our Markdown parser to understand the conventions and be able to render them to usable HTML. Could certainly use help. I was planning on posting a summary of what I'd decided on for conventions for things to get discussion going.

[MichaelHatherly]

+1. Do you have a plan for refs etc.?

For :func:, :ref:, :obj: I'm just translating

:func:`foobar` to [`foobar`](:func:`foobar`)

and special casing the rst rendering for any url that matches :func:, etc. If people like @mbauman's suggestion in #13047 (comment) then I can change it to that and just output :ref: rather than the assortment of different ones we have now. @StefanKarpinski, do you have a particular syntax in that branch for refs? I couldn't find anything.

[StefanKarpinski]

My plan for refs was to leverage the fact that docstrings are embedded in Julia code so unlike most documentation systems, we know what foobar is – we can know that it's a function and we can even know where it's defined. The user doesn't have to markup this fact at all – the cross reference can be generated completely automatically and all you have to write is foobar.

[MichaelHatherly]

Closed by #14378? There may be some further discussion needed for what our final ref syntax will be and perhaps some better formatting conventions in places, particularly .. note:: sections, but for the most part this issue is resolved.