[RFC] module/package level pragma {.doctype: markdown|rst.} to help migrate docs from RST to Markdown

[timotheecour]

There seems to be a consensus (or near-consensus) that markdown is preferred over RST for documentation, see https://forum.nim-lang.org/t/4280 (Drop RST and join the Markdown train?) and that the only valid argument for keeping RST is historical reason.

A number of people have also noted that supporting a hybrid will just cause more issues, so let's explore practical ways to gradually migrate docs to markdown.
To ease the migration I propose the following:

introduce a module/package-level annotation {.doctype: markdown|rst.} which states how to interpret documentation comments (as either markdown or RST depending on doctype).

usage

foo/
  nim.cfg # contains --doctype:markdown : applies to all modules under foo/
  foo_bar1.nim
  foo_bar2.nim # can override nim.cfg with `{.doctype: markdown|rst.}`
bar3.nim
bar4.nim # contains `{.doctype: markdown.}`

so doctype would only have to be specified at a few places (eg just 1, in nimble/nim.cfg/config.nims) until this becomes the new default.

automating documentation source code migration from rst to markdown

• tooling should be able to handle the 99% case fairly easily, the rest could be manual fixups; tooling should report when it can't transform a doc comment, and when the resulting markdown is invalid; this could be part of nim doc, nimpretty, or a separate tool.

[Araq]

How do I write

`term`:idx:

in markdown?

[alaviss]

You'd need pandoc-flavored markdown for it: https://pandoc.org/try/?text=%60term%60%3Aidx%3A&from=rst&to=markdown

[dom96]

If we're going to move to a new format then I vote for asciidoc ;)

[Araq]

Ok, so it's just some poorly thought-out proposal causing lots of friction, yay.

[arnetheduck]

How do I write

you don't. this, along with the phone book, has been deprecated in the modern world in favor of search.

[Araq]

Oh really, but Nim's search uses :idx:.

[arnetheduck]

Oh really, but Nim's search uses :idx:.

how.. quaint ;)

[timotheecour]

can we start with gradual migration of rst files to markdown?

Here's a typical example where rst falls short:

https://raw.githubusercontent.com/nim-lang/Nim/devel/doc/codeowners.rst renders as https://github.com/nim-lang/Nim/blob/devel/doc/codeowners.rst

see also rendering:

an entire table is missing (Compiler table), probably due to some obscure easy to miss syntax error noone noticed.

This happened to me with rst files in other context, where code snippets were missing.

[GULPF]

@timotheecour To be fair, that's only due to differences between githubs and Nims RST parsers. It renders fine on the website. That problem wouldn't go away by replacing RST - markdown is notoriously ill-specified, especially when it comes to non-standard features like tables.

IMO if Nim is going to switch to markdown, there first must be a pure Nim implementation. Preferable one that implements a well specified form of markdown like CommonMark (note however that commonmark does not support anything fancy like tables).

[timotheecour]

I think it's fair to assume a lot of people will browse through sources via github ( eg https://github.com/nim-lang/Nim/blob/devel/doc/codeowners.rst) so silently skipping entire sections (tables here, code blocks in other cases I ran into) is a bit of an issue...

note however that commonmark does not support anything fancy like tables

tables need to be supported...

That problem wouldn't go away by replacing RST - markdown is notoriously ill-specified

IIRC markdown will at least always render something (even if badly), but it won't skip anything, as is the case with RST as domonstrated above.

[haltcase]

@GULPF

IMO if Nim is going to switch to markdown, there first must be a pure Nim implementation

Luckily, @soasme is working on one 😄 https://github.com/soasme/nim-markdown

[soasme]

Thanks for the citing, @citycide. And Thanks again for @dom96 merging the package publishing PR the other days.

In the latest version bumping (v0.3.0), I've added the table support. In the next few days, I'll add more test cases to make it more stable. It could be awesome if nim-markdown can be used.

[Araq]

an entire table is missing (Compiler table), probably due to some obscure easy to miss syntax error noone noticed.

Yeah, RST sucks which is why Nim's RST parser deviates from the standard... And yeah, markdown has the same problems.

[dom96]

@soasme I'll happily use it on the Nim forum. :)

[timotheecour]

/cc @Araq

And yeah, markdown has the same problems

• well tables are supported by both github's markdown as well as https://github.com/soasme/nim-markdown/

• furthermore, markdown won't silently skip something it can't understand and will isntead render it the best it can (and then when readers see the bad rendering, it's easy to spot there's a mistake) ; whereas rst will just skip over and it can be hard to spot there's a mistake in the 1st place, such as the example I gave with a missing table)

[Araq]

So what, markdown has significant trailing whitespace. I mean seriously, it's all pretty bad.

[timotheecour]

has significant trailing whitespace

we don't have to use features we don't like and we can easily write a linter to reject that (eg in a github hook / in CI).

instead of using trailing whitespace for newline, we can just use the popular extension escaped_line_breaks.
github supports it and probably (EDIT: now also) nim-markdown, so we can write instead:

this is line1\
this is line2

[krux02]

Well I would like to add my two cents. Documentation in Nim is written in neither markdown nor RST. It is written in whatever the documentation parser understands. And the documentation parser understands an RST/Markdown hybrid. So people care write their comments in whatever they want to write it in.

Enforcing a specific style of doc comments is not something for a pragma, it is the job of a linter. So my vote for the pragma is a no.

[Araq]

There simply isn't so much of a difference between Markdown and RST. Rejected. Also: The docgen now supports more markdown features.