rst parser makes a line starting with > silently disappear

[timotheecour]

Last time i reported a bug in nimforum it turns out its root cause was nim's rst parser so I'm filing it here.

Example 1

see https://forum.nim-lang.org/t/7953#50676; i posted something like this:

foo https://github.com/nim-lang/Nim/issues/8258

> bar

Current Output

the 2nd line disappears

Expected Output

the 2nd line shouldn't be omitted, or at least it should give an error

Example 2

it works if there's something afterwards: this doesn't omit anything:

foo https://github.com/nim-lang/Nim/issues/8258

> bar

baz

Additional Information

1.5.1 72d6b59
/cc @a-mr

[a-mr]

rst.nim does not support > quotes currently. It's nimforum code, see https://github.com/nim-lang/nimforum/blob/35e0de7b91fc76230574b2882005d1eccd12e2ad/src/utils.nim#L97

I think we need to support > quotes properly in rst.nim, so that nimforum could drop its own implementation. So we can keep this issue as a feature request.

[timotheecour]

looks like > is indeed in both rst spec and markdown spec, and is a commonly used feature, so it's a good idea to support indeed:

rst: https://docutils.sourceforge.io/docs/user/rst/quickref.html

Per-line quoting can also be used on
unindented literal blocks::

> Useful for quotes from email and
> for Haskell literate programming.

markdown: https://wordpress.com/support/markdown-quick-reference/

> Quoted text.
> > Quoted quote.

> * Quoted 
> * List

is the 2nd example (with deeper nesting) also valid rst?

[a-mr]

is the 2nd example (with deeper nesting) also valid rst?

nope, it requires :: before it.

But now the general direction is to move to better Markdown compatibility anyway (according to discussion in #17827) so I'm going to do it according to the Markdown style.

[timotheecour]

can you give a quick summary of:

• what's already valid
• what will become valid
• what will stay invalid?
  (and whether this is for both nim rst and nim doc or just one of these)
  btw is nimforum based on nim's support for rst or nim's support for nim doc ?

i think the most common/useful things to start with are:

not quoted
> 1st level quote
>> 2nd level quote

• is an empty line required anywhere in the above snippet ?

[a-mr]

• nothing is valid, it's parsed just as normal text containing >
• ideally both RST and Markdown style will become valid, but minimum target is Markdown style
• nim doc and rst2html use the same parser and generator so it will be supported everywhere of course. nimforum uses helper rstgen.rstToHtml which is mostly trivial (FWIW it's not used in nim doc or rst2html). Basically it's all the same
• in Markdown style a blank line would not be required, I suppose.

[a-mr]

... I mean when roPreferMarkdown is set (which is currently default and cannot be changed) then blank line would not be required.

Because there will be more Markdown support, I'm thinking about providing an option like --pedantic or --pureRst to enforce pure RST-compliant behavior (in rst2html and probably doc), WDYT?

[timotheecour]

• --pedantic is too vague, not self-documenting
• --pureRst is better, but I much prefer enum-style flags, they're more naturally extensible/future proof/self-documenting, hence the move from --listfullpaths to --filenames:abs|canonical|legacyRelProj, --processing:dots|filenames|off for eg.

so I'd recommend: --docLang:nim|markdown|rst, which would default to --docLang:nim (nim's own mix of rst + markdown), whereas markdown would be pure markdown, rst pure rst.

this flag would compose and be honored by nim doc|rst2html|doc2rst

Also, --docLang starts with doc, like --docCmd, --docSeeSrcUrl, --docInternal, --docRoot, which makes it more discoverable.

(and ideally we can add aliases in future work for other doc-specific commands like --project, --index, but that's a separate topic)

[Araq]

so I'd recommend: --docLang:nim|markdown|rst, which would default to --docLang:nim (nim's own mix of rst + markdown), whereas markdown would be pure markdown, rst pure rst.

And then these flags end up in somebody's config causing subtle changes in the doc rendering pipeline. Or actually introducing more Nim dialects if we want to be picky. Nim is not about choices via the command line, if Nim offers choices, it should be done as language mechanisms. This is why we're moving from --gc:X to --gc:orc, not because Orc is the best GC, but because it works well with destructors and custom memory management so that we get an ecosystem of libraries that work well together.

Even a module specific "doclang" pragma would be preferable over a command line switch, command line switches don't compose.

[timotheecour]

i agree with your points regarding fact that this shouldn't be a cmdline flag;

Even a module specific "doclang" pragma would be preferable over a command line switch

that's pretty much what I had proposed a while back in nim-lang/RFCs#68: {.doctype: markdown|rst.} to help with migration on a module-by-module basis