Move Markdown list parsing closer to CommonMark

[jgm]

Currently we follow the 4-space rule. It would be good to switch to the CommonMark rule for nested list content, which is a bit more complex but allows more flexibility. Well-formatted 4-space rule lists should still work with no changes, but we'd also be able to handle 2-space indented sublists, for example. (See many closed bug reports relating to this...#3125 #2367 #2575 #2210 #1990 #1137 #744 #172 #137 #128.)

There'd be some danger of breakage of existing documents when indented code blocks are used inside list items. Currently these need to be indented 8 spaces relative to the outer list. With the CommonMark rule, the indentation needed depends on where the list content begins after the list marker. So, if you had a list like

- foo

        code

which pandoc currently interprets as

<ul>
<li><p>foo</p>
<pre><code>code</code></pre></li>
</ul>

with the CommonMark rules you'd get

<ul>
<li><p>foo</p>
<pre><code>  code
</code></pre></li>
</ul>

with two extra spaces at the beginning of the code. But if you had

  - foo

        code

you'd be fine, it would render the same.

All in all, I think the risk of breaking some existing documents is outweighed by the benefits of rendering nested lists where people clearly expect them, and reducing unexpected surprises when moving between Markdown implementations.

[jgm]

An extension four_space_rule could be added so the old behavior could be selectively enabled.

[jgm]

The tricky part of this is dealing with tabs.

[vvasuki]

In addition to the four space rule problem, there is also the problematic need to necessarily have a newline before a markdown list begins - please see second section here.

[jgm]

@vvasuki you can enable +lists_without_preceding_blankline if you want lists to be able to start without a blank line.

[bewuethr]

Should maybe the language in the first example in Block content in list items be updated? While "must be indented eight spaces" is true for the example because the whole list is indented, it's really "must be indented four spaces more than the first non-space content after the list marker", isn't it?