Improvement Request: Change documentation format from RST to MD

[drew2a]

Now we use RST format for our documentation.

Initially, it was introduced because it was the only format supported by readthedocs. But now, readthedocs supports MD as well.

My proposition is to change this format to MD.

MD is more natural to GitHub and GitHub users than RST. This change can lead to a more accurate and frequent maintance of our documentation base.

MD format also will be rendered more correctly in GitHub.

Here is an example of rendering development.rst. It seems to have no body:

It has been rendered from this code:

=======================================
Setting up your development environment
=======================================

This page contains instructions on how to setup your development environment to run Tribler from source.

Windows
=======

.. include:: development_on_windows.rst

MacOS
=====

.. include:: development_on_osx.rst

Linux
=====

.. include:: development_on_linux.rst


======================
Development guidelines
======================

.. include:: qt.rst

References

1. https://docs.readthedocs.io/en/stable/intro/getting-started-with-sphinx.html#using-markdown-with-sphinx
2. https://myst-parser.readthedocs.io/en/latest/sphinx/intro.html
3. https://www.reddit.com/r/programming/comments/5z49vw/restructuredtext_vs_markdown_for_technical/

[qstokkink]

GitHub has its own Markdown variant (see https://github.github.com/gfm/), which does not offer the same syntax options as MyST. This means that all of the cool stuff MyST offers will still only render on ReadTheDocs and not on GitHub. For example, things like included files and TOC tree generation will still not render on GitHub*.

As the GitHub syntax offers no means for file inclusion or generation from other files, previously auto-generated content will have to be pasted verbatim into a documentation file in order to render on GitHub. However, this is true for both GitHub's RST and MD rendering. Therefore, I see no need to switch documentation language in order to fix the problem of rendering on GitHub.

*To prove that MyST really doesn't render correctly on GitHub, here's the output of an example on the MyST website (note things like section labels and comments not working):

---

My nifty title

Some text!

:class: warning

Here's my admonition content

(section-two)=

Here's another section

And some more content.

% This comment won't make it into the outputs!
And here's {ref}a reference to this section <section-two>.
I can also reference the section {ref}section-two without specifying my title.

[drew2a]

GitHub has its own Markdown variant (see https://github.github.com/gfm/), which does not offer the same syntax options as MyST. This means that all of the cool stuff MyST offers will still only render on ReadTheDocs and not on GitHub.

Yes.
As I see it, these documents will have "boring", "not cool" normal markdown syntax.
Of course, it's may not be enough for our purposes (it's not obvious to me at this point).

As the GitHub syntax offers no means for file inclusion or generation from other files, previously auto-generated content will have to be pasted verbatim into a documentation file in order to render on GitHub. However, this is true for both GitHub's RST and MD rendering. Therefore, I see no need to switch documentation language in order to fix the problem of rendering on GitHub.

You are right. But MD will force you to have another document structure. As an option (in the example above): to merge these documents explicitly or to explicitly use links.

[synctext]

After bug fixing, we should at some point also give priority to improving the quality of our documentation.
Both Dispersy and Trustchain mentioned here are no longer used.

Lets schedule docs and .RST matters this after the big push on tagging coming months.

[qstokkink]

@synctext #4656 has been open since 2019 for exactly this. Nobody ever gives any love to the documentation. Tribler also evolves way too fast to keep stable documentation without tremendous effort (no matter what format we use). Perhaps we should do something radical and drop all documentation that is not a docstring or installation instructions? Requiring everyone to write docstrings seems like something we can realistically enforce.

@drew2a Fair enough. If you accept the required sacrifices to gain GitHub rendering and prefer working in MD, this becomes a matter of taste/personal preference and I have no further comment on this. However, it is important to know what you're "losing out"* on.

Some concrete examples that are actually used in IPv8:

• Automatic TOC generation (e.g., https://tribler.readthedocs.io/en/latest/?badge=main#welcome-to-tribler-s-documentation). Creating this by hand is annoying, but not impossible.
• Automatic code snippet testing. Separating code snippets into stand-alone files makes it easier to run them for automatic testing. You could also do this by extracting code snippets from the .md using a custom tool and making them runnable, this is just much harder. You could also opt to link to the external files instead of explicitly including them in the documentation, but this may make the documentation harder to read.
• Automatic class documentation (e.g., https://py-ipv8.readthedocs.io/en/latest/py-modindex.html). I'd consider this pretty much impossible to do without tooling (otherwise, you would have to mirror all functions and their docstrings in the documentation with manual standardized formatting). If you still want this using just .md files, all changes to the source code would require the documentation to be rebuilt using custom tooling and to be pushed to the repository.

That said, on the one hand there is a case to be made that Tribler will never need the above functionality. For example, Tribler's automatic class documentation has been broken for.. years (?) and nobody cares. On the other hand, based on our history (see also my answer to @synctext), I believe nobody will ever properly maintain Tribler documentation anyway and I have come to believe that all of our documentation should solely be automatically generated.

*You can still use a hybrid approach and leave some stuff unviewable on GitHub and only viewable on ReadTheDocs. This depends on how puristic you want to be about GitHub being the prime source of documentation.

[devos50]

We used to have .md format but, as I also commented in our Slack channel, we switched to the .rst format so we can render our documentation on readthedocs. I currently do not see much need to revert this back to Markdown.

I believe nobody will ever properly maintain Tribler documentation anyway and I have come to believe that all of our documentation should solely be automatically generated.

True. Much of our documentation is outdated and if updated, gets outdated again very fast. For example, our building/packaging instructions for Windows have changed significantly and requires updating when external packages get updates (e.g., libtorrent/visual studio/brew etc.).

[qstokkink]

Going with @devos50, we will stick to RST.