Improve the documentation toolchain and hosting

[stichbury]

We've used Sphinx to build markdown and API docs since day 1, and hosted on RTD. As I understand it, we have to use Sphinx (or MkDocs) to publish to RTD, so the toolchain and the host are somewhat connected.

However, things have changed. We now have a domain (kedro.org) and options to host docs on netlify, github pages or gitbook. We don't need to use RTD, so therefore we don't need to use Sphinx.

Why wouldn't we continue as we are?

• Nobody is that keen on Sphinx. It's not easy to debug, it's brittle and somewhat inflexible.
• We are tied to Kedro releases. Some of our docs updates are super improving but they only get out to the users when we release...and that's increasingly taking time to happen.
• We want to do more with our markdown docs than Sphinx allows us. We can embed video, extend it to add tooltips and use mermaid, but it's inflexible in terms of information architecture and the overall output isn't that attractive. If we want to encourage different paths for learners depending on their persona and circumstances/use case, then we need a better way to organise the docs and share/re-use content.
• We have a new requirement that we should build API docs from across several repos (Kedro datasets and Kedro framework) and pull them together. This isn't straightforward.
  • We have too many docs and need to push some of what we have into the API docs (I'm not sure this is an argument but just putting it in here)

The way ahead?

We need to look at the alternatives and decide where our future lies if not with Sphinx/RTD. It may be that @astrojuanlu has some ideas for how to solve the issues we face, but I'm of the view that it makes sense to look at what else may work for Kedro (and beyond).

For example:

• Docsify could be an alternative to Sphinx for markdown docs
• pdoc3 may work for API docs

We need to spend some time as a spike to evaluate it; I'd like to make sure to earmark some of my time nearer the end of the year to fork Kedro and try the docs on my personal repo.

Analysis

We need to consider:

• Hosting options
• Redirects with new domain & host
• Page indexing with a tool that doesn't build static HTML
• Multi-repo solutions
• Versioning (where do older versions live? how do we offer multiple versions of docs?)
• General reputation/stability of any tools we adopt

[merelcht]

Sub-tasks for this parent issue:

Alternatives to Sphinx

• Explore alternatives to Sphinx and RTD#2072

Sphinx enhancements

• Consider a more modern theme for docs#2394
• Add a sphinx extension to show last edit date#2393
• Add a sphinx extension to separate code snippets into different OS#2366
• Compress sidebar on docs#2432

[stichbury]

@astrojuanlu One for us to cover today 🤞

[astrojuanlu]

I added some comments to gh-2072. As much as I cherish Sphinx, it's undeniable that it's difficult to tame, so I agree with @stichbury's considerations on the top comment.

Some of these pains are note necessarily tied to our tooling, but definitely impacted or influenced by it. In particular:

Some of our docs updates are super improving but they only get out to the users when we release...and that's increasingly taking time to happen.

See my suggestion on #2072 (comment) to decouple the API docs from the narrative docs.

[noklam]

Have a chat with someone today. https://squidfunk.github.io/mkdocs-material/ seems to be a good candidate

These libraries are all using it.

• https://hatch.pypa.io/
• https://fastapi.tiangolo.com/
• https://beta.ruff.rs/docs/

[astrojuanlu]

Responding to the MkDocs question over #2072 in a moment :)

[noklam]

Thx, I was trying to look for that issue but I couldn't, thanks for linking it.

[astrojuanlu]

I think we're in a much better position now. Voting to close this issue for now.

[stichbury]

Yes, agreed.