Control excessive Kedro versions in search results

[astrojuanlu]

Description

Sometimes when I search content from the Kedro docs, too many versions appear:

Context

Ideally, only one version (the latest one) should be shown.

Possible Implementation

Read the Docs allows you to hide a version from search engines https://docs.readthedocs.io/en/stable/guides/deprecating-content.html#deprecating-versions however, doing so also hides the version from the flyout menu.

Possible Alternatives

Alternatively, we can create our own robots.txt file https://docs.readthedocs.io/en/stable/guides/technical-docs-seo-guide.html#use-a-robots-txt-file that disallows certain versions from search engines while leaving the flyout menu untouched. This would need a bit of maintenance over time.

cc @stichbury

[stichbury]

This has bothered me for a while too, and I did restrict some of the older versions in RTD for that reason, but figured the flyout needed to show most, if not all, recent versions. I like the idea of modifying robots.txt and also thinking about how we present versions when we come to customise the docs with our theme for the subprojects. Maybe there's a way to update in RTD to restrict visibility in the flyout, but then add a design element to allow access to all versions that's separate from the flyout for people that really want them?

[stichbury]

I'm going to suggest we put this in the coming sprint as a task for engineering (maybe @SajidAlamQB ?) but it's not a high priority if we need to push it over to the next.

[astrojuanlu]

Starting point: the automatically generated robots.txt https://docs.kedro.org/robots.txt

[stichbury]

We will need to guide @tynandebold and @SajidAlamQB on which versions we want to index and which we want to hide. Do we want to be fairly proscriptive and have just these indexed:

• 0.16.0
• 0.17.0
• 0.18.0
• Latest, stable

Or be a bit more generous and index these:

• 0.16.0
• 0.17.0
• 0.18.0,
• 0.18.5 - 0.18.13
• Latest, stable

[stichbury]

I think we may as well be quite strict but it does depend how familiar people are with the flyout at the bottom left, if they want more versions. Can we change the RTD settings so the docs always load with the flyout open?

[astrojuanlu]

Or maybe instead of .0, the latest ones from each minor version? Hence

• 0.18.5 onwards
• 0.17.7
• 0.16.6
• latest, stable

[stichbury]

Sounds reasonable, probably more useful than just going with the initial versions of each major release. Let's do it!

[stichbury]

I've changed my mind, and think we should now update robots.txt to just allow these for search indexing:

• 0.18.5 onwards
• 0.17.7
• latest
• stable

All the versions are still on the RTD site and accessible. Just not indexed for SERP presentation.