[Doc] Use sphinx-remove-toctrees to generated docstring pages from navigation (and reduce build time)

[asfimport]

The pydata-sphinx-theme tends to create large trees in the left-side navigation (and thus large generated large html sizes) because sphinx adds all generated docstring pages (from autosummary usage) to the global "toctrees". A small package (https://github.com/executablebooks/sphinx-remove-toctrees/) was created to help with this.

See https://pydata-sphinx-theme.readthedocs.io/en/latest/user_guide/configuring.html#selectively-remove-pages-from-your-sidebar

Reporter: Joris Van den Bossche / @jorisvandenbossche

_{Note: This issue was originally created as ARROW-14460. Please see the migration documentation for further details.}

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
I tried this out and it works nicely, but it also has a downside of then no longer "highlighting" the parent page in the navigation left-sidebar (which is somewhat annoying for our docs, since the sidebar is quite long and doesn't fit on a single page). See also executablebooks/sphinx-remove-toctrees#1 (comment)

On the other hand, with this change we can remove the if/else in the template here:

arrow/docs/source/_templates/docs-sidebar.html

Lines 13 to 17 in be665ef

	{% if "python/api" in pagename or "python/generated" in pagename %}
	{{ generate_nav_html("sidebar", startdepth=0, maxdepth=3, collapse=False, includehidden=True, titles_only=True) }}
	{% else %}
	{{ generate_nav_html("sidebar", startdepth=0, maxdepth=4, collapse=False, includehidden=True, titles_only=True) }}
	{% endif %}

(and have it a single generate_nav_html call instead).

[asfimport]

Todd Farmer / @toddfarmer:
This issue was last updated over 90 days ago, which may be an indication it is no longer being actively worked. To better reflect the current state, the issue is being unassigned. Please feel free to re-take assignment of the issue if it is being actively worked, or if you plan to start that work soon.

[AlenkaF]

I tried this out on the branch with the updated pydata sphinx theme and it works well. The fact that the left sidebar has a missing parent page is a bit confusing but not so much anymore because the parent page is visible in the article header, see: