GH-35403: [Docs] Support sphinx 6 for building the docs

[AlenkaF]

What changes are included in this PR?

Enable the docs to be built with sphinx 6.x.

• Update the pin in the requirements.txt
• Add sphinxcontrib.jquery to include jQuery which is used in the dropdown.

Are these changes tested?

The Sphinx version update and the dropdown are tested locally.

Are there any user-facing changes?

No.

• Closes: [Docs] Support sphinx 6 for building the docs #35403

[AlenkaF]

I am not sure why is the installed Sphinx version on the CI still 5.3.0 (ci/conda_env_sphinx.txt is updated)
https://github.com/apache/arrow/actions/runs/5387119645/jobs/9778004280?pr=36296#step:6:1351

So this change is currently not tested on the CI.

[jorisvandenbossche]

Not sure why it doesn't automatically use sphinx 6, but we could also just force sphinx=6 in the conda env file, I think that would be fine (from testing locally there doesn't seem to be a conflict when doing that).

[AlenkaF]

Not sure why it doesn't automatically use sphinx 6, but we could also just force sphinx=6 in the conda env file, I think that would be fine (from testing locally there doesn't seem to be a conflict when doing that).

Just on the CI or in general requirements also?

[jorisvandenbossche]

The docs/requirements.txt file has a comment that it should be kept in sync with conda_env_sphinx.txt, so in both places

[AlenkaF]

Oh forgot about that, sry. Will change 👍

[AlenkaF]

The issue on the failing build is

Running Sphinx v6.2.0

Exception occurred:
  File "/opt/conda/envs/arrow/lib/python3.9/site-packages/breathe/renderer/sphinxrenderer.py", line 104, in DomainDirectiveFactory
    'function': (python.PyModulelevel, 'function'),
AttributeError: module 'sphinx.domains.python' has no attribute 'PyModulelevel'

but Sphinx deprecated sphinx.domains.python.PyModulelevel in 4.0, so this should have come up before already?
https://www.sphinx-doc.org/en/master/extdev/deprecated.html

[jorisvandenbossche]

Hmm, that should already have been removed in breathe a long time ago: breathe-doc/breathe#512 (released in breathe 4.17.0)
What version of breathe do we have installed?

[jorisvandenbossche]

OK, so we have an old version of breathe (4.15), this might be because more recent versions on conda-forge have an upper pin to sphinx: https://github.com/conda-forge/breathe-feedstock/blob/e22b71d442aa41ed3e7614eddb4d129c893bacab/recipe/meta.yaml#L28

[jorisvandenbossche]

@AlenkaF you can maybe open a PR on the conda feedstock to update the upper pin (eg to <7), since it seems breathe itself is now testing with sphinx 6 (breathe-doc/breathe@ef4e245), so that should be expected to work

[AlenkaF]

Sure will do! 👍

[AlenkaF]

Here it is: conda-forge/breathe-feedstock#59

[AlenkaF]

There is no issue with the docs build on the CI after the PR on conda-forge/breathe-feedstock got merged 👍 The version of Sphinx used on the CI is 6.2.0 and the version of Breathe is 4.35.0.

Note: in the last commit I have added a make option for MacOS related issues with editable libs. I am not sure if the approach is correct though.

[conbench-apache-arrow]

Conbench analyzed the 6 benchmark runs on commit 82bf1e28.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.