Regression in 7.4: jinja2.exceptions.TemplateNotFound error from jinja2glue

[lucascolley]

Describe the bug

x-ref pydata/pydata-sphinx-theme#1930

---

This started showing up in SciPy CI in the past day or so. x-ref scipy/scipy#21195.

Traceback (most recent call last):
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sphinx/events.py", line 98, in emit
    results.append(listener.handler(self.app, *args))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/pydata_sphinx_theme/__init__.py", line 189, in update_and_remove_templates
    context[section] = utils._update_and_remove_templates(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/pydata_sphinx_theme/utils.py", line 139, in _update_and_remove_templates
    rendered = app.builder.templates.render(template, ctx)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sphinx/jinja2glue.py", line 200, in render
    return self.environment.get_template(template).render(context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/jinja2/environment.py", line 1013, in get_template
    return self._load_template(name, globals)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/jinja2/environment.py", line 972, in _load_template
    template = self.loader.load(self, name, self.make_globals(globals))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/jinja2/loaders.py", line 126, in load
    source, filename, uptodate = self.get_source(environment, name)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sphinx/jinja2glue.py", line 222, in get_source
    raise TemplateNotFound(msg)
jinja2.exceptions.TemplateNotFound: 's.html' not found in ['/home/circleci/repo/doc/source/_templates', '/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/pydata_sphinx_theme/theme/pydata_sphinx_theme/components', '/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/pydata_sphinx_theme/theme/pydata_sphinx_theme', '/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sphinx/themes/basic']

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sphinx/cmd/build.py", line 337, in build_main
    app.build(args.force_all, args.filenames)
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sphinx/application.py", line 378, in build
    self.builder.build_update()
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 297, in build_update
    self.build(to_build,
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 367, in build
    self.write(docnames, list(updated_docnames), method)
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 614, in write
    self._write_serial(sorted(docnames))
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 624, in _write_serial
    self.write_doc(docname, doctree)
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sphinx/builders/html/__init__.py", line 666, in write_doc
    self.handle_page(docname, ctx, event_arg=doctree)
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sphinx/builders/html/__init__.py", line 1101, in handle_page
    newtmpl = self.app.emit_firstresult('html-page-context', pagename,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sphinx/application.py", line 515, in emit_firstresult
    return self.events.emit_firstresult(event, *args,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sphinx/events.py", line 124, in emit_firstresult
    for result in self.emit(name, *args, allowed_exceptions=allowed_exceptions):
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sphinx/events.py", line 109, in emit
    raise ExtensionError(
sphinx.errors.ExtensionError: Handler <function update_and_remove_templates at 0x7fad10c52a20> for event 'html-page-context' threw an exception (exception: 's.html' not found in ['/home/circleci/repo/doc/source/_templates', '/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/pydata_sphinx_theme/theme/pydata_sphinx_theme/components', '/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/pydata_sphinx_theme/theme/pydata_sphinx_theme', '/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sphinx/themes/basic'])

Extension error (pydata_sphinx_theme):
Handler <function update_and_remove_templates at 0x7fad10c52a20> for event 'html-page-context' threw an exception (exception: 's.html' not found in ['/home/circleci/repo/doc/source/_templates', '/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/pydata_sphinx_theme/theme/pydata_sphinx_theme/components', '/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/pydata_sphinx_theme/theme/pydata_sphinx_theme', '/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sphinx/themes/basic'])
make: *** [Makefile:114: html-build] Error 2
Skipping build```

How to Reproduce

The line in PST looks to be app.builder.templates.render(template, ctx). Can look into this further if needed.

Environment Information

I don't have this info right now but we can look into it if needed.

Sphinx extensions

No response

Additional context

No response

[AA-Turner]

Thanks @lucascolley for reporting.

The root cause is the following lines in SciPy's conf.py, which sets a value of html_sidebars to a string.

html_sidebars = {
    "index": "search-button-field",
    "**": ["search-button-field", "sidebar-nav-bs"]
}

Support for this was removed in Sphinx 2.0 (though I'll admit the wording is poor in our docs) (see #4178, a7cab00, #5370).

In a recent commit (f10f0b6, #12578), I coerced the value of the matched value from html_sidebars to a list, as users might pass in a tuple or some other data structure. This ended up transforming "search-button-field" to ['s', 'e', 'a', 'r', 'c', 'h', '-', 'b', 'u', 't', 't', 'o', 'n', '-', 'f', 'i', 'e', 'l', 'd'], and then an event hook in pydata-sphinx-theme attempted to render a template for each of these elements, clearly failing when no such s.html Jinja template was found.

The fix is to revert to the status quo ante (no type coercion) for now and adds a warning error message instructing the user what to do when they encounter this problem. EDIT: See #12600

I will also open a PR to SciPy to fix the upstream issue. EDIT: See scipy/scipy#21200

A

[lucascolley]

Thanks @AA-Turner, makes sense!

[AA-Turner]

Sphinx 7.4.5 has been released with a fix.

A