Incompatible with Towncrier 24.7

[hynek]

Looks like the just-released Towncrier 24.7 has broken some API you're using:

Running Sphinx v7.4.7
loading translations [en]... done
Converting `source_suffix = '.rst'` to `source_suffix = {'.rst': 'restructuredtext'}`.
myst v3.0.1: MdParserConfig(commonmark_only=False, gfm_only=False, enable_extensions={'smartquotes', 'colon_fence', 'deflist'}, disable_syntax=[], all_links_external=False, links_external_new_tab=False, url_schemes=('http', 'https', 'mailto', 'ftp'), ref_domains=None, fence_as_directive=set(), number_code_blocks=[], title_to_header=False, heading_anchors=0, heading_slug_func=None, html_meta={}, footnote_transition=True, words_per_minute=200, substitutions={}, linkify_fuzzy_links=True, dmath_allow_labels=True, dmath_allow_space=True, dmath_allow_digits=True, dmath_double_inline=False, update_mathjax=True, mathjax_classes='tex2jax_process|mathjax_process|math|output_area', enable_checkboxes=False, suppress_warnings=[], highlight_code_blocks=True)
loading intersphinx inventory 'python' from https://docs.python.org/3/objects.inv...
building [mo]: targets for 0 po files that are out of date
writing output...
building [html]: targets for 16 source files that are out of date

Traceback (most recent call last):
  File "/Users/hynek/FOSS/attrs/.tox/docs/lib/python3.12/site-packages/sphinx/events.py", line 98, in emit
    results.append(listener.handler(self.app, *args))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hynek/FOSS/attrs/.tox/docs/lib/python3.12/site-packages/sphinxcontrib/towncrier/ext.py", line 306, in get_outdated_docs
    towncrier_fragment_paths = lookup_towncrier_fragments(
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hynek/FOSS/attrs/.tox/docs/lib/python3.12/site-packages/sphinxcontrib/towncrier/_fragment_discovery.py", line 78, in lookup_towncrier_fragments
    _fragments, fragment_filenames = find_fragments(
                                     ^^^^^^^^^^^^^^^
TypeError: find_fragments() takes 3 positional arguments but 4 were given

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/hynek/FOSS/attrs/.tox/docs/lib/python3.12/site-packages/sphinx/cmd/build.py", line 337, in build_main
    app.build(args.force_all, args.filenames)
  File "/Users/hynek/FOSS/attrs/.tox/docs/lib/python3.12/site-packages/sphinx/application.py", line 378, in build
    self.builder.build_update()
  File "/Users/hynek/FOSS/attrs/.tox/docs/lib/python3.12/site-packages/sphinx/builders/__init__.py", line 297, in build_update
    self.build(to_build,
  File "/Users/hynek/FOSS/attrs/.tox/docs/lib/python3.12/site-packages/sphinx/builders/__init__.py", line 318, in build
    updated_docnames = set(self.read())
                           ^^^^^^^^^^^
  File "/Users/hynek/FOSS/attrs/.tox/docs/lib/python3.12/site-packages/sphinx/builders/__init__.py", line 389, in read
    for docs in self.events.emit('env-get-outdated', self.env, added, changed, removed):
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hynek/FOSS/attrs/.tox/docs/lib/python3.12/site-packages/sphinx/events.py", line 109, in emit
    raise ExtensionError(
sphinx.errors.ExtensionError: Handler <bound method TowncrierDraftEntriesEnvironmentCollector.get_outdated_docs of <sphinxcontrib.towncrier.ext.TowncrierDraftEntriesEnvironmentCollector object at 0x10583e180>> for event 'env-get-outdated' threw an exception (exception: find_fragments() takes 3 positional arguments but 4 were given)

Extension error (sphinxcontrib.towncrier.ext):
Handler <bound method TowncrierDraftEntriesEnvironmentCollector.get_outdated_docs of <sphinxcontrib.towncrier.ext.TowncrierDraftEntriesEnvironmentCollector object at 0x10583e180>> for event 'env-get-outdated' threw an exception (exception: find_fragments() takes 3 positional arguments but 4 were given)

See https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:29 for full CI failure.

[8](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:29)
docs: commands[0]> sphinx-build -n -T -W -b html -d /home/runner/work/attrs/attrs/.tox/docs/tmp/doctrees docs docs/_build/html
[29](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:30)
Running Sphinx v7.4.7
[30](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:31)
loading translations [en]... done
[31](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:32)
making output directory... done
[32](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:33)
Converting `source_suffix = '.rst'` to `source_suffix = {'.rst': 'restructuredtext'}`.
myst v3.0.1: MdParserConfig(commonmark_only=False, gfm_only=False, enable_extensions={'smartquotes', 'colon_fence', 'deflist'}, disable_syntax=[], all_links_external=False, links_external_new_tab=False, url_schemes=('http', 'https', 'mailto', 'ftp'), ref_domains=None, fence_as_directive=set(), number_code_blocks=[], title_to_header=False, heading_anchors=0, heading_slug_func=None, html_meta={}, footnote_transition=True, words_per_minute=200, substitutions={}, linkify_fuzzy_links=True, dmath_allow_labels=True, dmath_allow_space=True, dmath_allow_digits=True, dmath_double_inline=False, update_mathjax=True, mathjax_classes='tex2jax_process|mathjax_process|math|output_area', enable_checkboxes=False, suppress_warnings=[], highlight_code_blocks=True)
[34](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:35)
loading intersphinx inventory 'python' from https://docs.python.org/3/objects.inv...
[35](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:36)
building [mo]: targets for 0 po files that are out of date
[36](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:37)
writing output... 
[37](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:38)
building [html]: targets for 16 source files that are out of date
[38](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:39)

[39](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:40)
Traceback (most recent call last):
[40](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:41)
  File "/home/runner/work/attrs/attrs/.tox/docs/lib/python3.12/site-packages/sphinx/events.py", line 98, in emit
[41](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:42)
    results.append(listener.handler(self.app, *args))
[42](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:43)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[43](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:44)
  File "/home/runner/work/attrs/attrs/.tox/docs/lib/python3.12/site-packages/sphinxcontrib/towncrier/ext.py", line 306, in get_outdated_docs
[44](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:45)
    towncrier_fragment_paths = lookup_towncrier_fragments(
[45](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:46)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[46](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:47)
  File "/home/runner/work/attrs/attrs/.tox/docs/lib/python3.12/site-packages/sphinxcontrib/towncrier/_fragment_discovery.py", line 78, in lookup_towncrier_fragments
[47](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:48)
    _fragments, fragment_filenames = find_fragments(
[48](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:49)
                                     ^^^^^^^^^^^^^^^
[49](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:50)
TypeError: find_fragments() takes 3 positional arguments but 4 were given
[50](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:51)

[51](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:52)
The above exception was the direct cause of the following exception:
[52](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:53)

[53](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:54)
Traceback (most recent call last):
[54](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:55)
  File "/home/runner/work/attrs/attrs/.tox/docs/lib/python3.12/site-packages/sphinx/cmd/build.py", line 337, in build_main
[55](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:56)
    app.build(args.force_all, args.filenames)
[56](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:57)
  File "/home/runner/work/attrs/attrs/.tox/docs/lib/python3.12/site-packages/sphinx/application.py", line 378, in build
[57](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:58)
    self.builder.build_update()
[58](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:59)
  File "/home/runner/work/attrs/attrs/.tox/docs/lib/python3.12/site-packages/sphinx/builders/__init__.py", line 297, in build_update
[59](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:60)
    self.build(to_build,
[60](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:61)
  File "/home/runner/work/attrs/attrs/.tox/docs/lib/python3.12/site-packages/sphinx/builders/__init__.py", line 318, in build
[61](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:62)
    updated_docnames = set(self.read())
[62](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:63)
                           ^^^^^^^^^^^
[63](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:64)
  File "/home/runner/work/attrs/attrs/.tox/docs/lib/python3.12/site-packages/sphinx/builders/__init__.py", line 389, in read
[64](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:65)
    for docs in self.events.emit('env-get-outdated', self.env, added, changed, removed):
[65](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:66)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[66](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:67)
  File "/home/runner/work/attrs/attrs/.tox/docs/lib/python3.12/site-packages/sphinx/events.py", line 109, in emit
[67](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:68)
    raise ExtensionError(
[68](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:69)
sphinx.errors.ExtensionError: Handler <bound method TowncrierDraftEntriesEnvironmentCollector.get_outdated_docs of <sphinxcontrib.towncrier.ext.TowncrierDraftEntriesEnvironmentCollector object at 0x7faecfa3a510>> for event 'env-get-outdated' threw an exception (exception: find_fragments() takes 3 positional arguments but 4 were given)
[69](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:70)

[70](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:71)
Extension error (sphinxcontrib.towncrier.ext):
[71](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:72)
Handler <bound method TowncrierDraftEntriesEnvironmentCollector.get_outdated_docs of <sphinxcontrib.towncrier.ext.TowncrierDraftEntriesEnvironmentCollector object at 0x7faecfa3a510>> for event 'env-get-outdated' threw an exception (exception: find_fragments() takes 3 positional arguments but 4 were given)
[72](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:73)
updating environment: docs: exit 2 (2.45 seconds) /home/runner/work/attrs/attrs> sphinx-build -n -T -W -b html -d /home/runner/work/attrs/attrs/.tox/docs/tmp/doctrees docs docs/_build/html pid=1877
[73](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:74)
.pkg: _exit> python /opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/pyproject_api/_backend.py True hatchling.build
[74](https://github.com/python-attrs/attrs/actions/runs/10176118290/job/28144823882#step:6:75)
docs: FAIL ✖ in 3.83 seconds

[jaraco]

This issue also affecting pypa/setuptools#4533.

[webknjaz]

Thanks for notifying me!

[commonism]

Would it be possible to create a release which limits the version of the towncrier dependency?

[webknjaz]

@commonism it'd be pointless as pip's depresolver would just backtrack to the previous version.

[dolfinus]

@commonism it'd be pointless as pip's depresolver would just backtrack to the previous version.

Pip does not do that

[webknjaz]

I can assure you, with my PyPA hat on, that pip's dependency resolver (resolvelib) does exactly that in exactly this corner case, unless the end-user adds a constraint that would prevent it from considering those versions, which is already possible, anyway.

[dolfinus]

If sphinxcontrib-towncrier==0.4.1 will have a constraint towncrier<24.7, and user installs sphinxcontrib-towncrier without any explicit version constraint, then pip will install sphinxcontrib-towncrier==0.4.1, towncrier==23.11.0. I don't see why pip will apply some kind of backtracking in this case.

Of course, things may be more complicated if some other package passed to pip install depends on other towncrier version, or specific package version is not available on current environment (Python version and so on).

[webknjaz]

That's it. You still have to manually constrain the user request on what to install. But if you don't — backtracking happens.
OTOH, you can already constrain it now, even without a release with different metadata. And arguably, you should pin your entire environment, not just bits and pieces: https://hynek.me/articles/semver-will-not-save-you/#taking-responsibility.

That said, I don't see how this would be different with or without a release just bumping the metadata. When I find time to work on this, I won't be looking into trivial stop-gaps and will just fix the incompatibility 🤷‍♂️ It doesn't make sense to me to spend that time twice.

[bennyrowland]

@webknjaz I was just about to offer some help on fixing this issue but it looks like you have something planned for tomorrow already :-)
In case you haven't looked into this in any detail yet I will summarise my progress so far: from a quick review of the update to Towncrier they have switched the find_fragments() function to taking in the Config dataclass, rather than only some components from it. I note that in this project currently the Towncrier config is only used in the lookup_towncrier_fragments() function. At the moment it is loaded through _towncrier.get_towncrier_config() which has various contortions to cope with different Towncrier versions, including converting the dataclass Config object (since Towncrier 22.12.0rc1) back into the dict it used to be. However, since only the one lookup_towncrier_fragments() uses this function, it should be safe to allow the function to return the dataclass instead.

There are basically then three pathways based on the version of Towncrier:

1. Towncrier >= 24.7.0: call new find_fragments() with the dataclass config
2. 22.12.0rc1 <= Towncrier < 24.7.0: call old find_fragments() with attributes of the dataclass config (or convert it to a dict and go for option 3.)
3. Towncrier < 22.12.0rc1: call old find_fragments() with parameters extracted from the dict config

The main question is how to distinguish pathway 1 from 2: checking for the current TypeError is not ideal because if another parameter is added to the new find_fragments() in a future Towncrier version then it will break. We could use importlib.metadata as advocated here https://setuptools-scm.readthedocs.io/en/latest/usage/ if you are happy to go to Python>=3.8 or add the backport as a dependency.

Happy to put together a PR if that would be helpful, just let me know your thoughts.

[webknjaz]

@bennyrowland well, I added it on the board of the aiohttp sprint because it's used in the infra and it would be nice to have. IIRC I was trying hard to keep Python 3.6 support for one final release and was planning to drop all EOL versions right after, shrinking the matrix by a lot. But I didn't take into account some bits of this repos infra and broke it. Never had time to pick it up.
Feel free to work on this, I do not think if someone would be able to do so tomorrow anyway, just trying to fill the board with possibilities.
FWIW, seeing that I don't have as much time as I'd like (I'm writing this at VIE awaiting my flight to LHR to lead that Man Group hackathon sprint on aiohttp), it's okay to drop all EOL versions already.
I recommend starting with an infra cleanup/fix PR, followed by the new towncrier compat. It could be more PRs — the smaller it is the higher the chance of a quick review.
Thanks for offering to help!
If you're not going to work on this today, I recommend waiting until Saturday noon to see if someone picks it up. Otherwise, I can assign you for visibility whenever.