"ValueError: At least one file selection option must be defined" error with hatchling-v1.19.0

[tleyden]

Did hatchling-v1.19.0 introduce breaking changes? My project is suddenly failing to build with this error:

# pip install -U '.[dev]'
Processing project_dir
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [30 lines of output]
      Traceback (most recent call last):
        File "/usr/local/lib/python3.11/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/usr/local/lib/python3.11/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/local/lib/python3.11/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 152, in prepare_metadata_for_build_wheel
          whl_basename = backend.build_wheel(metadata_directory, config_settings)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-n8gm4qgc/overlay/local/lib/python3.11/dist-packages/hatchling/build.py", line 58, in build_wheel
          return os.path.basename(next(builder.build(directory=wheel_directory, versions=['standard'])))
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-n8gm4qgc/overlay/local/lib/python3.11/dist-packages/hatchling/builders/plugin/interface.py", line 155, in build
          artifact = version_api[version](directory, **build_data)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-n8gm4qgc/overlay/local/lib/python3.11/dist-packages/hatchling/builders/wheel.py", line 405, in build_standard
          for included_file in self.recurse_included_files():
        File "/tmp/pip-build-env-n8gm4qgc/overlay/local/lib/python3.11/dist-packages/hatchling/builders/plugin/interface.py", line 176, in recurse_included_files
          yield from self.recurse_selected_project_files()
        File "/tmp/pip-build-env-n8gm4qgc/overlay/local/lib/python3.11/dist-packages/hatchling/builders/plugin/interface.py", line 180, in recurse_selected_project_files
          if self.config.only_include:
             ^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-n8gm4qgc/overlay/local/lib/python3.11/dist-packages/hatchling/builders/config.py", line 774, in only_include
          only_include = only_include_config.get('only-include', self.default_only_include()) or self.packages
                                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-n8gm4qgc/overlay/local/lib/python3.11/dist-packages/hatchling/builders/wheel.py", line 231, in default_only_include
          self.set_default_file_selection()
        File "/tmp/pip-build-env-n8gm4qgc/overlay/local/lib/python3.11/dist-packages/hatchling/builders/wheel.py", line 209, in set_default_file_selection
          raise ValueError(message)
      ValueError: At least one file selection option must be defined, see: https://hatch.pypa.io/latest/config/build/
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

I tried the following workaround to revert the version to hatchling-v1.18.0, which fixed the error:

[build-system]
requires = ["hatchling<=1.18.0"]  
build-backend = "hatchling.build"

[pquentin]

Yes, this is mentioned in the changelog: https://github.com/pypa/hatch/releases/tag/hatchling-v1.19.0.

An error will now be raised for the wheel target if no file selection options are defined

File selection options are documented here: https://hatch.pypa.io/latest/config/build/#file-selection

I'm wondering what exact rules apply, though. urllib3 does not seem to be affected as it defines include in [tool.hatch.build.targets.sdist] while rally-tracks is affected while defining exclude in [tool.hatch.build.targets.sdist], which looks like a file selection option. Is include needed in that case?

[ofek]

This is specifically for the wheel target. Previously if projects were building wheels successfully then you were hitting the fall back and basically including every Python file found that doesn't look like a test file. That I think is undesirable which is why there is now an error.

I will keep this open and improve that message to indicate that issue is specific to wheel config.

[ofek]

@pquentin https://hatch.pypa.io/latest/plugins/builder/wheel/#default-file-selection

[sbidoul]

@ofek Users are hitting this when using my hatch-odoo plugin where no file selection is provided in pyproject.toml, but the plugin does the selection with force_include (with a mapping from source directory to target directory in the wheel) like this:

https://github.com/acsone/hatch-odoo/blob/d1fae86c0522ffdc368db979ac2eab294e108a9d/src/hatch_odoo/build_hook.py#L27-L36

Should I adapt the plugin?

[ofek]

Does the source directory exist?

[sbidoul]

Does the source directory exist?

Yes all the source directories I add in the force_include dict exist.

[ofek]

Do you have an example traceback?

[sbidoul]

Sure here it is:

      Traceback (most recent call last):
        File "/home/thomas/.virtualenvs/migration-odoo-silog/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/thomas/.virtualenvs/migration-odoo-silog/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/thomas/.virtualenvs/migration-odoo-silog/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 152, in prepare_metadata_for_build_wheel
          whl_basename = backend.build_wheel(metadata_directory, config_settings)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-s7eqwz1i/overlay/lib/python3.11/site-packages/hatchling/build.py", line 58, in build_wheel
          return os.path.basename(next(builder.build(directory=wheel_directory, versions=['standard'])))
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-s7eqwz1i/overlay/lib/python3.11/site-packages/hatchling/builders/plugin/interface.py", line 154, in build
          with self.config.set_build_data(build_data):
        File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
          return next(self.gen)
                 ^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-s7eqwz1i/overlay/lib/python3.11/site-packages/hatchling/builders/config.py", line 900, in set_build_data
          self.build_reserved_paths.add(self.get_distribution_path(os.path.relpath(source, self.root)))
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-s7eqwz1i/overlay/lib/python3.11/site-packages/hatchling/builders/config.py", line 803, in get_distribution_path
          for source, replacement in self.sources.items():
                                     ^^^^^^^^^^^^
        File "/tmp/pip-build-env-s7eqwz1i/overlay/lib/python3.11/site-packages/hatchling/builders/config.py", line 693, in sources
          for relative_path in self.packages:
                               ^^^^^^^^^^^^^
        File "/tmp/pip-build-env-s7eqwz1i/overlay/lib/python3.11/site-packages/hatchling/builders/config.py", line 712, in packages
          packages = package_config.get('packages', self.default_packages())
                                                    ^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-s7eqwz1i/overlay/lib/python3.11/site-packages/hatchling/builders/wheel.py", line 225, in default_packages
          self.set_default_file_selection()
        File "/tmp/pip-build-env-s7eqwz1i/overlay/lib/python3.11/site-packages/hatchling/builders/wheel.py", line 209, in set_default_file_selection
          raise ValueError(message)
      ValueError: At least one file selection option must be defined, see: https://hatch.pypa.io/latest/config/build/

[ofek]

Unless I'm misunderstanding, it looks like that is a valid error and has nothing to do with your plugin. What am I missing?

FYI for everyone here, this behavior changed due to multiple requests one of which was this: #992

[sbidoul]

Unless I'm misunderstanding, it looks like that is a valid error and has nothing to do with your plugin. What am I missing?

The error appeared with hatchling 1.19. My point is that I have no file selection option that I can provide in pyproject.toml since all the file selection is done by the hatch-odoo plugin.

[ofek]

I see. So the fix (for your particular case) would be to prevent this check if a plugin does provide forced inclusion patterns, is that right? I plan to fix this and release today so I want to be sure.

[sbidoul]

prevent this check if a plugin does provide forced inclusion patterns

@ofek, yes that sounds good.

Note however that I tried to add this to my config as a workaround and it still fails with the same error, there might be something else at play:

[tool.hatch.build.targets.wheel.force-include]
"pyproject.toml" = "./pyproject.toml"

[hugovk]

We're also hitting this with Sphinx Lint:

❯ git clone https://github.com/sphinx-contrib/sphinx-lint && cd sphinx-lint
...
❯ pip install -e .
Obtaining file:///private/tmp/sphinx-lint
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing editable metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [39 lines of output]
      Traceback (most recent call last):
        File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 167, in prepare_metadata_for_build_editable
          hook = backend.prepare_metadata_for_build_editable
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      AttributeError: module 'hatchling.build' has no attribute 'prepare_metadata_for_build_editable'

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 176, in prepare_metadata_for_build_editable
          whl_basename = build_hook(metadata_directory, config_settings)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/p6/lf2s1s5d4kb335g2n1td8z8c0000gn/T/pip-build-env-z5yx9mlz/overlay/lib/python3.12/site-packages/hatchling/build.py", line 82, in build_editable
          return os.path.basename(next(builder.build(directory=wheel_directory, versions=['editable'])))
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/p6/lf2s1s5d4kb335g2n1td8z8c0000gn/T/pip-build-env-z5yx9mlz/overlay/lib/python3.12/site-packages/hatchling/builders/plugin/interface.py", line 155, in build
          artifact = version_api[version](directory, **build_data)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/p6/lf2s1s5d4kb335g2n1td8z8c0000gn/T/pip-build-env-z5yx9mlz/overlay/lib/python3.12/site-packages/hatchling/builders/wheel.py", line 423, in build_editable
          return self.build_editable_detection(directory, **build_data)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/p6/lf2s1s5d4kb335g2n1td8z8c0000gn/T/pip-build-env-z5yx9mlz/overlay/lib/python3.12/site-packages/hatchling/builders/wheel.py", line 434, in build_editable_detection
          for included_file in self.recurse_selected_project_files():
        File "/private/var/folders/p6/lf2s1s5d4kb335g2n1td8z8c0000gn/T/pip-build-env-z5yx9mlz/overlay/lib/python3.12/site-packages/hatchling/builders/plugin/interface.py", line 180, in recurse_selected_project_files
          if self.config.only_include:
             ^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/p6/lf2s1s5d4kb335g2n1td8z8c0000gn/T/pip-build-env-z5yx9mlz/overlay/lib/python3.12/site-packages/hatchling/builders/config.py", line 774, in only_include
          only_include = only_include_config.get('only-include', self.default_only_include()) or self.packages
                                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/p6/lf2s1s5d4kb335g2n1td8z8c0000gn/T/pip-build-env-z5yx9mlz/overlay/lib/python3.12/site-packages/hatchling/builders/wheel.py", line 231, in default_only_include
          self.set_default_file_selection()
        File "/private/var/folders/p6/lf2s1s5d4kb335g2n1td8z8c0000gn/T/pip-build-env-z5yx9mlz/overlay/lib/python3.12/site-packages/hatchling/builders/wheel.py", line 209, in set_default_file_selection
          raise ValueError(message)
      ValueError: At least one file selection option must be defined, see: https://hatch.pypa.io/latest/config/build/
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Similarly for pip install ., and so it also fails to install as a pre-commit hook (which uses pip install . from a Git clone).

It's fine with Hatchling 1.18.

Sphinx Lint also uses hatch-vcs: https://github.com/sphinx-contrib/sphinx-lint/blob/main/pyproject.toml

[hugovk]

Adding this fixes it:

[tool.hatch.build.targets.wheel]
packages = ["sphinx-lint"]

• hugovk/sphinx-lint@62aebe3
• https://github.com/hugovk/sphinx-lint/actions/runs/7171624448

[ofek]

I'm so sorry all of these errors are happening but it indicates to me that the users who asked for this were indeed correct and that people have been shipping wheels improperly in a subset of cases. In your case, the project name:

https://github.com/sphinx-contrib/sphinx-lint/blob/74db402328abca2bb1f1a72950b28ca1fee0d53c/pyproject.toml#L9

does not match the directory name (no hyphen) so as a fallback it tries every Python file that does not look like a test.

[hugovk]

Ah right, so we actually need this without a hyphen to get wheels containing source and not just metadata:

[tool.hatch.build.targets.wheel]
packages = ["sphinxlint"]

Thanks, this now makes it create wheels identical to those with Hatchling 1.18.

[CAM-Gerlach]

does not match the directory name (no hyphen) so as a fallback it tries every Python file that does not look like a test.

Does the logic at least check for the common trivial case of distribution-package-name -> import_package_name?

[ofek]

Yes it does, the logic can be found here: https://github.com/pypa/hatch/blob/hatchling-v1.19.0/backend/src/hatchling/builders/wheel.py#L177-L209

[chemelnucfin]

Can you describe how to update the pyproject.toml to get the default old behavior or otherwise the recommended new default behavior please?

[ofek]

Yes certainly! The old behavior is:

[tool.hatch.build.targets.wheel]
include = ["*.py"]
exclude = ["test*"]

I'm embarrassed that I thought that was a good fallback. I kinda just copied what everyone else did for the fallback but in hindsight it should have always been an error to protect the user.

The new recommendation is simply to define any inclusion options you want. The most common will be packages, for example:

[tool.hatch.build.targets.wheel]
packages = ["src/foo"]

Of course, in the majority of cases configuration is unnecessary because the project name would fuzzy match the directory name.

[jorng]

This should really be called out in the documentation under the build config, specifically as a change.

This was a breaking change, and should have been versioned as such.

[ofek]

I do understand that viewpoint and I went back and forth on whether or not to ship a new major version. I decided not to based on two factors:

1. I searched for uses of Hatchling on GitHub and it was <1% that actually pinned or had a version cap so this would not have helped very many people.
2. This is also a bug fix. What you don't see in this issue is the dozens of other people that I had to help and also I've seen when searching for issues that experienced confusion about why their builds aren't behaving as expected or even users complaining to the maintainers that the already-shipped distribution is incorrect.

I do however understand that this sucks, and for that I am genuinely sorry.

[ofek]

Part one of fixing this issue: #1119

The final second part is an actual bug fix to help with the plugin that was discussed above

[ofek]

https://github.com/pypa/hatch/releases/tag/hatchling-v1.19.1

[ofek]

Real life example of the new error 🙂 https://github.com/python/docs-community/actions/runs/7177951476/job/19545271892?pr=93#step:4:171