[BUG] TypeError when an empty .dist-info directory is present when running setup.py sdist

[a68366]

setuptools version

setuptools==62.1.0

Python version

3.10

OS

Fedora 35

Additional environment information

The error only occurs with setuptools>=60.9.0

E.g. setuptools==60.8.2 is fine

Description

When project directory contains an empty .dist-info directory python setup.py sdist fails with TypeError.

Expected behavior

It should probably work as it worked before setuptools>=60.9.0?

How to Reproduce

1. Create an empty .dist-info directory alongside the package directory
2. Run python setup.py sdist

tree -a:

.
├── .dist-info
├── my_pkg
│   └── __init__.py
└── setup.py

setup.py:

from setuptools import setup, find_packages

setup(
    name='test',
    version='1.0.0',
    url='https://example.org',
    author='test',
    author_email='[email protected]',
    description='test',
    packages=find_packages(),    
)

Output

$  python setup.py sdist
Traceback (most recent call last):
  File "/tmp/123/setup.py", line 3, in <module>
    setup(
  File "/home/user/.local/lib/python3.10/site-packages/setuptools/__init__.py", line 86, in setup
    _install_setup_requires(attrs)
  File "/home/user/.local/lib/python3.10/site-packages/setuptools/__init__.py", line 75, in _install_setup_requires
    dist = MinimalDistribution(attrs)
  File "/home/user/.local/lib/python3.10/site-packages/setuptools/__init__.py", line 57, in __init__
    super().__init__(filtered)
  File "/home/user/.local/lib/python3.10/site-packages/setuptools/dist.py", line 460, in __init__
    for ep in metadata.entry_points(group='distutils.setup_keywords'):
  File "/usr/lib64/python3.10/importlib/metadata/__init__.py", line 1009, in entry_points
    return SelectableGroups.load(eps).select(**params)
  File "/usr/lib64/python3.10/importlib/metadata/__init__.py", line 459, in load
    ordered = sorted(eps, key=by_group)
  File "/usr/lib64/python3.10/importlib/metadata/__init__.py", line 1006, in <genexpr>
    eps = itertools.chain.from_iterable(
  File "/usr/lib64/python3.10/importlib/metadata/_itertools.py", line 16, in unique_everseen
    k = key(element)
  File "/usr/lib64/python3.10/importlib/metadata/__init__.py", line 941, in _normalized_name
    return self._name_from_stem(stem) or super()._normalized_name
  File "/usr/lib64/python3.10/importlib/metadata/__init__.py", line 622, in _normalized_name
    return Prepared.normalize(self.name)
  File "/usr/lib64/python3.10/importlib/metadata/__init__.py", line 871, in normalize
    return re.sub(r"[-_.]+", "-", name).lower().replace('-', '_')
  File "/usr/lib64/python3.10/re.py", line 209, in sub
    return _compile(pattern, flags).sub(repl, string, count)
TypeError: expected string or bytes-like object

[abravalheri]

Hi @a68366 thanks for reporting this.

What happens when you use python -m build --sdist (need build to be installed) instead of python setup.py sdist?

I am curious to try to understand how you ended up with an empty .dist-info folder...

[a68366]

Hello @abravalheri

I just happened to need to build this project, which for some reason had a .dist-info folder in the repo.

$ python -m build --sdist
* Creating venv isolated environment...
* Installing packages in isolated environment... (setuptools >= 40.8.0, wheel)
* Getting dependencies for sdist...
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 363, in <module>
    main()
  File "/home/user/.local/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 345, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/home/user/.local/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 297, in get_requires_for_build_sdist
    return hook(config_settings)
  File "/tmp/build-env-6zwcyby4/lib64/python3.10/site-packages/setuptools/build_meta.py", line 181, in get_requires_for_build_sdist
    return self._get_build_requires(config_settings, requirements=[])
  File "/tmp/build-env-6zwcyby4/lib64/python3.10/site-packages/setuptools/build_meta.py", line 159, in _get_build_requires
    self.run_setup()
  File "/tmp/build-env-6zwcyby4/lib64/python3.10/site-packages/setuptools/build_meta.py", line 281, in run_setup
    super(_BuildMetaLegacyBackend,
  File "/tmp/build-env-6zwcyby4/lib64/python3.10/site-packages/setuptools/build_meta.py", line 174, in run_setup
    exec(compile(code, __file__, 'exec'), locals())
  File "setup.py", line 3, in <module>
    setup(
  File "/tmp/build-env-6zwcyby4/lib64/python3.10/site-packages/setuptools/__init__.py", line 86, in setup
    _install_setup_requires(attrs)
  File "/tmp/build-env-6zwcyby4/lib64/python3.10/site-packages/setuptools/__init__.py", line 75, in _install_setup_requires
    dist = MinimalDistribution(attrs)
  File "/tmp/build-env-6zwcyby4/lib64/python3.10/site-packages/setuptools/__init__.py", line 57, in __init__
    super().__init__(filtered)
  File "/tmp/build-env-6zwcyby4/lib64/python3.10/site-packages/setuptools/dist.py", line 460, in __init__
    for ep in metadata.entry_points(group='distutils.setup_keywords'):
  File "/usr/lib64/python3.10/importlib/metadata/__init__.py", line 1009, in entry_points
    return SelectableGroups.load(eps).select(**params)
  File "/usr/lib64/python3.10/importlib/metadata/__init__.py", line 459, in load
    ordered = sorted(eps, key=by_group)
  File "/usr/lib64/python3.10/importlib/metadata/__init__.py", line 1006, in <genexpr>
    eps = itertools.chain.from_iterable(
  File "/usr/lib64/python3.10/importlib/metadata/_itertools.py", line 16, in unique_everseen
    k = key(element)
  File "/usr/lib64/python3.10/importlib/metadata/__init__.py", line 941, in _normalized_name
    return self._name_from_stem(stem) or super()._normalized_name
  File "/usr/lib64/python3.10/importlib/metadata/__init__.py", line 622, in _normalized_name
    return Prepared.normalize(self.name)
  File "/usr/lib64/python3.10/importlib/metadata/__init__.py", line 871, in normalize
    return re.sub(r"[-_.]+", "-", name).lower().replace('-', '_')
  File "/usr/lib64/python3.10/re.py", line 209, in sub
    return _compile(pattern, flags).sub(repl, string, count)
TypeError: expected string or bytes-like object

ERROR Backend subproccess exited when trying to invoke get_requires_for_build_sdist

[abravalheri]

I don't know exactly what is the best way of proceeding here...

I wouldn't say this is a problem with setuptools, but rather a consequence of how Python itself and its standard library are designed to operate.

For example, if I try to use importlib.metadata in a context that is different from packaging, I can see the same error:

mkdir -p /tmp/quicktest/.dist-info
cd /tmp/quicktest/
python3.10
# Python 3.10.4 (main, Mar 24 2022, 16:12:56) [GCC 9.4.0] on linux
# Type "help", "copyright", "credits" or "license" for more information.
>>> entry_points(group="console_scripts")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 1009, in entry_points
    return SelectableGroups.load(eps).select(**params)
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 459, in load
    ordered = sorted(eps, key=by_group)
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 1006, in <genexpr>
    eps = itertools.chain.from_iterable(
  File "/usr/lib/python3.10/importlib/metadata/_itertools.py", line 16, in unique_everseen
    k = key(element)
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 941, in _normalized_name
    return self._name_from_stem(stem) or super()._normalized_name
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 622, in _normalized_name
    return Prepared.normalize(self.name)
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 871, in normalize
    return re.sub(r"[-_.]+", "-", name).lower().replace('-', '_')
  File "/usr/lib/python3.10/re.py", line 209, in sub
    return _compile(pattern, flags).sub(repl, string, count)
TypeError: expected string or bytes-like object

Now if I remove the .dist-info folder and try again, everything is fine:

rm -rf *.dist-info
python3.10
# Python 3.10.4 (main, Mar 24 2022, 16:12:56) [GCC 9.4.0] on linux
# Type "help", "copyright", "credits" or "license" for more information.
>>> entry_points(group="console_scripts")
[EntryPoint(name= ...)]

Therefore this just seems to be the way importlib works.

The linked repository does not contain a .dist-info folder, so I am assuming it got created somehow (a) during an incomplete build, or (b) after a complete build followed by an incomplete cleanup.

Would any of assumptions (a) or (b) be the case here? If not, what were the commands you used that ended up generating an incomplete .dist-info folder?

In terms of action points for setuptools, the only thing we can do is ensure that no command is creating .dist-info folders incorrectly. So we need a reproducer that will generate the .dist-info folder using the supported workflows¹.

But if you want to tackle the fact that importlib fails when an empty .dist-info folder exists in the CWD, then the best would be opening an issue in the Python issue tracker.

Footnotes

1. Currently the supported workflows use one of python -m build, pip install . or pip install -e .. The workflows with python setup.py ... are deprecated. ↩

[a68366]

The folder was in the repo before it was removed in the last commit, e.g. here.
Apparently, the author himself is not sure why was it called .dist-info: cher-nov/cryptg#17 (comment)

So, if this error is not some sort of regression (since setuptools before 60.9.0 just ignored this folder and successfully built the package) and everything works as intended, I think this issue should be closed.

[abravalheri]

Thank you very much @a68366.

Setuptools 60.9.0 replaced pkg_resources with importlib-metadata for handling entry-points.

We can argue that the error was not happening before, but this failure is not something caused by setuptools code itself, but rather by a "backport" that replicates the latest implementation in the standard library.

It might be worthy reporting this to Python or importlib-metadata trackers, but there seems to be no direct action point on setuptools side (specially considering that reverting to pkg_resources is something we want to avoid).