UnicodeDecodeError on Python2

[Flamefire]

I'm trying to install wheel on Python 2 and getting an UnicodeDecodeError caused by a non-ASCII filename in the test tree of the package, see pypa/wheel#331 (comment). When debugging this I found that this is likely a bug in this repo.

Short: This appeared after some change while it worked with the same file before.

Traceback (most recent call last):
  File "setup.py", line 4, in <module>
    setup(maintainer=u'Alex Grnholm')
  File "build/bdist.linux-x86_64/egg/setuptools/__init__.py", line 145, in setup
  File "/usr/lib64/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib64/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "build/bdist.linux-x86_64/egg/setuptools/command/install.py", line 67, in run
  File "build/bdist.linux-x86_64/egg/setuptools/command/install.py", line 109, in do_egg_install
  File "/usr/lib64/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "build/bdist.linux-x86_64/egg/setuptools/command/bdist_egg.py", line 204, in run
  File "build/bdist.linux-x86_64/egg/setuptools/command/bdist_egg.py", line 321, in copy_metadata_to
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 40: ordinal not in range(128)

I found the change which led to the issue: pypa/wheel@e29a5bd

It adds package_dir={'': 'src'} which is picked up by setuptools as type unicode which leads to an egg-info folder name of type unicode while it was str before. See https://github.com/pypa/setuptools/blob/v44.0.0/setuptools/command/egg_info.py#L218 and https://github.com/pypa/setuptools/blob/v44.0.0/setuptools/command/egg_info.py#L223

This leads to a mix of str and unicode paths at https://github.com/pypa/setuptools/blob/v44.0.0/setuptools/command/bdist_egg.py#L321 (path is str, prefix is unicode) and then this issue when trying to convert one to the other which it didn't before (as both were str)

Will there be any bugfix release for Python2? Is any workaround possible from side of the wheel package?

To reproduce:

I'm using Python 2.7.5 to install setuptools 44.0.0, then wheel 0.34.2. Both done by downloading the tarball from PyPi and running:

• python setup.py build
• python setup.py install --prefix=<somedir>

[Flamefire]

A possible solution I found is to add a self.egg_info = self.egg_info.encode('utf-8') after https://github.com/pypa/setuptools/blob/v44.0.0/setuptools/command/egg_info.py#L223 possibly guarded by if six.PY2:

[jaraco]

Will there be any bugfix release for Python2?

Setuptools will support bugfix releases against the maint/44.x branch as long as they're warranted (worth the trouble of cutting a release). The use-case you describe seems like a reasonable one to support. I'm surprised this issue is only just now being reported, given that src directories have been supported in setup.cfg for some time.

[jaraco]

Interestingly, the issue doesn't happen when using pip to build/install wheel:

draft $ python2.7 -m pip install --no-binary :all: -t temp wheel                                                                                                    
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting wheel
  Using cached wheel-0.34.2.tar.gz (58 kB)
Skipping wheel build for wheel, due to binaries being disabled for it.
Installing collected packages: wheel
    Running setup.py install for wheel ... done
Successfully installed wheel-0.34.2

I'm not sure setup.py install should be supported.

[Flamefire]

The use case runs this on the extracted source tree. This is used because sometimes patches are required.

I don't understand why setup.py install should not be supported. Isn't that how to install packages from source?

[pganssle]

I don't understand why setup.py install should not be supported. Isn't that how to install packages from source?

No, the preferred way to install packages is with pip; in the case of installing it from the repo root, you'd use pip install ..

I'm not sure setup.py install should be supported.

I agree with that. I think generally our policy has been "if pip install works, we should advise people to use that." Unless this fix is particularly easy, I'm not sure it's worth the effort of backporting it to 44.x considering it only affects people using a combination of two deprecated technologies: Python 2.7 and setup.py install (though admittedly the Python 2 deprecation has been much louder than the setup.py install deprecation).

[Flamefire]

In our case we use it to install pip: first setuptools, then wheel, then pip, see easybuilders/easybuild-easyconfigs#9859 (look at changes, you'll get the gist)

so at this point we need the setup.py.

Anyway I'm surprised pip works. Doesn't it also run setup.py at some point?

[jaraco]

I'd recommend not installing pip that way. The pip instructions recommend installing using get-pip.py, which bypasses the setuptools build step. By using setup.py install, you're invoking easy_install, so you're getting older and deprecated behaviors.

Anyway I'm surprised pip works. Doesn't it also run setup.py at some point?

It does, but it passes other parameters and probably doesn't pass --prefix. And if wheel were to declare its build with PEP 517, setup.py would not be invoked and instead the setuptools would be called directly.

---

I realize this approach is an inversion from the approach that may have been recommended a decade ago, but things have changed. Get pip first (with the bootstrap script if necessary), then install build tools (if needed; we're working on making setuptools not an implicit requirement for any package).

[Flamefire]

In the case of installing it from the repo root, you'd use pip install ..

Unfortunately this doesn't work right away. To complete the information:

Goal: Install pip, setuptools and wheel with specified versions into a given prefix using the system python. It should then be usable as a module (basically: set PYTHONPATH and PATH so binaries and packages are found)

With the "old" approach running `setup.py this works (until the update to wheel showing a bug in setuptools).

Now there may or may not be a system installed pip. So following approaches:

1. python get-pip.py --no-setuptools --no-wheel --prefix /tmp/pip-prefix
   --> Fails because it tries to uninstall system pip which it cannot due to permissions
2. pip install --upgrade --prefix /tmp/pip-prefix pip
   --> Same as above
3. Download wheel tarball from PyPi and run pip install --prefix=/tmp/pip-prefix .
   --> Installs "UNKNOWN" (creating pip-egg-info/UNKNOWN.egg-info)

However it seems using -I/--ignore-installed does indeed work. Is this "the valid solution"(TM)?

BTW: Seems that using pip to install the package it doesn't touch the test files so it doesn't encounter the invalid path. And using printf debugging it doesn even run into the function at https://github.com/pypa/setuptools/blob/v44.0.0/setuptools/command/bdist_egg.py#L321
So question: Is there a (valid) use of pip install which is supposed to run into that code?

Bonus question: Is there a way to have some security? get-pip doesn't seem to have any releases or checksums to verify. This would be needed also for reproducibility.

[jaraco]

However it seems using -I/--ignore-installed does indeed work. Is this "the valid solution"(TM)?

Perhaps. If it works, then I'd say use it. Regardless, we've now established that setup.py install should not be used and we'll rely on pip to provide improved bootstrapping as needed. Also, by now, support for Python 2 is pretty much dropped. Thanks for your understanding.