pip wheel fails with UnicodeDecodeError when setup.py and default encoding don't match

[SpotlightKid]

When trying to build a wheel for a package, whose setup.py has a non-ascii encoding (even with proper encoding declaration), it fails with the following error:

$ pip wheel --wheel-dir .  -v pysmfDownloading/unpacking pysmf
  Downloading pysmf-0.1.1.tar.gz (50kB): 
  Downloading from URL https://pypi.python.org/packages/source/p/pysmf/pysmf-0.1.1.tar.gz#md5=8354f3cd7752b12e2668d4ed9aaa25c9 (from https://pypi.python.org/simple/pysmf/)
...Downloading pysmf-0.1.1.tar.gz (50kB): 50kB downloaded
  Storing download in cache at ./.pip_download_cache/https%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2Fp%2Fpysmf%2Fpysmf-0.1.1.tar.gz
  Running setup.py (path:/home/chris/tmp/pip_build_chris/pysmf/setup.py) egg_info for package pysmf
    running egg_info
    creating pip-egg-info/pysmf.egg-info
    writing pip-egg-info/pysmf.egg-info/PKG-INFO
    writing top-level names to pip-egg-info/pysmf.egg-info/top_level.txt
    writing dependency_links to pip-egg-info/pysmf.egg-info/dependency_links.txt
    writing manifest file 'pip-egg-info/pysmf.egg-info/SOURCES.txt'
    warning: manifest_maker: standard file '-c' not found

    reading manifest file 'pip-egg-info/pysmf.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'pip-egg-info/pysmf.egg-info/SOURCES.txt'
Building wheels for collected packages: pysmf
  Running setup.py bdist_wheel for pysmf
  Destination directory: /home/chris
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
      return codecs.ascii_decode(input, self.errors)[0]
  UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1623: ordinal not in range(128)
  Complete output from command /usr/bin/python -c "import setuptools;__file__='/home/chris/tmp/pip_build_chris/pysmf/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /home/chris:
  Traceback (most recent call last):

  File "<string>", line 1, in <module>

  File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode

    return codecs.ascii_decode(input, self.errors)[0]

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1623: ordinal not in range(128)

----------------------------------------
  Failed building wheel for pysmf
Failed to build pysmf
Cleaning up...
  Removing temporary dir /home/chris/tmp/pip_build_chris...

The offending code is calls to read() in the command to read and run setup.py. pip passes this to Python on the command line. In plain, unraveled code it would look like:

import setuptools
__file__ = 'setup.py'
fp = open(__file__)
setup = fp.read()
setup = setup.replace('\r\n', '\n')
code = compile(setup, __file__, 'exec')
exec(code)

The fp.read() call uses the system dependent default encoding (locale.getpreferredencoding()), since the open() call didn't specify an encoding. Therefore the above pip wheel invocation failed on my Arm system, but not on my Linux x86_64 laptop.

To fix this, pip would need to honor the source encoding declaration in setup.py, which is a rather complicated logic. Why does pip read & compile the setup.py file itself anyway? Could it not just call subprocess.check_call([sys.executable, 'setup.py' 'bdist_wheel'])?

[nils-werner]

Note this only happens when using Python 3.

[hansmbakker]

I'm also having trouble building wheels out of phue and python-nest using pip3. This is because their setup.py's are utf8.

[gumblex]

The offending code is at https://github.com/pypa/pip/blob/develop/pip/wheel.py#L687
@SpotlightKid The 'complicated logic' is implemented in tokenize.detect_encoding (we can use tokenize.open)

[SpotlightKid]

Yes, but it's still not clear to me, why setup.py isn't run directly. Anybody have any insights on this?

[xavfernandez]

Since sdist install works, we could use the same trick used for sdist install: https://github.com/pypa/pip/blob/e42e822/pip/req/req_install.py#L478-L482

[xavfernandez]

Yes, but it's still not clear to me, why setup.py isn't run directly. Anybody have any insights on this?

Because we want to use setuptools for all setup.py invocation, this is mandatory to be able to perform a clean uninstall.

[xavfernandez]

Also related to #1233

[xavfernandez]

@gumblex could you try #3265 (or https://github.com/xavfernandez/pip/tree/setuptool_shim) and confirm it is fixed ?

[gumblex]

It works.