Cannot install packages with files that have names with non-ascii characters

[elfosardo]

Environment

• pip version: 20.0.2
• Python version: 3.6
• OS: Ubuntu Bionic / Centos 8

Virtualenv generated using python3 -m venv

Description
Installation of a python package fails because some files in it have non-ascii characters in filenames:

   Downloading python-dateutil-2.8.1.tar.gz (331 kB)
   Installing build dependencies: started
   Installing build dependencies: finished with status 'error'
   ERROR: Command errored out with exit status 2:
    command: /opt/ironic-python-agent/bin/python3 /opt/ironic-python-agent/lib/python3.6/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-4t7qg7hp/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools; python_version != '"'"'3.3'"'"'' 'setuptools<40.0; python_version == '"'"'3.3'"'"'' wheel setuptools_scm
        cwd: None
   Complete output (28 lines):
   Ignoring setuptools: markers 'python_version == "3.3"' don't match your environment
   Collecting setuptools
     Downloading setuptools-45.1.0.zip (859 kB)
   Collecting wheel
     Downloading wheel-0.34.1.tar.gz (55 kB)
   ERROR: Exception:
   Traceback (most recent call last):
     File "/opt/ironic-python-agent/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 186, in _main
       status = self.run(options, args)
     File "/opt/ironic-python-agent/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 331, in run
       resolver.resolve(requirement_set)
     File "/opt/ironic-python-agent/lib/python3.6/site-packages/pip/_internal/legacy_resolve.py", line 177, in resolve
       discovered_reqs.extend(self._resolve_one(requirement_set, req))
     File "/opt/ironic-python-agent/lib/python3.6/site-packages/pip/_internal/legacy_resolve.py", line 333, in _resolve_one
       abstract_dist = self._get_abstract_dist_for(req_to_install)
     File "/opt/ironic-python-agent/lib/python3.6/site-packages/pip/_internal/legacy_resolve.py", line 282, in _get_abstract_dist_for
       abstract_dist = self.preparer.prepare_linked_requirement(req)
     File "/opt/ironic-python-agent/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 482, in prepare_linked_requirement
       hashes=hashes,
     File "/opt/ironic-python-agent/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 287, in unpack_url
       hashes=hashes,
     File "/opt/ironic-python-agent/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 164, in unpack_http_url
       unpack_file(from_path, location, content_type)
     File "/opt/ironic-python-agent/lib/python3.6/site-packages/pip/_internal/utils/unpacking.py", line 261, in unpack_file
       untar_file(filename, location)
     File "/opt/ironic-python-agent/lib/python3.6/site-packages/pip/_internal/utils/unpacking.py", line 222, in untar_file
       with open(path, 'wb') as destfp:
   UnicodeEncodeError: 'ascii' codec can't encode characters in position 72-74: ordinal not in range(128)
   ----------------------------------------
ERROR: Command errored out with exit status 2: /opt/ironic-python-agent/bin/python3 /opt/ironic-python-agent/lib/python3.6/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-4t7qg7hp/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools; python_version != '"'"'3.3'"'"'' 'setuptools<40.0; python_version == '"'"'3.3'"'"'' wheel setuptools_scm Check the logs for full command output.

In the wheel package you can find this file: wheel-0.34.1/tests/testdata/unicode.dist/unicodedist/åäö_日本語.py that seems to be triggering the issue.

Expected behavior
The package is installed correctly

[uranusjr]

This could be #7667. Could you check your system locale to confirm?

[elfosardo]

@uranusjr it seems like it, I did a test with different locale and it fails only with:

LANG=C
LANGUAGE=
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=
ubuntu@bionic-pip-test:~/test-ipa-pip-fail$ localectl status
   System Locale: LANG=C
       VC Keymap: n/a
      X11 Layout: us
       X11 Model: pc105

[uranusjr]

Thanks! I’ll leave this for the maintainers to label as a duplicate.