UnicodeEncodeError when installing Wheel 0.36.1 as numpy requirement

[Stikus]

Hello, looks like #331 strikes back in some situations.

After we tried to update our deepvariant installation to new release we encountered pretty much same problem (link to code line):

pip3 install "${PIP_ARGS[@]}" --no-binary=:all: "numpy==1.18.5"


ERROR: Command errored out with exit status 2:
   command: /usr/local/bin/python3 /root/.local/lib/python3.6/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-jst0mysx/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel 'Cython>=0.29.14'
       cwd: None
  Complete output (47 lines):
  Collecting Cython>=0.29.14
    Downloading Cython-0.29.21.tar.gz (2.1 MB)
  Collecting setuptools
    Downloading setuptools-51.0.0.zip (2.1 MB)
  Collecting wheel
    Downloading wheel-0.36.1.tar.gz (64 kB)
  ERROR: Exception:
  Traceback (most recent call last):
    File "/root/.local/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 224, in _main
      status = self.run(options, args)
    File "/root/.local/lib/python3.6/site-packages/pip/_internal/cli/req_command.py", line 180, in wrapper
      return func(self, options, args)
    File "/root/.local/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 321, in run
      reqs, check_supported_wheels=not options.target_dir
    File "/root/.local/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 122, in resolve
      requirements, max_rounds=try_to_avoid_resolution_too_deep,
    File "/root/.local/lib/python3.6/site-packages/pip/_vendor/resolvelib/resolvers.py", line 445, in resolve
      state = resolution.resolve(requirements, max_rounds=max_rounds)
    File "/root/.local/lib/python3.6/site-packages/pip/_vendor/resolvelib/resolvers.py", line 339, in resolve
      failure_causes = self._attempt_to_pin_criterion(name, criterion)
    File "/root/.local/lib/python3.6/site-packages/pip/_vendor/resolvelib/resolvers.py", line 207, in _attempt_to_pin_criterion
      criteria = self._get_criteria_to_update(candidate)
    File "/root/.local/lib/python3.6/site-packages/pip/_vendor/resolvelib/resolvers.py", line 198, in _get_criteria_to_update
      for r in self._p.get_dependencies(candidate):
    File "/root/.local/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/provider.py", line 102, in get_dependencies
      for r in candidate.iter_dependencies(with_requires)
    File "/root/.local/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/provider.py", line 101, in <listcomp>
      r
    File "/root/.local/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 252, in iter_dependencies
      requires = self.dist.requires() if with_requires else ()
    File "/root/.local/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 234, in dist
      self._prepare()
    File "/root/.local/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 221, in _prepare
      dist = self._prepare_distribution()
    File "/root/.local/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 314, in _prepare_distribution
      self._ireq, parallel_builds=True,
    File "/root/.local/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 480, in prepare_linked_requirement
      return self._prepare_linked_requirement(req, parallel_builds)
    File "/root/.local/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 505, in _prepare_linked_requirement
      self.download_dir, hashes,
    File "/root/.local/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 263, in unpack_url
      unpack_file(file.path, location, file.content_type)
    File "/root/.local/lib/python3.6/site-packages/pip/_internal/utils/unpacking.py", line 270, in unpack_file
      untar_file(filename, location)
    File "/root/.local/lib/python3.6/site-packages/pip/_internal/utils/unpacking.py", line 233, in untar_file
      with open(path, 'wb') as destfp:
  UnicodeEncodeError: 'ascii' codec can't encode characters in position 105-107: ordinal not in range(128)
  ----------------------------------------
ERROR: Command errored out with exit status 2: /usr/local/bin/python3 /root/.local/lib/python3.6/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-jst0mysx/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel 'Cython>=0.29.14' Check the logs for full command output.

Interesting fact - we can install wheel with pip and we're doing it prior numpy install - everything ok:

Collecting pip
  Downloading pip-20.3.1-py2.py3-none-any.whl (1.5 MB)
Collecting setuptools
  Downloading setuptools-51.0.0-py3-none-any.whl (785 kB)
Collecting wheel
  Downloading wheel-0.36.1-py2.py3-none-any.whl (34 kB)
Installing collected packages: pip, setuptools, wheel
Successfully installed pip-20.3.1 setuptools-51.0.0 wheel-0.36.1

The only difference I've found is using .tar.gz in a failed attempt.

My manual debug result is that installing newer versions of numpy became successful at 1.19.3:

root@b6b80d8493d2:/soft/deepvariant-1.1.0-src# pip3 install "${PIP_ARGS[@]}" --no-binary=:all: "numpy==1.19.2"        
Collecting numpy==1.19.2                                                                                              
  Downloading numpy-1.19.2.zip (7.3 MB)                                                                               
     |################################| 7.3 MB 2.3 MB/s                                                               
  Installing build dependencies ... error                                                                             
  ERROR: Command errored out with exit status 2:                                                                      
   command: /usr/bin/python /root/.local/lib/python3.6/site-packages/pip install --ignore-installed --no-user --prefix
 /tmp/pip-build-env-wkdt148z/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.
org/simple -- 'setuptools<49.2.0' wheel 'Cython>=0.29.21'                                                             
       cwd: None                                                                                                      
  Complete output (47 lines):                                                                                         

...

    File "/root/.local/lib/python3.6/site-packages/pip/_internal/utils/unpacking.py", line 233, in untar_file         
      with open(path, 'wb') as destfp:                                                                                
  UnicodeEncodeError: 'ascii' codec can't encode characters in position 105-107: ordinal not in range(128)            
  ----------------------------------------                                                                            
ERROR: Command errored out with exit status 2: /usr/bin/python /root/.local/lib/python3.6/site-packages/pip install --
ignore-installed --no-user --prefix /tmp/pip-build-env-wkdt148z/overlay --no-warn-script-location --no-binary :all: --
only-binary :none: -i https://pypi.org/simple -- 'setuptools<49.2.0' wheel 'Cython>=0.29.21' Check the logs for full c
ommand output.                                                                                                        

root@b6b80d8493d2:/soft/deepvariant-1.1.0-src# pip3 install "${PIP_ARGS[@]}" --no-binary=:all: "numpy==1.19.3"        
Collecting numpy==1.19.3                                                                                              
  Downloading numpy-1.19.3.zip (7.3 MB)                                                                               
     |################################| 7.3 MB 565 kB/s                                                               
  Installing build dependencies ... done                                                                              
  Getting requirements to build wheel ... done                                                                        
    Preparing wheel metadata ... done                                                                                 
Building wheels for collected packages: numpy                                                                         
  Building wheel for numpy (PEP 517) ... done                                                                         
  Created wheel for numpy: filename=numpy-1.19.3-cp36-cp36m-linux_x86_64.whl size=13742596 sha256=24f913edf3c6ef18f6c3
a4a93d4d3ac20604427914a325234a4a254b9622edce                                                                          
  Stored in directory: /root/.cache/pip/wheels/8a/f4/13/a5eee6699f842da9a8d824976f3c9e5dec2fee089082724f1b            
Successfully built numpy                                                                                              
Installing collected packages: numpy                                                                                  
  Attempting uninstall: numpy                                                                                         
    Found existing installation: numpy 1.19.4                                                                         
    Uninstalling numpy-1.19.4:                                                                                        
      Successfully uninstalled numpy-1.19.4                                                                           
Successfully installed numpy-1.19.3              

And exactly numpy 1.19.3 set upper versions for wheel dependency: Changelog, PR.

Can you give me any advice? Looks like there are several problems, one of them is using Python 3.6 (I can install numpy=1.18.5 on Python 3.7 with same method).

[agronholm]

All of those tracebacks come from pip, not wheel. Can we agree on that? Can you give me instructions to reproduce the problem?

[agronholm]

From a glance this looks like you have an environment that uses ascii as its file system encoding. Since the wheel tarball contains a non-ASCII file name, it cannot be extracted successfully.

[agronholm]

Yeah, so reading through #331 again I remember what that was all about. The automated releases are currently made using Python 3.8 which uses the new tar format. Does this mean we can never have non-ASCII file names in sdists?

[Stikus]

Can you tell me why wheel 0.34.2 or 0.35.1 dont't have this problem?

[agronholm]

They were built using Travis instead of Github Actions.

[agronholm]

I can make GH Actions build it with py3.7 but this problem is going to come around again eventually unless I remove that particular file from the sdist. What I would like to figure out is whether any sdist should be allowed to package non-ASCII files or not.

[pfmoore]

For what it's worth PEP 517 states that "The generated tarball should use the modern POSIX.1-2001 pax tar format, which specifies UTF-8 based file names".

So in theory yes, non-ASCII filenames are allowed. But it may not be the case that tools are following that standard yet.

[agronholm]

That leaves the question, is wheel doing something wrong? What are the tools supposed to do when the filesystem encoding is set to ascii and they encounter a non-ASCII file name?

[pfmoore]

Ah, right - that's probably a question for the packaging discourse. My instinct is to say you should report an error that suggests the user needs to use a filesystem that supports the files in the sdist, but that's very much just a knee-jerk reaction, and only a personal view.

[Stikus]

@agronholm Should I keep this open for you to continue discussion or should I close this?

[agronholm]

Maybe keep this open until a course of action can be figured out. I'm not sure what to do here. The problem does not lie with wheel or even the way it's packaged, but that facts that 1) you're trying to unpack a non-ascii file name in an environment that, for some reason, only allows ASCII file names, 2) are not using the wheel of wheel to install it (which would circumvent the problem).

There are two obvious workarounds:

1. Remove the offending file, choosing to generate it on the fly instead
2. Use Python 3.7 to package it, violating PEP 517

I'm hoping other people will chime in here.

[Stikus]

It's not me, it's numpy :)
But thanks for tips. My workaround was installing numpy without --no-binary=:all: which use already installed wheel with .whl.

I'll keep this open, hope you'll find the right answer for this question.

[agronholm]

Is there a problem with --no-binary=numpy? Install numpy from sources but other tools from wheels?

[Stikus]

You are correct - --no-binary=numpy solves the problem for me too. I'll wait for deepvariant team decision about this.

[mattip]

NumPy dev here. I am not sure I follow the discussion. Is something actionable NumPy could do to help out here? 1.20 will be released in January ...

[agronholm]

Is there a reason your build process installs the build tools from source instead of using wheels?

[Stikus]

@mattip Sorry for some confusion - finally I understood that reason was --no-binary=:all: and not numpy. I don't think numpy is involved in this issue now.

@agronholm This is deepvariant bash script and it had reason for doing that for Ubuntu 14.04:
https://github.com/google/deepvariant/blob/r1.1/run-prereq.sh#L117

# Because of an issue with pypi's numpy on Ubuntu 14.04. we need to compile from
# source. But we know that on 16.04 we don't need to compile from source
# See https://github.com/tensorflow/tensorflow/issues/6968#issuecomment-279061085

Deepvariant still using 16.04 to build and don't test their scripts for 18.04, there for their code tries to build from source all non-16.04 numpy installations. According to comments in my issue - they will update to 18.04 and (I assume) stop using source in favor of wheels.

[agronholm]

Sounds like you can close the issue?

[Stikus]

For me - yes. I thought you need it for discussion.

[agronholm]

The right place to have that discussion would be https://discuss.python.org/c/packaging/14.