Non-ascii error with pip install

[OisinMoran]

Trying to use pip install but keep getting this message. Think it may have to do with the 'í' in my name (Oisín) as that is in my path name.

PS C:\Users\Oisín\audiogrep> pip install -r requirements.txt
Collecting pydub==0.9.5 (from -r requirements.txt (line 1))
 Exception:
 Traceback (most recent call last):
   File "C:\Python27\lib\site-packages\pip\basecommand.py", line 232, in main
     status = self.run(options, args)
   File "C:\Python27\lib\site-packages\pip\commands\install.py", line 339, in run
     requirement_set.prepare_files(finder)
   File "C:\Python27\lib\site-packages\pip\req\req_set.py", line 333, in prepare_files
     upgrade=self.upgrade,
   File "C:\Python27\lib\site-packages\pip\index.py", line 305, in find_requirement
     page = self._get_page(main_index_url, req)
   File "C:\Python27\lib\site-packages\pip\index.py", line 783, in _get_page
     return HTMLPage.get_page(link, req, session=self.session)
   File "C:\Python27\lib\site-packages\pip\index.py", line 872, in get_page
     "Cache-Control": "max-age=600",
   File "C:\Python27\lib\site-packages\pip\_vendor\requests\sessions.py", line 473, in get
     return self.request('GET', url, **kwargs)
   File "C:\Python27\lib\site-packages\pip\download.py", line 365, in request
     return super(PipSession, self).request(method, url, *args, **kwargs)
   File "C:\Python27\lib\site-packages\pip\_vendor\requests\sessions.py", line 461, in request
     resp = self.send(prep, **send_kwargs)
   File "C:\Python27\lib\site-packages\pip\_vendor\requests\sessions.py", line 610, in send
     r.content
   File "C:\Python27\lib\site-packages\pip\_vendor\requests\models.py", line 730, in content
     self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
   File "C:\Python27\lib\site-packages\pip\_vendor\requests\models.py", line 655, in generate
     for chunk in self.raw.stream(chunk_size, decode_content=True):
   File "C:\Python27\lib\site-packages\pip\_vendor\requests\packages\urllib3\response.py", line 256
     data = self.read(amt=amt, decode_content=decode_content)
   File "C:\Python27\lib\site-packages\pip\_vendor\requests\packages\urllib3\response.py", line 186
     data = self._fp.read(amt)
   File "C:\Python27\lib\site-packages\pip\_vendor\cachecontrol\filewrapper.py", line 54, in read
     self.__callback(self.__buf.getvalue())
   File "C:\Python27\lib\site-packages\pip\_vendor\cachecontrol\controller.py", line 217, in cache_
     self.serializer.dumps(request, response, body=body),
   File "C:\Python27\lib\site-packages\pip\download.py", line 268, in set
     return super(SafeFileCache, self).set(*args, **kwargs)
   File "C:\Python27\lib\site-packages\pip\_vendor\cachecontrol\caches\file_cache.py", line 83, in
     with FileLock(name) as lock:
   File "C:\Python27\lib\site-packages\pip\_vendor\lockfile\mkdirlockfile.py", line 18, in __init__
     LockBase.__init__(self, path, threaded, timeout)
   File "C:\Python27\lib\site-packages\pip\_vendor\lockfile\__init__.py", line 189, in __init__
     hash(self.path)))
   File "C:\Python27\lib\ntpath.py", line 84, in join
     result_path = result_path + p_path
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xed in position 3: ordinal not in range(128)''''

Using pip 6.0.8 on Windows with Python 2.7

[piotr-dobrogost]

This was raised in #1713 and supposedly fixed in #2008 but it looks like it's still the case.

[OisinMoran]

Is there any way I can fix this just for myself? If not when should I expect this to be fixed?

[remram44]

I think that the correct fix is to not use such paths. Although unicode filenames are a reality of modern systems, pip (and Python) are known for their terrible support of it. There are probably a lot of such places throughout the code (see #2521), and unless the pip team comes to care and test for it, it will never be reliable.
I for one wish they did, but I understand they only have time for so much, and I can't do this for them either.

[pfmoore]

Looking at the trace, it appears the issue is in the vendored lockfile module. If you could reproduce the issue just using lockfile and report it to that project, that would be a huge help, as we could then upgrade to the fixed version once it gets released. That's all the pip developers would do anyway, as we don't, as policy, modify code that we vendor.

[OisinMoran]

For anyone else who may get this problem this is how I fixed it for myself:

• Change your name (with no loss of data) to a fully ascii version by following these instructions http://www.sevenforums.com/tutorials/147545-user-profile-folder-change-user-account-folder-name.html
• Remove all versions of Python
• Reinstall Python 3.4 ( Which includes pip - you could presumably also reinstall 2.7 and install pip separately but I didn't )
• Remove all references to Python27 in user variable PATH and system variable Path. (
  Start->right click Computer->Properties->Advanced system settings->Environment Variables)
• Ensure these both contain C:\Python34 and C:\Python34\Scripts
• Add a few restarts to taste

Pro-tip: Avoid giving your children names with non-ascii characters

[remram44]

Please someone reopen this. This is definitely a bug with pip, it is supposed to be unicode safe. The fact that there is a workaround doesn't erase the fault.

[pfmoore]

Agreed this is a bug. It's possibly only in 2.x (the OP hit the bug in 2.x, but his "fix" uses no non-ASCII username and uses 3.4, so it's not possible to tell) because it seems to be a result of string/unicode confusion. I believe that it's the sort of problem that "just works" in 3.x, because it's a consequence of the broken "mixed bytes/string model" in Python 2.x.

Also, it's probably a bug with one of the vendored packages (CacheControl or lockfile) being unable to cope with being given a Unicode path for the cache/lockfile directory. From the traceback, I'm not sure which of the two would accept responsibility for the bug. Pip uses a directory under the user's home directory for the cache - cachecontrol puts the cache there and appears to ask lockfile to create its lock file in that directory, so either or both needs to be prepared to be passed a non-ASCII directory name. My bet's on the bug being with lockfile.

I have reopened the issue, but it needs someone to reproduce the bug and open an issue against the lockfile project. Ideally such a bug report should include a test case that is independent of pip. This issue can then be closed when that reported bug is fixed and a new release of the relevant library is vendored into pip. @remram44 would you be able to reproduce and raise such a report? I'm assuming the OP isn't interested in doing so, as he's now worked around the issue...

[remram44]

I can't reproduce this on pip 7.0.3 in a win7 VM with a non-ascii username and homepath (but I'm probably not running into the right condition?) When is the lockfile used?

[pfmoore]

Can it be reproduced with pip 6.0.8? Maybe it's fixed at the pip end since then? Were you not encountering this issue yourself?

[remram44]

Not with 6.0.8 either. I don't have a Windows machine anymore, I'm running this in a clean VM.

[pfmoore]

OK. @OisinMoran do you have the means to reproduce this any more?

If we can't reproduce it, I'm inclined to re-close the issue as cannot reproduce. When it comes up again, we'll get a new issue with a test case and we can report it to the lockfile project

[OisinMoran]

Fortunately/unfortunately it works fine for me now so cannot reproduce the error sorry!

[pfmoore]

OK, I'm going to (re-) close the issue as not reproducible. Should it occur again, hopefully we can get a test case across to the lockfile project.

Or if someone wants to raise an issue now against lockfile, noting that when asked to create a lockfile in a directory with a non-ASCII name, it fails, pointing at this issue as an example, then that would be fine too :-)

[Andrej730]

I figure out what was wrong in my case - my full computer name was "ПК" which is "PC" in russian and whenever join function trying to
result_path = result_path + p_path
Python 2 implicitly trying to decode bytes from p_path to unicode as ascii and cause thats not possible i've got this decoding error. So, my solution - add decoding as latin1 in LockBase's init (in C:\Python27\Lib\site-packages\pip-7.0.3-py2.7.egg\pip_vendor\lockfile_init_.py):
this

self.hostname = socket.gethostname()

to this

self.hostname = socket.gethostname().decode('latin1') 

[mitsuhiko]

This was just reported by a user of lektor as something that is still happening on Russian windows PCs:

  File "c:\temp\site\venv\lib\site-packages\pip\_vendor\lockfile\__init__.py", line 189, in __init__
    hash(self.path)))
  File "c:\temp\site\venv\lib\ntpath.py", line 85, in join
    result_path = result_path + p_path
UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 0: ordinal not in range(128)

[danilaml]

Just had this issue myself. Installing latest Python 2.7.12 didn't help, but @Andrej730 workaround worked!
Only the path was a little different for me:
C:\Python27\Lib\site-packages\pip\_vendor\lockfile

Tried to update the pip and it immedeatly reverted to the old buggy code, so the issue is definitely not fixed!

[asmaier]

I can confirm this bug still exists in pip 8.1.1 . I'm using a russian Windows 7 Version 6.1 (Build 7601: Service Pack 1) with a user name and a computer name that uses cyrillic letters.

C:\> pip install jinja2
...
File "C:\Python27\lib\ntpath.py", line 85, in join
     result_path = result_path + p_path
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xce in position 0: ordinal not in range(128)

I can also confirm that the workaround from @Andrej730 solves the issue.

[rvilla87]

Another report with pip 9.0.1:

Error:

File "C:\Users\Rubén\AppData\Local\conda\conda\envs\version2\lib\site-packages\pip\wheel.py", line 184, in fix_script
    exename = sys.executable.encode(sys.getfilesystemencoding())
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 12: ordinal not in range(128)

Pip version:
pip 9.0.1 from C:\Users\Rubén\AppData\Local\conda\conda\envs\version2\lib\site-packages (python 2.7)

[danilaml]

@pfmoore maybe reopen this issue? Hard to tell if our feedbacks are getting ignored because they are considered to be fixed.

[pfmoore]

@danilaml To reopen, we'd need a reproducible test case. Specifically, that means a script or set of instructions that reproduces the issue, without relying on the setup of the user's PC. Specifically, if it only happens when the user's name contains non-ASCII characters, then the reproduction script would need to mock out the relevant stdlib calls, as you can't guarantee that the user trying to fix the issue has a non-ASCII username. As this is clearly an issue that keeps cropping up, we'd really want to add a test to pip's test suite to demonstrate it - so a report that included precisely such a test would be ideal. But with sufficiently clear instructions, we could write such a test (but again, note that our Windows CI server, Appveyor, doesn't run the tests with a non-ASCII user account, so we're back to the point about needing to give us a test that an ASCII-only user can run).

I appreciate this is a lot to ask. But as a volunteer group, with no-one paid to work on pip, the pip developers simply don't have the resources to spend trying to track down this issue. We have tried, by simple code inspection, and you're telling me that we haven't managed yet, so we need to look at other options.

An alternative would be for someone to submit a PR containing a fix. But ideally I'd still want such a fix to include a test, so that we don't risk regressions.

So, I still think this issue should not be reopened. If someone has a full set of reproduction steps, feel free to open a new issue describing those steps. If I can run them, and demonstrate the issue, on my PC, then we'll be making progress. I can at least mark the issue as verified in that case (I won't promise to be able to fix it for you - volunteer organisation, remember :-) - but at least we'll have something someone can then work on productively). Or if someone wants to provide a fix, then they can create a PR, and reference this issue. The people who've reported still having the problem here can then test with the fix, and feed back on the PR as to whether it resolves the problem.

Note The other point is that the reports all still seem to have been showing that the vendored lockfile module is the cause. Has anyone still reporting the issue tried with the latest unreleased pip, to confirm we haven't already fixed this by re-vendoring a newer version? Or tried reporting the bug upstream to lockfile? Because if the issue is with lockfile, we wouldn't make or accept a fix here anyway as we don't carry patches to vendored libraries within pip, as a matter of policy.

[danilaml]

@pfmoore sorry for the late reply, but I've investigated the issue and it seems there is a way to run tests under user with non-ascii characters in user name on appveyor (thanks to their awesome support). Here is my proof of concept branch: https://github.com/danilaml/pip/blob/test-non-ascii-user/appveyor.yml
as you can see here: https://ci.appveyor.com/project/danilaml/pip/build/1.0.24
the tests fail an all py3 configurations (btw, I think I've encountered my issues with python2 too, so I'm not sure whether they were fixed since then or that tests just don't exhibit them)

[pfmoore]

@danilaml Thanks very much for doing that - apart from the specific case here, it's nice to have an example of how to write tests like this on Appveyor.

Having said that, the failures here are in tox, before we even start running the pip tests. So it's not showing an issue with pip. It might be worth raising the problem you've demonstrated with the tox project, but it doesn't help with confirming the situation on this issue, sadly.

[zoltanbedi]

Sam as @rvilla87. Any update on this?

[pfmoore]

@B3zo0 This is a different issue. Please open a new issue with a full description of how to reproduce including the full traceback you get. The report by @rvilla87 is confusing, as it shows a decode error from an encode operation. My suspicion is that this may be somehow related to Python 2's limited Unicode model, but we'd need more testing to confirm that.

[zujajafatima]

Hi
i have installed python3.4.2 and working on pip 6.0.8
when i installed numpy then after 95% in downloading , i received the error. can anyone help me to fix this error. the error is
Traceback (most recent call last):
File "C:\Python34\lib\site-packages\pip_vendor\requests\packages\urllib3\response.py", line 186, in read
data = self._fp.read(amt)
File "C:\Python34\lib\site-packages\pip_vendor\cachecontrol\filewrapper.py", line 49, in read
data = self.__fp.read(amt)
File "C:\Python34\lib\http\client.py", line 500, in read
return super(HTTPResponse, self).read(amt)
File "C:\Python34\lib\http\client.py", line 539, in readinto
n = self.fp.readinto(b)
File "C:\Python34\lib\socket.py", line 374, in readinto
return self._sock.recv_into(b)
File "C:\Python34\lib\ssl.py", line 751, in recv_into
return self.read(nbytes, buffer)
File "C:\Python34\lib\ssl.py", line 623, in read
v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Python34\lib\site-packages\pip\basecommand.py", line 232, in main
status = self.run(options, args)
File "C:\Python34\lib\site-packages\pip\commands\install.py", line 339, in run
requirement_set.prepare_files(finder)
File "C:\Python34\lib\site-packages\pip\req\req_set.py", line 355, in prepare_files
do_download, session=self.session,
File "C:\Python34\lib\site-packages\pip\download.py", line 782, in unpack_url
session,
File "C:\Python34\lib\site-packages\pip\download.py", line 667, in unpack_http_url
from_path, content_type = _download_http_url(link, session, temp_dir)
File "C:\Python34\lib\site-packages\pip\download.py", line 843, in _download_http_url
_download_url(resp, link, content_file)
File "C:\Python34\lib\site-packages\pip\download.py", line 615, in _download_url
for chunk in progress_indicator(resp_read(4096), 4096):
File "C:\Python34\lib\site-packages\pip\utils\ui.py", line 46, in iter
for x in it:
File "C:\Python34\lib\site-packages\pip\download.py", line 580, in resp_read
decode_content=False):
File "C:\Python34\lib\site-packages\pip_vendor\requests\packages\urllib3\response.py", line 256, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "C:\Python34\lib\site-packages\pip_vendor\requests\packages\urllib3\response.py", line 201, in read
raise ReadTimeoutError(self._pool, None, 'Read timed out.')
pip._vendor.requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='pypi.python.org', port=443): Read timed out.