'pip inspect' crashes with python-3.10.10 and pip-23.0.0 on Windows 11 (an utf-8 problem) #11798
Comments
stonebig
added
S: needs triage
stonebig
changed the title
|
hum, with this exception addition in the code, I can find the package causing the crash: yarl
|
|
well in fact, i have 61 'zzzzzzzzzzzzzzzzzzzzzz' line, so 61 packages have a description that do crash the thing when using my full list of about 600 packages |
|
simple reproducer shall be, open a windows console (dos or powershell): |
|
It doesn't fail for me. I imagine it's Unicode characters in the description, and when you redirect output in Windows to a file, the encoding is not UTF-8. Probably something to do with your environment that I can't reproduce. |
|
I'm using "pip inspect" as it is the only API provided per pip, so it shall not crash, at least put a 'zzzzzzzzz' on the output problem. |
|
All I can suggest at this point:
But please understand, that encoding errors are almost always a problem with your configuration. Pip simply prints the metadata, and if that data includes characters which are unprintable in your environment, doing so will always fail (or produce mojibake, which is not something we would want to do). |
|
This is a reoccurring class of problem on Windows, not exclusively a Rich issue. Essentially Rich dutifully uses the reported encoding when writing output, and you get an encoding error. Setting the env var Frustratingly, Rich has done the right thing here by respecting the encoding. But its not great for dev or users. Rich could set @pfmoore Do you have any insight on the best way for Rich to tackle this? |
|
on my example, Windows was utf-8-SIG until recently, to not blind guess the pagecode I see in PEP-686 it is becoming the intended default (for 3.12 ?) https://peps.python.org/pep-0686/ https://discuss.python.org/t/pep-686-make-utf-8-mode-default-round-2/14737/15 |
|
Personally, I think that I'm happy with the current approach. I'd rather display things accurately, or fail clearly, than potentially mangle output. As @stonebig said, longer term the issue will be fixed in core Python. I'm still curious what it is about @stonebig's environment that makes it not use UTF-8 by default. I tried to reproduce it, on Windows 11, with Python 3.10.6 and 3.11.0 from python.org, in Powershell in Windows Terminal, and it worked fine. But given that it seems to be cause by something specific in the user's environment, I'm inclined not to worry about it from pip's point of view. |
|
i'm using launcher icons from an old nsis.exe to launch my environment via even by obscure old windows script tech.... I guess at a moment, all will be a ProgressiveWebApp and these futil pre-historic questions will go away https://stonebig.github.io/my-demo2/lab/index.html .... replaced per new problems |
|
Sorry, I don't follow what you're saying here. Are you not running a copy of pip installed in a standard Python virtual environment? You gave no indication of that in your bug report 🙁 |
|
it's pip launched on a WinPython terminal, via launcher icons. |
|
If you can’t reproduce in a standard terminal, I’d suspect WinPython, then. |
|
I wonder if it would make sense to provide a way to configure Rich to use |
pradyunsg
added
kind: crash
|
I can reproduce this issue on the official python (from MS Store) in Windows 11 and use pdb to locate the problem.
If both are true, it will call In the code section just below this line, the if use_legacy_windows_render:
...
legacy_windows_render(buffer, LegacyWindowsTerm(self.file))
else:
# Either a non-std stream on legacy Windows, or modern Windows.
text = self._render_buffer(self._buffer[:])
# https://bugs.python.org/issue37871
write = self.file.write
for line in text.splitlines(True):
try:
write(line)
except UnicodeEncodeError as error:
error.reason = f"{error.reason}\n*** You may need to add PYTHONIOENCODING=utf-8 to your environment ***"
raiseIt would be nice to use the same error message here to ask the user to set This bug has already be reported upstream in rich/issures/212 |
|
The root problem is that Python uses Testing in Powershell 5.1 of Windows 11: PS C:\Users> python -c "import sys;print(sys.stdout.encoding,type(sys.stdout.buffer.raw))"
# Output: utf-8 <class '_io._WindowsConsoleIO'>
# Redirect stdout to file or another cmdlet
PS C:\Users> python -c "import sys;print(sys.stdout.encoding,type(sys.stdout.buffer.raw))" | %{echo $_}
# Output: cp1252 <class '_io.FileIO'>
# Assign to PS variable
PS C:\Users> $msg = python -c "import sys;print(sys.stdout.encoding)" ; echo $msg
# Output: cp1252Testing in CMD of Windows 11: C:\Users> python3.10 -c "import sys;print(sys.stdout.encoding)" | findstr /R /C:".*"
cp1252The most innocent pip command will fail on Unicode-path due to this bug: (new_venv) PS D:\tmp\ひらがな> $pipinfo = pip show pip
--- Logging error ---
Traceback (most recent call last):
...
File "D:\tmp\ひらがな\new_venv\lib\site-packages\pip\_vendor\rich\console.py", line 1999, in _check_buffer
legacy_windows_render(You will notice that setting (new_venv) PS D:\tmp\ひらがな> $env:PYTHONIOENCODING="utf8"
(new_venv) PS D:\tmp\ひらがな> echo (pip show pip)
...
Location: d:\tmp\ひらがな\new_venv\lib\site-packagesThat's because windows console use UTF-16-LE by default. The tested python distribution comes from MS APP Store, although I don't think it matters. The python team has known this issue for a long time and introduced the "UTF-8" mode in PEP 540 in 2016. However this mode is not and will not be default in near future because of those reasons mentioned in the PEP. Thus, it is the users' responsibility to set the correct encoding and the libraries' responsibility to remind the users to do so. So, it would be enough for pip to generate a warning/error message asking the user to set |
|
I was hit per about-time package recently, rich package is not the only problem around.
|
stonebig
changed the title
|
I understood how to use importlib.metadata, so pip inspect command line and Windows utf-8 tricks are no more a problem |
Description
trying
pip inspect >> text.txton my big list of wheels, I get the following crash:Expected behavior
no crash
pip version
pip-23.0.0
Python version
cpython-3.10.10
OS
Windows11
How to Reproduce
you may try on windows
I was using this list of packages:
pip list
Package Version
idna 3.1
msvc-runtime 14.32.31326
multidict 6.0.2
pip 23.0
setuptools 67.2.0
simpy 4.0.1
sqlite-bro 0.12.2
wheel 0.38.4
winpython 6.0.20230212
yarl 1.7.2
Output
Code of Conduct
The text was updated successfully, but these errors were encountered: