Pip crashes in progress bar rendering on Windows

[SimplyProgrammer]

I have installed Graalpy 24.1.0 as a regular CPython on Windows 10 with pyenv-win and set is as currently used version with pyenv global command.
Then I have generated venv using graalpy -m venv .pyvenv and tried to
pyvenv/Scripts/python.exe -m pip --no-cache-dir install numpy.

However as it turns out I cant because it throws this:

Attempting to download something else, matplotlib for example resulted in the same thing...
So Is there something that I am doing wrong?

Because If you cant install the libraries such as numpy or matplotlib which are one of the most used ones, then that's a bit problematic to say at least...

[msimacek]

Hi @SimplyProgrammer. I just tried installing something in a Windows VM and it worked fine. From the traceback it seems it's hitting some code path for console handling in pip that we haven't seen in our testing. It says it's using some "legacy" windows renderer, what windows version do you run it on? Can you please post what console program did you use? Are you using some sort of remote access? Can you try with --progress-bar off?

[SimplyProgrammer]

Hi, thanks for the reply.
Well "legacy windows rendered" is strange. My laptop is kinda "naughty" sometimes but certainly not old, its only 2-something years old, and mostly reliable.
I am running Windows 10 22H2 - x64 on Intel i7 with only integrated iRISx graphics. 16G of RAM.
I am not running it in VM nor remotely... I was running it with plain old Windows cmd...
Also I can confirm that regular pythons pip is having no issues (but that's python 3.12.6)
And when trying to run it with the mentioned --progress-bar off it protested that there is "no such option: --progress-bar".

However I have also tried to run it in something less barbaric than cmd, in GitBash to be more specific and in that thing, after some waiting and I mean quite LONG waiting... :) it kinda worked, but then it still failed on "Preparing metadata"... but it at least did no crash instantly but still not exactly great...

user@DESKTOP-7IC6J86 MINGW64 ~/.pyvenv/Scripts
$ ./python.exe -m pip --no-cache-dir install numpy
Looking in indexes: https://pypi.org/simple, https://www.graalvm.org/python/wheels/
Collecting numpy
  Downloading numpy-2.0.2.tar.gz (18.9 MB)
     ---------------------------------------- 18.9/18.9 MB 8.0 MB/s eta 0:00:00
auto-patching C API usages in C:\Users\user\AppData\Local\Temp\pip-install-puaai2it\numpy_0ebbbb2e595c4f5eb6bb296fcdd0e7bf\numpy\_core\include\numpy\ndarrayobject.h
auto-patching C API usages in C:\Users\user\AppData\Local\Temp\pip-install-puaai2it\numpy_0ebbbb2e595c4f5eb6bb296fcdd0e7bf\numpy\_core\include\numpy\npy_3kcompat.h
auto-patching C API usages in C:\Users\user\AppData\Local\Temp\pip-install-puaai2it\numpy_0ebbbb2e595c4f5eb6bb296fcdd0e7bf\numpy\_core\src\multiarray\compiled_base.c
auto-patching C API usages in C:\Users\user\AppData\Local\Temp\pip-install-puaai2it\numpy_0ebbbb2e595c4f5eb6bb296fcdd0e7bf\numpy\_core\src\multiarray\dlpack.c
auto-patching C API usages in C:\Users\user\AppData\Local\Temp\pip-install-puaai2it\numpy_0ebbbb2e595c4f5eb6bb296fcdd0e7bf\numpy\_core\src\multiarray\dtype_transfer.c
auto-patching C API usages in C:\Users\user\AppData\Local\Temp\pip-install-puaai2it\numpy_0ebbbb2e595c4f5eb6bb296fcdd0e7bf\numpy\_core\src\multiarray\methods.c
auto-patching C API usages in C:\Users\user\AppData\Local\Temp\pip-install-puaai2it\numpy_0ebbbb2e595c4f5eb6bb296fcdd0e7bf\numpy\_core\src\multiarray\stringdtype\dtype.c
auto-patching C API usages in C:\Users\user\AppData\Local\Temp\pip-install-puaai2it\numpy_0ebbbb2e595c4f5eb6bb296fcdd0e7bf\numpy\_core\src\umath\override.c
auto-patching C API usages in C:\Users\user\AppData\Local\Temp\pip-install-puaai2it\numpy_0ebbbb2e595c4f5eb6bb296fcdd0e7bf\numpy\_core\src\umath\_rational_tests.c
Looking for GraalPy patches for numpy
Patching package numpy using C:\Users\user\.pyenv\pyenv-win\versions\graalpy-24.1.0-windows-amd64\lib-graalpython\patches\numpy\numpy-2.0.0.patch
(Stripping trailing CRs from patch; use --binary to disable.)
patching file numpy/_core/include/numpy/ndarrayobject.h
(Stripping trailing CRs from patch; use --binary to disable.)
patching file numpy/_core/src/multiarray/compiled_base.c
(Stripping trailing CRs from patch; use --binary to disable.)
patching file numpy/_core/src/multiarray/shape.c
(Stripping trailing CRs from patch; use --binary to disable.)
patching file numpy/_core/src/multiarray/stringdtype/dtype.c
Hunk #1 succeeded at 832 (offset 15 lines).
(Stripping trailing CRs from patch; use --binary to disable.)
patching file numpy/_core/src/multiarray/temp_elide.c
(Stripping trailing CRs from patch; use --binary to disable.)
patching file numpy/_core/src/npymath/ieee754.c.src
(Stripping trailing CRs from patch; use --binary to disable.)
patching file numpy/_core/src/npymath/ieee754.cpp
(Stripping trailing CRs from patch; use --binary to disable.)
patching file vendored-meson/meson/mesonbuild/utils/universal.py
Hunk #1 succeeded at 728 (offset 1 line).
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started


  Preparing metadata (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [14 lines of output]
      + C:\Users\user\.pyvenv\Scripts\python.exe C:\Users\user\AppData\Local\Temp\pip-install-puaai2it\numpy_0ebbbb2e595c4f5eb6bb296fcdd0e7bf\vendored-meson\meson\meson.py setup C:\Users\user\AppData\Local\Temp\pip-install-puaai2it\numpy_0ebbbb2e595c4f5eb6bb296fcdd0e7bf C:\Users\user\AppData\Local\Temp\pip-install-puaai2it\numpy_0ebbbb2e595c4f5eb6bb296fcdd0e7bf\.mesonpy-1dem3pcf -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=C:\Users\user\AppData\Local\Temp\pip-install-puaai2it\numpy_0ebbbb2e595c4f5eb6bb296fcdd0e7bf\.mesonpy-1dem3pcf\meson-python-native-file.ini
      The Meson build system
      Version: 1.4.99
      Source dir: C:\Users\user\AppData\Local\Temp\pip-install-puaai2it\numpy_0ebbbb2e595c4f5eb6bb296fcdd0e7bf
      Build dir: C:\Users\user\AppData\Local\Temp\pip-install-puaai2it\numpy_0ebbbb2e595c4f5eb6bb296fcdd0e7bf\.mesonpy-1dem3pcf
      Build type: native build
      Project name: NumPy
      Project version: 2.0.2

      ..\meson.build:1:0: ERROR: Found GNU link.exe instead of MSVC link.exe in C:\Program Files\Git\usr\bin\link.EXE.
      This link.exe is not a linker.
      You may need to reorder entries to your %PATH% variable to resolve this.

      A full log can be found at C:\Users\user\AppData\Local\Temp\pip-install-puaai2it\numpy_0ebbbb2e595c4f5eb6bb296fcdd0e7bf\.mesonpy-1dem3pcf\meson-logs\meson-log.txt
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

installing matplotlib resulted in something quiet similar, it only took even longer and error was slightly different...
And trying to run it in a Windows PowerShell was about as "successful" as running it in regular cmd...

[SimplyProgrammer]

Isn't there maybe a problem with the virtual environment, like it was generated incorrectly or something?
I also find it rather strange that I have to generate this virtual environment thing in order to use pip and download packages, especially when regular Python has pip build in out of the box...
Also this is maybe unrelated but even if those packages where installed successfully, will they be installed into the main Graalpy installation or only to the virtual environment? Because if only to the virtual environment then I will have to explicitly wire it into system environmental variables before I will be able to use it which is quiet tedious and unnecessary step... but yes I accept that this whole thing is essentially still in beta...

[msimacek]

Well "legacy windows rendered" is strange. My laptop is kinda "naughty" sometimes but certainly not old, its only 2-something years old, and mostly reliable.
I am running Windows 10 22H2 - x64 on Intel i7 with only integrated iRISx graphics. 16G of RAM.
I am not running it in VM nor remotely... I was running it with plain old Windows cmd...

My VM was Windows 11, I need to try Windows 10.

And when trying to run it with the mentioned --progress-bar off it protested that there is "no such option: --progress-bar".

Did you pass it after the install? Like this:
pip install --progress-bar off click
(please test with some pure-python package first, like click above, so we can separate the pip issues from native build issues)

I mean quite LONG waiting

That's unfortunately expected at this point. numpy is a native package written in C. On CPython, numpy ships prebuild binary wheels, but those are only for CPython (they are not even compatible across different CPython version, not to mention other interpreters). We ship some prebuilt wheels ourselves, but currently only for Linux x86_64. So it needs to be built from source on your machine. pip's "Preparing metadata" message is misleading, it's actually building the package and that always takes time.

Regarding the failure, it seems that it's picking up compiler tools from GitBash. I'm not familiar with GitBash, but I think those tools are not meant to do native builds for windows, they are for mingw. The package build needs to pick up Visual Studio's compiler and tools from your system. I think we first need to deal with the console issue and run it outside GitBash.

I also find it rather strange that I have to generate this virtual environment thing in order to use pip and download packages, especially when regular Python has pip build in out of the box...

You don't strictly have to create a virtualenv, we just recommend it. Installing directly into the installation that you got with pyenv should work as well.

Also this is maybe unrelated but even if those packages where installed successfully, will they be installed into the main Graalpy installation or only to the virtual environment? Because if only to the virtual environment then I will have to explicitly wire it into system environmental variables before I will be able to use it which is quiet tedious and unnecessary step... but yes I accept that this whole thing is essentially still in beta...

If you installed packages into a virtual environment, they will be available only in that environment. If you're embedding into Java, you can use our maven/gradle plugins that can create and wire the virtualenv for you (example) or wire it yourself, it's just one option (example) and you can point it to a relative path.

[SimplyProgrammer]

Well, there is 1 thing. It seems that I was mistaken when I originally wrote that I am using GraalPy 23.1.0. I screwed up the major version. It is GraalPy 24.1.0 (currently the latest version) that I am using not 23. Sorry for this. I hope it does not change the situation in a significant way.

To the original problem. Yes you are right I was running it as python.exe -m pip --no-cache-dir --progress-bar off install...
After running it like python.exe -m pip --no-cache-dir install --progress-bar off click it indeed worked:

But when trying to python.exe -m pip --no-cache-dir install --progress-bar off numpy it presumably worked at first, but then it unfortunately "croked" on seemingly the same thing as the GitBash attempt...

About the "slow" thing. Yes I have seen the Stepan Sindelars presentation about GraalPy so yes I can understand and live with "slow" but "not working at all" is a bit more problematic...

If you installed packages into a virtual environment, they will be available only in that environment.
That maybe changes the situation a bit... I am Java developer but I am not planning to use Graal py for embedding python into Java (yet).

Perhaps I should have clarified what I want to use GraalPy for... I am a University student and we have a subject about AI. And one of the assignments is about Clustering and Classification algorithms (k-means). And that thing is quiet resource expensive... I mean really slow to run. So any kind of performance increase is welcomed. So I am trying to use GraalPy as a replacement for the default CPython. But there are libraries that I want to use, NumPy is one of them... others are Tkinter, Matplotlib and Mplcursors for visualization...
So that's why virtual environment is not very desirable for me.

You don't strictly have to create a virtualenv, we just recommend it. Installing directly into the installation that you got with pyenv should work as well.

So ifit is possible, then I would like to know how. Because trying to run pip from the default graalpy installation downloaded by pyenv does not seem to include them:

Since pip does not seem to be the part of python, python3 nor graalpy commands...

[msimacek]

But when trying to python.exe -m pip --no-cache-dir install --progress-bar off numpy it presumably worked at first, but then it unfortunately "croked" on seemingly the same thing as the GitBash attempt...

It's a different error. It seems now it can't find the compiler at all. You need to have MSVC compiler available. @timfel do we have some guide how to set it up?

Perhaps I should have clarified what I want to use GraalPy for... I am a University student and we have a subject about AI. And one of the assignments is about Clustering and Classification algorithms (k-means). And that thing is quiet resource expensive... I mean really slow to run. So any kind of performance increase is welcomed. So I am trying to use GraalPy as a replacement for the default CPython. But there are libraries that I want to use, NumPy is one of them... others are Tkinter, Matplotlib and Mplcursors for visualization...

The native libraries like numpy won't be any faster, they run the same C code under every interpreter. You can only get speed ups for the python code. Also we currently don't support the Tkinter standard module at all.

Because trying to run pip from the default graalpy installation downloaded by pyenv does not seem to include them

Then I think that's a bug in pyenv-win, because the "normal" pyenv includes it. @timfel is it something we should make a PR for?

[timfel]

Then I think that's a bug in pyenv-win, because the "normal" pyenv includes it. @timfel is it something we should make a PR for?

From what I can see this is how pyenv-win works, also for PyPy.

@SimplyProgrammer It seems to me you are not running in a visual studio shell? The only supported + tested configuration for installing GraalPy on Windows is using Windows 11 and the latest Visual Studio Build Tools 2022, and running the pip install in an activated venv in a Developer PowerShell for VS 2022. I just tried this on my machine, with graalpy-24.1.0 installed via pyenv-win and a venv I had lying around without anything it it:

.\lxmlvenv\Scripts\Activate.ps1
pip install numpy

The installation looks like this: