pytensor and blas problems on on MacOS 15 Sequoia with Apple Silicon #1005
Labels
Comments
|
Thanks a lot @danieltomasz for the very high quality report. @lucianopaz, do you have any thoughts regarding the BLAS selection mechanism? |
|
One thing I learned & might be useful - numpy 2.2 installed via pip use accelerate, numpy 2.2 installed via the same conda installs with openblas (I checked this via numpy.show_config()) I installed it in separate env just to check, bc pytensor doesnt support yet numpy >= 2.0 |
|
That is indeed very interesting, thanks @danieltomasz. The Conda dependency chain is:
One way to get more flexibility to help debug this is to instead use the @danieltomasz, does this give you something to experiment with? I don't have a Mac myself, so unfortunately I can't directly debug this. |
|
When I force It install with success, but gives me errors (segmentation faults) Some of those error are discussed here https://discourse.pymc.io/t/environment-not-working-anymore-on-macos/14210 |
|
@danieltomasz, could you please try using the |
|
|
Ok, so that's installing numpy with openblas. And what happens now if you try and force |
|
@maresb as I wrote here it install fine, but when trying to test it give segmentation error #1005 (comment) |
|
Also the output from numpy even if I force accelerate in conda |
|
When installing only numpy with forced accelerate I need to leave, but I might try something out of box, like if installing pytensor via "pixi" pulls accelerate (there might be something particular to my setup how conda is linking and trying different package manager tool might help), maybe someone with Apple Sillicon can replicate in meantime |
|
Thanks so much for all the diagnosis @danieltomasz! For when you find some more time, I wonder if lower versions of NumPy might work? For example |
|
Unfortunately neither pixi, nor changing pyhon version to 3.11 or asking for lower version of numpy provide accelarate libraries (it is openblas by default); When I installe numpy via pip it intalled numpy 2.2 with accelerate, but adding pytensor to this envioronment downgrade numpy to one' that is using openblas64 |
Thanks @danieltomasz for getting back to me! Are you able to find some earlier conda-forge version of numpy that works with accelerate on your system? |
|
hi @maresb, I think conda and numpy worked fine earlier (the latest numpy version <2 is from february) , I cannot pinpoint exact moment, but I was probably update to MacOS 15 that changed things ~ 2 weeks ago (also recently I think that conda might changed clang compiler that it uses with the python it ships, but I am not sure about this); what could be worth to see :
|
|
It is just my intuition, but forcing blas to accelerate works, but it creates the create error when running due to problems with compilers on MacOS 15, but it works with openblas Also there was updates to accelerate in MacOS 15 https://developer.apple.com/documentation/accelerate/blas/ and this discussion might be relevant conda-forge/blas-feedstock#103 and here conda-forge/numpy-feedstock#253 |
|
Quick update: when I install via pip numpy seems to use is below I am now on MacOS 15.1 |
|
@danieltomasz, the phrase import logging
logger = logging.getLogger("pytensor.link.c.cmodule")
logger.setLevel(logging.DEBUG)
import pytensorAfter the last import, you should see all of the detailed logs from I would like to see what errors pytensor is running into when it tries to determine the |
|
The environment wasn't completely clean, but I uninstalled Output from pytensor And in the same session |
|
Thanks @danieltomasz , the logs say that we couldn’t find a blas library in the search directories. I can think of a couple of dumb causes but I’ll have to ask you to run a couple of other tests.
|
|
Hi @lucianopaz, thanks for all the comments! Regarding 2 and 3 accelerate was installed but with the following error happens #1005 (comment) this happens also with the newer version of the miniconda Regarding 4, in the pyenv installed cpython: Would be great if any other person on Apple processor can confirm it, if this is pecular to my setup or something more general (I started to have this problem after update to MacOS 15, MacOS 15 ships accelerate with blas 3.11, I wonder if this might be a problem |
|
That last thing that you tried means that we could add those flags as a check and Mac would link to Accelerate. I'll open a small patch PR so that you can try it out. |
11 tasks
|
@danieltomasz, try this PR out. It should set |
|
@lucianopaz seems promising |
|
but with the above flag results of are defaulting to the error I posted above Results of |
|
Awesome @danieltomasz! I can reproduce that problem locally now. The latest commit to the PR I had mentioned before should have fixed it. Let me know if it works for you. If it did, I'll try to setup a test on Mac ARM in our CI matrix so that this can be verified. |
|
Thanks @lucianopaz, everything seems to work ok now with cpython and pip install! and |
|
@danieltomasz, this should now be fixed with #1056. If you want, you can try to run from the current pytensor main branch and check if it works. I had to do a bunch of extra changes to ensure compilation actually used blas symbols. |
|
nice, after running flags are different now And the time of running is shorter (down to around 10-13s from 14-16s) |
Yes, I changed the flags to make them aligned with what other blas flag specs that we use. And the execution time should be shorter because it’s actually linking to accelerate now. Before, it was failing to do so because of things that were handing downstream. |
This is great news! Is there a way to have my conda environment use this version of PyTensor? Or alternatively, when is the next release going to be such that this code is available to be installed normally via conda? |
|
@Edderic just did: https://github.com/pymc-devs/pytensor/releases/tag/rel-2.26.0 However if you are using PyTensor for PyMC, that will also need a bump in the dependency due to major changes. |
|
I've been following this thread as I recently got an M1 Mac, too, and I'm still not getting Accelerate to work with 2.26.0 :/ Here's the output with logging enabled in a fresh conda environment created with Replicating some of the other tests that @lucianopaz asked above:
Yes it is.
If I try the same commands in an environment with From what I can see, |
|
This didn't work for me on a fresh Miniforge install on MacOS 15.1. My pytensor logging output is below. I created my environment using:
The pytensor import logs: Running
outputs: I then ran:
Same result when trying the above. I tried a few combinations and orders of this, including installing pytensor-base first. Finally I gave up and created a fresh environment with only python=3.12, then used pip to install pytensor. Now the check_blas.py output is: |
|
Thanks @aurimas-ww and @areding. I also started to run into a similar problem in a different environment. I think that I know the cause, it’s related to the linker that Mac deployed with Xcode 15. Both of your logs say that a symbol wasn’t found in the tapi library, but the next line says that the linker actually died trying to go through that dylib with a signal. On my machine I managed to see a segfault with signal 11. Googling around, I found this thread that might have the solution we need. The explanatory post says that Xcode 15 brought on a new linker. This linker seems to not like the layout of some dylib and dies. We can luckily ask Xcode’s linker to behave like the good old linker by supplying some flags to the compiler. I haven’t gotten around to implementing this in pytensor yet, but maybe you could try to add a configuration flag for the extra compile flags that has |
|
Not sure if that's the right way to do it, but adding these flags (either just |
That’s because you left out the rpath from the compilation flags. You don’t need to drop them, just add a new flag |
11 tasks
|
@aurimas-ww and @areding, we just merged #1083. Could you please install pytensor from the current state of the main branch and check if your issues go away? We can't be sure if this was fixed with the CI runs only because we haven't found a snippet that fails systematically, but the people that we've asked so far told us that their setup is working now. |
|
That did it. Thank you! |
|
Same here - I can confirm it works on my machine, too. Thanks!! |
|
Hey, can you guys post the steps you used to get a successful installation? I'm able to replicate each of the steps outlined by @aurimas-ww, but when pip installing |
|
@scroobiustrip - I just tried it in a fresh virtual environment, simple (uv) pip install - and it worked: |
|
@ricardoV94, we should make a release with the patch. Just a bug fix release. That way everyone can just use conda or pip as usual |
@lucianopaz already did and also the latest PyMC is linked to it |
I don't see #1083 commits included in the latest release. Those were the ones that made the last errors go away. |
|
@lucianopaz my bad. I'll release now. Since it's not a major release it should become automatically compatible with PyMC |
|
@lucianopaz can you edit the PR title to be more informative than "LD64"? |
Done |
|
Patch is in https://github.com/pymc-devs/pytensor/releases/tag/rel-2.26.3 Should be available in the common channels soon |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the issue:
Since update to MacOS 15 I have a problem with using Apple implementation of BLAS.
Installing
pytensorfromminiconda3-3.12-24.7.1-0viaconda create -n voxel-bayes-3.12 -c conda-forge pytensorseems to installopenblasinstead of accelerate.Running this the check
And when I try to run the same command but in env with pip installed pytensor results in this
When I try to specify the accelerate the old way via "libblas=*=*accelerate" when installing the conda environment, when I try to run this it fails , I copied the output here https://discourse.pymc.io/t/pytensor-support-to-apple-accelerate-blas-with-conda-forge-on-macos-15/15131/2
Reproducable code example:
Error message:
No response
PyTensor version information:
conda-forge/osx-arm64::pytensor-2.25.4-py312h3f593ad_0
Context for the issue:
No response
The text was updated successfully, but these errors were encountered: