[Python] a segfault on import pyarrow on MacOS 11.6 after pip update to pyarrow version 16.1.0
#41696
Comments
|
@ianna thanks for the report! The crash seems to be related to jemalloc. Not a solution, but just curious: could you set |
@jorisvandenbossche - thanks for prompt reply! Setting the env variable does not help :-( % python
Python 3.11.5 (main, Sep 11 2023, 08:19:27) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> print(os.environ["ARROW_DEFAULT_MEMORY_POOL"])
system
>>> import pyarrow
zsh: segmentation fault python |
|
I can reproduce on MacOS x86_64 with |
|
I have the same
|
until the issue apache/arrow#41696 is fixed we cannot use 16.1.0
until the issue apache/arrow#41696 is fixed we cannot use 16.1.0
|
I did a quick google of the stack trace, and there's a note in jemalloc specifically about the function that crashes (may or may not be related): Reproduced: Some links to possible places referred to by the stack trace:
|
|
The diff between 16.0.0 and 16.1.0 is here: apache-arrow-16.0.0...apache-arrow-16.1.0. The only change that seems somewhat related is #41567 (unless the fixes for scalar scratch space fix for some reason caused this), although this is also strange because in the past this macos-latest should also have defaulted to macos-13 at some point, I assume? cc @raulcd |
@paleolimbot with which version of macOS could you reproduce this? |
|
I tested this on MacOS 11! |
|
@paleolimbot one more question: could you test with the latest nightly wheel (I assume nothing changed or fixed it, but just to be sure, because if it is still failing we should mark this as a blocker for the release I think) |
|
From seeing similar segfaults in other projects with a complex C++ dependency (duckdb/duckdb#12199 (comment), geopandas/pyogrio#417 (comment) building with GDAL), my general assumption is that this is something on the macOS side, and with macOS 11 no longer being supported, probably the only thing we can do is bump the deployment target for our wheels to macOS 12. That will mean that someone trying to install pyarrow on macOS 11 (as done in this issue) will no longer have a wheel available and then pip will try to install from source. That will typically fail at installation time, but at least not segfault at runtime .. |
I just tested this again (MacOS 11.7.10, Python 3.11) and it resulted in a segfault as before! |
|
Should we bump the deployment target to macOS 12 before the 17.0.0 release? @jorisvandenbossche @pitrou |
FYI, The message from github actions is that "The macOS-11 environment is deprecated and will be removed on June 28th, 2024." |
|
Since MacOS 11 is no longer supported by the manufacturer, I think it would be fine to bump to MacOS 12. Sounds like GH actions will force us to drop it anyways. |
|
I don't think this is a release blocker, as this is already an issue, and I am facing some issues when building and linking Arrow with |
…tead of 11 (#43137) ### Rationale for this change As shown on the associated issue there seems to be a problem with `MACOSX_DEPLOYMENT_TARGET` 11 on the wheels. ### What changes are included in this PR? Update `MACOSX_DEPLOYMENT_TARGET` everywhere to the latest supported macOS version. ### Are these changes tested? Via CI, even though the issue was not reproducible on CI. ### Are there any user-facing changes? Yes, wheels won't be available for macOS 11 but those were crashing on the previous release. * GitHub Issue: #41696 Authored-by: Raúl Cumplido <[email protected]> Signed-off-by: Raúl Cumplido <[email protected]>
|
Issue resolved by pull request 43137 |
…12 instead of 11 (apache#43137) ### Rationale for this change As shown on the associated issue there seems to be a problem with `MACOSX_DEPLOYMENT_TARGET` 11 on the wheels. ### What changes are included in this PR? Update `MACOSX_DEPLOYMENT_TARGET` everywhere to the latest supported macOS version. ### Are these changes tested? Via CI, even though the issue was not reproducible on CI. ### Are there any user-facing changes? Yes, wheels won't be available for macOS 11 but those were crashing on the previous release. * GitHub Issue: apache#41696 Authored-by: Raúl Cumplido <[email protected]> Signed-off-by: Raúl Cumplido <[email protected]>
…12 instead of 11 (apache#43137) ### Rationale for this change As shown on the associated issue there seems to be a problem with `MACOSX_DEPLOYMENT_TARGET` 11 on the wheels. ### What changes are included in this PR? Update `MACOSX_DEPLOYMENT_TARGET` everywhere to the latest supported macOS version. ### Are these changes tested? Via CI, even though the issue was not reproducible on CI. ### Are there any user-facing changes? Yes, wheels won't be available for macOS 11 but those were crashing on the previous release. * GitHub Issue: apache#41696 Authored-by: Raúl Cumplido <[email protected]> Signed-off-by: Raúl Cumplido <[email protected]>
Describe the bug, including details regarding any error messages, version, and platform.
Hi,
I running into an issue with a newly released 16.1.0 version. The 16.0.0 works without issues, but updating to 16.1.0 causes a segfault. Any help is greatly appreciated. Thanks!
The
test_pyarrow.pyhas only one line:import pyarrow:Component(s)
Python, Release
The text was updated successfully, but these errors were encountered: