Rebuild for PyPy3.8 and PyPy3.9

[regro-cf-autotick-bot]

This PR has been triggered in an effort to update pypy38.

Notes and instructions for merging this PR:

1. Please merge the PR only after the tests have passed.
2. Feel free to push to the bot's branch to update this PR if needed.

Please note that if you close this PR we presume that the feedstock has been rebuilt, so if you are going to perform the rebuild yourself don't close this PR until the your rebuild has been merged.

If this PR was opened in error or needs to be updated please add the bot-rerun label to this PR. The bot will close this PR and schedule another one. If you do not have permissions to add this label, you can use the phrase @conda-forge-admin, please rerun bot in a PR comment to have the conda-forge-admin add it for you.

_{This PR was created by the regro-cf-autotick-bot. The regro-cf-autotick-bot is a service to automatically track the dependency graph, migrate packages, and propose package version updates for conda-forge. Feel free to drop us a line if there are any issues! This PR was generated by https://github.com/regro/autotick-bot/actions/runs/2182200263, please use this URL for debugging.}

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[h-vetinari]

@conda-forge-admin, please rerender

[h-vetinari]

Error is:

/home/conda/feedstock_root/build_artifacts/arrow-cpp-ext_1663434993458/work/cpp/src/arrow/python/datetime.cc: In function 'void arrow::py::internal::InitDatetime()':
/home/conda/feedstock_root/build_artifacts/arrow-cpp-ext_1663434993458/work/cpp/src/arrow/python/datetime.cc:104:59: error: 'PyDateTime_CAPSULE_NAME' was not declared in this scope
  104 |       reinterpret_cast<PyDateTime_CAPI*>(PyCapsule_Import(PyDateTime_CAPSULE_NAME, 0));
      |                                                           ^~~~~~~~~~~~~~~~~~~~~~~
/home/conda/feedstock_root/build_artifacts/arrow-cpp-ext_1663434993458/work/cpp/src/arrow/python/datetime.cc: In function 'PyObject* arrow::py::internal::MonthDayNanoIntervalToNamedTuple(const arrow::MonthDayNanoIntervalType::MonthDayNanos&)':
/home/conda/feedstock_root/build_artifacts/arrow-cpp-ext_1663434993458/work/cpp/src/arrow/python/datetime.cc:609:3: error: 'PyStructSequence_SetItem' was not declared in this scope; did you mean 'PyPySequence_SetItem'?
  609 |   PyStructSequence_SetItem(tuple.obj(), /*pos=*/0, PyLong_FromLong(interval.months));
      |   ^~~~~~~~~~~~~~~~~~~~~~~~
      |   PyPySequence_SetItem

There's an immediately preceding warning that arrow redefines PyDateTimeAPI coming from PyPy, which is likely the culprit:

In file included from /home/conda/feedstock_root/build_artifacts/arrow-cpp-ext_1663434993458/work/cpp/src/arrow/python/datetime.cc:17:
/home/conda/feedstock_root/build_artifacts/arrow-cpp-ext_1663434993458/work/cpp/src/arrow/python/datetime.h:35: warning: "PyDateTimeAPI" redefined
   35 | #define PyDateTimeAPI ::arrow::py::internal::datetime_api
      | 
In file included from /home/conda/feedstock_root/build_artifacts/arrow-cpp-ext_1663434993458/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/include/pypy3.8/Python.h:122,
                 from /home/conda/feedstock_root/build_artifacts/arrow-cpp-ext_1663434993458/work/cpp/src/arrow/python/platform.h:27,
                 from /home/conda/feedstock_root/build_artifacts/arrow-cpp-ext_1663434993458/work/cpp/src/arrow/python/datetime.h:23,
                 from /home/conda/feedstock_root/build_artifacts/arrow-cpp-ext_1663434993458/work/cpp/src/arrow/python/datetime.cc:17:
/home/conda/feedstock_root/build_artifacts/arrow-cpp-ext_1663434993458/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/include/pypy3.8/pypy_decl.h:1206: note: this is the location of the previous definition
 1206 | #define PyDateTimeAPI PyPyDateTimeAPI
      |

CC @mattip

[mattip]

The datetime API problems sounded familiar, I tracked down arrow-2651 which led to this patch. I don't know if that ever made it into upstream. There is also this fork which at least in 2020 built and ran on PyPy.

[h-vetinari]

Thanks for the links! As it so happens, that issue was reopened recently, and I commented there. There's some interest in making this happen by some consumers. I pointed them to your patch and offered to test any work that comes of this. Just FYI. :)

[h-vetinari]

So I backported apache/arrow#14539, but we're not getting that much further it seems:

[4/29] Building CXX object CMakeFiles/arrow_python_objlib.dir/arrow/python/datetime.cc.o
FAILED: CMakeFiles/arrow_python_objlib.dir/arrow/python/datetime.cc.o 
/home/conda/feedstock_root/build_artifacts/apache-arrow_1670149527249/_build_env/bin/x86_64-conda-linux-gnu-c++ -DARROW_HAVE_RUNTIME_AVX2 -DARROW_HAVE_RUNTIME_AVX512 -DARROW_HAVE_RUNTIME_BMI2 -DARROW_HAVE_RUNTIME_SSE4_2 -DARROW_PYTHON_EXPORTING -isystem /home/conda/feedstock_root/build_artifacts/apache-arrow_1670149527249/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/pypy3.9/site-packages/numpy/core/include -isystem /home/conda/feedstock_root/build_artifacts/apache-arrow_1670149527249/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/include/pypy3.9 -isystem /home/conda/feedstock_root/build_artifacts/apache-arrow_1670149527249/work/python/pyarrow/src -isystem /home/conda/feedstock_root/build_artifacts/apache-arrow_1670149527249/work/python/pyarrow/src/src -Wall -fno-semantic-interposition -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/conda/feedstock_root/build_artifacts/apache-arrow_1670149527249/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/apache-arrow_1670149527249/work=/usr/local/src/conda/pyarrow-10.0.1 -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/apache-arrow_1670149527249/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol=/usr/local/src/conda-prefix -isystem /usr/local/cuda/include -fdiagnostics-color=always -fuse-ld=gold -O2 -DNDEBUG -ftree-vectorize   -DNDEBUG -fPIC -std=c++1z -MD -MT CMakeFiles/arrow_python_objlib.dir/arrow/python/datetime.cc.o -MF CMakeFiles/arrow_python_objlib.dir/arrow/python/datetime.cc.o.d -o CMakeFiles/arrow_python_objlib.dir/arrow/python/datetime.cc.o -c /home/conda/feedstock_root/build_artifacts/apache-arrow_1670149527249/work/python/pyarrow/src/arrow/python/datetime.cc
/home/conda/feedstock_root/build_artifacts/apache-arrow_1670149527249/work/python/pyarrow/src/arrow/python/datetime.cc: In function 'PyObject* arrow::py::internal::MonthDayNanoIntervalToNamedTuple(const arrow::MonthDayNanoIntervalType::MonthDayNanos&)':
/home/conda/feedstock_root/build_artifacts/apache-arrow_1670149527249/work/python/pyarrow/src/arrow/python/datetime.cc:612:3: error: 'PyStructSequence_SetItem' was not declared in this scope
   PyStructSequence_SetItem(tuple.obj(), /*pos=*/0, PyLong_FromLong(interval.months));
   ^~~~~~~~~~~~~~~~~~~~~~~~
/home/conda/feedstock_root/build_artifacts/apache-arrow_1670149527249/work/python/pyarrow/src/arrow/python/datetime.cc:612:3: note: suggested alternative: 'PyPySequence_SetItem'
   PyStructSequence_SetItem(tuple.obj(), /*pos=*/0, PyLong_FromLong(interval.months));
   ^~~~~~~~~~~~~~~~~~~~~~~~
   PyPySequence_SetItem
[5/29] Building CXX object CMakeFiles/arrow_python_objlib.dir/arrow/python/deserialize.cc.o
[6/29] Building CXX object CMakeFiles/arrow_python_objlib.dir/arrow/python/arrow_to_pandas.cc.o
/home/conda/feedstock_root/build_artifacts/apache-arrow_1670149527249/work/python/pyarrow/src/arrow/python/arrow_to_pandas.cc: In function 'arrow::enable_if_t<std::is_same<Type, arrow::MonthDayNanoIntervalType>::value, arrow::Status> arrow::py::{anonymous}::ObjectWriterVisitor::Visit(const Type&) [with Type = arrow::MonthDayNanoIntervalType]':
/home/conda/feedstock_root/build_artifacts/apache-arrow_1670149527249/work/python/pyarrow/src/arrow/python/arrow_to_pandas.cc:611:5: warning: 'memo_index' may be used uninitialized in this function [-Wmaybe-uninitialized]
     if (memo_index == memo_size) {
     ^~
/home/conda/feedstock_root/build_artifacts/apache-arrow_1670149527249/work/python/pyarrow/src/arrow/python/arrow_to_pandas.cc:609:13: note: 'memo_index' was declared here
     int32_t memo_index;
             ^~~~~~~~~~
ninja: build stopped: subcommand failed.

[mattip]

PyStructSequence_SetItem is part of the upcoming 7.3.10 release

[mattip]

I want to try this out locally, but I do not see pypy-specific files in ./ci_support. Has the command line interface for building locally changed?

[h-vetinari]

I want to try this out locally, but I do not see pypy-specific files in ./ci_support. Has the command line interface for building locally changed?

This recipe has been collapsed into a job per architecture that does all the python versions. To explode it back to what it was pre arrow 10, you can add a "fake" python host dependence to libarrow and rerender.

[mattip]

I added a line - python in the host section of libarrow, and tried to rerender. Is this a known error?:

$ conda smithy rerender

<trimmed lots of logging>

INFO:conda_smithy.configure_feedstock:Applying migrations: /home/matti/oss/arrow-cpp-feedstock/.ci_support/migrations/cuda_112_ppc64le_aarch64.yaml,/tmp/tmpvmsl6wvg/share/conda-forge/migrations/pypy38.yaml,/tmp/tmpvmsl6wvg/share/conda-forge/migrations/python311.yaml,/home/matti/oss/arrow-cpp-feedstock/.ci_support/migrations/aws_sdk_cpp19375.yaml,/home/matti/oss/arrow-cpp-feedstock/.ci_support/migrations/ucx1410.yaml,/tmp/tmpvmsl6wvg/share/conda-forge/migrations/libgrpc154.yaml,/tmp/tmpvmsl6wvg/share/conda-forge/migrations/libevent2112.yaml
Traceback (most recent call last):
  File "/home/matti/miniconda3/envs/python3.9/bin/conda-smithy", line 10, in <module>
    sys.exit(main())
  File "/home/matti/miniconda3/envs/python3.9/lib/python3.9/site-packages/conda_smithy/cli.py", line 669, in main
    args.subcommand_func(args)
  File "/home/matti/miniconda3/envs/python3.9/lib/python3.9/site-packages/conda_smithy/cli.py", line 485, in __call__
    self._call(args, tmpdir)
  File "/home/matti/miniconda3/envs/python3.9/lib/python3.9/site-packages/conda_smithy/cli.py", line 490, in _call
    configure_feedstock.main(
  File "/home/matti/miniconda3/envs/python3.9/lib/python3.9/site-packages/conda_smithy/configure_feedstock.py", line 2318, in main
    render_azure(env, config, forge_dir, return_metadata=True)
  File "/home/matti/miniconda3/envs/python3.9/lib/python3.9/site-packages/conda_smithy/configure_feedstock.py", line 1412, in render_azure
    return _render_ci_provider(
  File "/home/matti/miniconda3/envs/python3.9/lib/python3.9/site-packages/conda_smithy/configure_feedstock.py", line 662, in _render_ci_provider
    migrated_combined_variant_spec = migrate_combined_spec(
  File "/home/matti/miniconda3/envs/python3.9/lib/python3.9/site-packages/conda_smithy/configure_feedstock.py", line 595, in migrate_combined_spec
    combined_spec = variant_add(combined_spec, migration)
  File "/home/matti/miniconda3/envs/python3.9/lib/python3.9/site-packages/conda_smithy/variant_algebra.py", line 239, in variant_add
    return VARIANT_OP[operation](v1, v2)
  File "/home/matti/miniconda3/envs/python3.9/lib/python3.9/site-packages/conda_smithy/variant_algebra.py", line 168, in op_variant_key_add
    new_value[new_key_position] = v2[key][pkey_ind]
KeyError: 'cuda_compiler'

[h-vetinari]

Sorry, that was a left-over of some local migration that conflicted with newer changes to the global pinning. I've done this for you now.

[h-vetinari]

... except, the jobs weren't split up yet. 🤦

[h-vetinari]

... except, the jobs weren't split up yet. 🤦

@mattip, this builds correctly now and reproduces the previous failures.

[mattip]

Thanks!

[mattip]

The segfault is in __pyx_getprop_6pandas_5_libs_6tslibs_10timedeltas_10_Timedelta_days, which comes from site-packages/pandas/_libs/tslibs/timedeltas.pypy39-pp73-x86_64-linux-gnu.so, when running the test_array.py::test_array_from_numpy_timedelta[timedelta64[ns]-type3] test. The numpy array

array([         'NaT', 86400000000000,     1000000000],
      dtype='timedelta64[ns]')

is turned into the pyarrow.lib.DurationArray via arr = pa.array(np_arr)

<pyarrow.lib.DurationArray object at 0x000055bb4b65ff20>
[
  null,
  86400000000000,
  1000000000
]

Even if I do

np_arr2 = np.array([1, 1, 1], dtype=dtype);
arr2 = pa.array(np_arr2);
arr2.to_pylist()

I still get a segfault, so the problem is somewhere with the timedelta64[ns] dtype. Maybe inside pandas?

[mattip]

yup, pandas.array(np.array([1, 1, 1], dtype="timdelta64[ns]")) segfaults, without pyarrow

[mattip]

OK, this is pandas-dev/pandas#50817

[mattip]

TL;DR: the problem is due to PyPy, not Cython nor pandas, and exposed in the pyarrow tests.

The problem is:

• pandas overrides the timedelta.days getter with a cython-based one that assumes the object is fully constructed (i.e. it has set the __pyx_vtab, which is done one line after calling tp_new()
• pypy assumed it could safely call that getter descriptor when converting a pure-python datetime.timedelta into a C PyDateTime_Delta object with a days field in the C struct
• When calling tp_new() from C, it is not safe to use the descriptor, since __pyx_vtab is not set.

The solution is for PyPy to not assume it can call the days getter from C, rather it should use the private _days attribute which is well defined once the python object is created. Fixed in 267c2f5eca33

I guess I should make a patch to the pypy feedstock.

[h-vetinari]

Good news: conda-forge/pypy3.6-feedstock#103 did take effect.

Bad news: whack-a-segfault continues

test_fs.py::test_filesystem_is_functional_after_pickling[HadoopFileSystem] SKIPPED [ 26%]
test_fs.py::test_filesystem_is_functional_after_pickling[_MockFileSystem()] XFAIL [ 26%]
test_fs.py::test_filesystem_is_functional_after_pickling[PyFileSystem(ProxyHandler(LocalFileSystem()))] PASSED [ 26%]
Fatal Python error: Segmentation fault

The corresponding test on CPython is failing too, but marked as such:

test_fs.py::test_filesystem_is_functional_after_pickling[HadoopFileSystem] SKIPPED [ 26%]
test_fs.py::test_filesystem_is_functional_after_pickling[_MockFileSystem()] XFAIL [ 26%]
test_fs.py::test_filesystem_is_functional_after_pickling[PyFileSystem(ProxyHandler(LocalFileSystem()))] PASSED [ 26%]
test_fs.py::test_filesystem_is_functional_after_pickling[PyFileSystem(ProxyHandler(_MockFileSystem()))] XFAIL [ 26%]

[mattip]

I am seeing a segfault before that, in test_flight.py Here is the C call stack

#0  0x00007fff2117ecb9 in arrow::flight::FlightClient::Close() ()
   from /home/matti/miniconda3/envs/python3.9/conda-bld/apache-arrow_1691664817326/_test_env/lib/pypy3.9/site-packages/pyarrow/../../../libarrow_flight.so.1200
#1  0x00007fff212b388b in __pyx_pw_7pyarrow_7_flight_12FlightClient_28close(_object*, _object*) ()
   from /home/matti/miniconda3/envs/python3.9/conda-bld/apache-arrow_1691664817326/_test_env/lib/pypy3.9/site-packages/pyarrow/_flight.pypy39-pp73-x86_64-linux-gnu.so
#2  0x00007ffff60ebbca in ?? () from /home/matti/miniconda3/envs/python3.9/conda-bld/apache-arrow_1691664817326/_test_env/bin/../lib/libpypy3.9-c.so
#3  0x00007ffff592c155 in ?? () from /home/matti/miniconda3/envs/python3.9/conda-bld/apache-arrow_1691664817326/_test_env/bin/../lib/libpypy3.9-c.so
#4  0x00007ffff5ada304 in ?? () from /home/matti/miniconda3/envs/python3.9/conda-bld/apache-arrow_1691664817326/_test_env/bin/../lib/libpypy3.9-c.so
#5  0x00007ffff5a32030 in ?? () from /home/matti/miniconda3/envs/python3.9/conda-bld/apache-arrow_1691664817326/_test_env/bin/../lib/libpypy3.9-c.so
#6  0x00007ffff5ad97ee in ?? () from /home/matti/miniconda3/envs/python3.9/conda-bld/apache-arrow_1691664817326/_test_env/bin/../lib/libpypy3.9-c.so
#7  0x00007ffff5fafa15 in ?? () from /home/matti/miniconda3/envs/python3.9/conda-bld/apache-arrow_1691664817326/_test_env/bin/../lib/libpypy3.9-c.so
#8  0x00007ffff5fcc8e1 in ?? () from /home/matti/miniconda3/envs/python3.9/conda-bld/apache-arrow_1691664817326/_test_env/bin/../lib/libpypy3.9-c.so
#9  0x00007fff2128ebc2 in __pyx_pw_7pyarrow_7_flight_12FlightClient_30__del__(_object*, _object*) ()

That seems to be coming from here

     def close(self):
        """Close the client and disconnect."""
        check_flight_status(self.client.get().Close())

    def __del__(self):
        # Not ideal, but close() wasn't originally present so
        # applications may not be calling it
        self.close()

In general, it is a bad idea to call python code from a destructor (__del__ is calling close, close is calling check_flight_status which has python code in the error path).

If I skip all of test_flight, I get past the one you saw and get another one in test_io.py"

lib/pypy3.9/site-packages/pyarrow/tests/test_io.py::test_python_file_get_stream[100-5] PASSED                                                                          [ 32%]
lib/pypy3.9/site-packages/pyarrow/tests/test_io.py::test_python_file_get_stream[100-100] PASSED                                                                        [ 32%]
lib/pypy3.9/site-packages/pyarrow/tests/test_io.py::test_python_file_read_at PASSED                                                                                    [ 32%]
lib/pypy3.9/site-packages/pyarrow/tests/test_io.py::test_python_file_readall PASSED                                                                                    [ 32%]
lib/pypy3.9/site-packages/pyarrow/tests/test_io.py::test_python_file_readinto 
Thread 1 "python" received signal SIGSEGV, Segmentation fault.
__memmove_sse2_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:418
418	../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory.
(gdb) bt
#0  __memmove_sse2_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:418
#1  0x00007ffff2c2dd18 in arrow::py::PyReadableFile::Read(long, void*)::{lambda()#1}::operator()() const ()
   from /home/matti/miniconda3/envs/python3.9/conda-bld/apache-arrow_1691664817326/_test_env/lib/pypy3.9/site-packages/pyarrow/libarrow_python.so
#2  0x00007ffff2c2e047 in arrow::py::PyReadableFile::Read(long, void*) ()
   from /home/matti/miniconda3/envs/python3.9/conda-bld/apache-arrow_1691664817326/_test_env/lib/pypy3.9/site-packages/pyarrow/libarrow_python.so
#3  0x00007ffff2e4d773 in __pyx_pw_7pyarrow_3lib_10NativeFile_53readinto(_object*, _object*) ()
   from /home/matti/miniconda3/envs/python3.9/conda-bld/apache-arrow_1691664817326/_test_env/lib/pypy3.9/site-packages/pyarrow/lib.pypy39-pp73-x86_64-linux-gnu.so
#4  0x00007ffff60ec2b1 in ?? () from /home/matti/miniconda3/envs/python3.9/conda-bld/apache-arrow_1691664817326/_test_env/bin/../lib/libpypy3.9-c.so

At that point I gave, up, this seems insurmoutable without some serious effort.

[h-vetinari]

Thanks for checking in here again. This PR is quite seriously out of date, but I can rebase it after arrow 13 is out soon.

I would also be fine with patching out that __del__ for pypy. Flight has segfaults on osx too BTW (much to my chagrin, but I received no real help on that and I decided not to hold up everything until that's fixed).

The io tests sound more important though. Not sure what insights make you think that's hard to fix. I see sse2 (which should be switched off actually) and aligned memory boundaries appearing in the stack trace. Perhaps we can figure it out...

In any case, as long as arrow is not a hard dependency of pandas yet, this is definitely less urgent than fixing pandas (where you've made some progress as well AFAICT!).

[mattip]

I still see self.close() in the __del__ function so expect segfaults.

[h-vetinari]

Closing as obsolete now that pypy got dropped