GH-36730: [Python] Add support for Cython 3.0.0 #36745

kou · 2023-07-18T09:30:49Z

Rationale for this change

Cython 3.0.0 is the latest release. PyArrow should work with Cython 3.0.0.

What changes are included in this PR?

Don't use vector[XXX]&&
Add a declaration for postincrement
- See also: https://cython.readthedocs.io/en/latest/src/userguide/migrating_to_cy30.html#c-postincrement-postdecrement-operator
Ignore C4551 warning (function call missing argument list) with MSVC
- See also: [BUG] code emitted for Python 3.10 causes warning C4551 in MSVC cython/cython#4445
Add missing const to CLocation's static methods.
Don't use StopIteration to stop generator
- See also: https://cython.readthedocs.io/en/latest/src/userguide/migrating_to_cy30.html#python-3-syntax-semantics

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes.

Closes: [CI][Python] Cython 3.0 seems to make our build of pyarrow fail #36730

github-actions · 2023-07-18T09:31:20Z

⚠️ GitHub issue #36730 has been automatically assigned in GitHub to PR creator.

pitrou · 2023-07-18T10:09:06Z

This may fix the compilation error but there are other issues during tests (see macOS CI?).

kou · 2023-07-18T23:43:31Z

I've fixed CI failures.

jorisvandenbossche · 2023-07-19T07:43:00Z

@kou there were also some runtime errors in the minimal build that are caused by cython 3 (but it's strange that this is only failing in a single build). Based on your diff, I don't think that will already be fixed here, but let me run them to be sure.

jorisvandenbossche · 2023-07-19T07:43:04Z

@github-actions crossbow submit example-python-minimal-build-fedora-conda

github-actions · 2023-07-19T07:45:42Z

Revision: dba6bb4

Submitted crossbow builds: ursacomputing/crossbow @ actions-f3bbf75ac8

Task	Status
example-python-minimal-build-fedora-conda

pitrou · 2023-07-19T07:45:55Z

Can this PR also undo the pinning on various Python builds?

kou · 2023-07-19T20:48:09Z

@github-actions crossbow submit example-python-minimal-build-fedora-conda

github-actions · 2023-07-19T20:50:37Z

Revision: 81493b1

Submitted crossbow builds: ursacomputing/crossbow @ actions-492c6d5c7d

Task	Status
example-python-minimal-build-fedora-conda

kou · 2023-07-19T23:35:15Z

I've fixed the generator related problem.
I'll rebase on main and unpin Cython version.

kou · 2023-07-20T07:59:28Z

Hmm. Can we detect Cython version in .{pyx,pxi}?

kou · 2023-07-20T08:00:48Z

I want to write something like the following:

diff --git a/python/pyarrow/scalar.pxi b/python/pyarrow/scalar.pxi
index f438c8847..61de38203 100644
--- a/python/pyarrow/scalar.pxi
+++ b/python/pyarrow/scalar.pxi
@@ -793,7 +793,10 @@ cdef class MapScalar(ListScalar):
         """
         arr = self.values
         if array is None:
-            raise StopIteration
+            if Cython.__version__ < '3.0.0':
+                raise StopIteration
+            else:
+                return
         for k, v in zip(arr.field('key'), arr.field('value')):
             yield (k.as_py(), v.as_py())

pitrou · 2023-07-20T08:02:04Z

There should not be any need to raise StopIteration.

jorisvandenbossche · 2023-07-20T08:04:17Z

python/pyarrow/ipc.pxi

    def __iter__(self):
        while True:
-            yield self.read_next_message()
+            try:
+                yield self.read_next_message()
+            except StopIteration:
+                # For Cython >= 3.0.0
+                return


I don't know if it is this code that gives problems to be written in a way that works for both cython 0.29 and 3, but a potential different way of writing it:

def __iter__(self): return self def __next__(self): return self.read_next_message()

Thanks! I'll try it.

pitrou · 2023-07-20T08:39:42Z

python/pyarrow/scalar.pxi

        arr = self.values
-        if array is None:
+        if arr is None:
            raise StopIteration
        for k, v in zip(arr.field('key'), arr.field('value')):
            yield (k.as_py(), v.as_py())


Do you actually test these changes before pushing them?

You can't use yield in a __next__ function.
Either you write __iter__ as a generator (using yield), or you write a pair of __iter__ and __next__ functions (without yield). You can't mix the two styles.

And FWIW, for this case, I would leave the __iter__ as is (in the other cases such as RecordBatchReader, we already have a public "next" method that can just be called from __next__)

Sorry. I didn't test on local before I pushed a commit because I don't have enough time yesterday.

I reverted the __iter__ + __next__ change.

lidavidm

Flight changes LGTM, thanks!

kou · 2023-07-20T23:37:59Z

@jorisvandenbossche Do you have any idea why the following failure was happen?

https://github.com/apache/arrow/actions/runs/5615711782/job/15216554355?pr=36745#step:6:6074

 =================================== FAILURES ===================================
_______________________ test_cloudpickle_dataset[False] ________________________

tempdir = PosixPath('/tmp/pytest-of-root/pytest-0/test_cloudpickle_dataset_False0')
datadir = PosixPath('/opt/conda/envs/arrow/lib/python3.8/site-packages/pyarrow/tests/data/parquet')
use_legacy_dataset = False

    @pytest.mark.pandas
    @parametrize_legacy_dataset
    def test_cloudpickle_dataset(tempdir, datadir, use_legacy_dataset):
        cp = pytest.importorskip('cloudpickle')
        dataset = _make_dataset_for_pickling(tempdir, use_legacy_dataset)
>       _assert_dataset_is_picklable(
            dataset, pickler=cp, use_legacy_dataset=use_legacy_dataset)

opt/conda/envs/arrow/lib/python3.8/site-packages/pyarrow/tests/parquet/test_dataset.py:1572: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
opt/conda/envs/arrow/lib/python3.8/site-packages/pyarrow/tests/parquet/test_dataset.py:1540: in _assert_dataset_is_picklable
    assert is_pickleable(dataset)
opt/conda/envs/arrow/lib/python3.8/site-packages/pyarrow/tests/parquet/test_dataset.py:1538: in is_pickleable
    return obj == pickler.loads(pickler.dumps(obj))
opt/conda/envs/arrow/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py:73: in dumps
    cp.dump(obj)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <cloudpickle.cloudpickle_fast.CloudPickler object at 0x7f25aa27aa10>
obj = <pyarrow.parquet.core._ParquetDatasetV2 object at 0x7f25a9ec9690>

    def dump(self, obj):
        try:
>           return Pickler.dump(self, obj)
E           _pickle.PicklingError: Can't pickle <cyfunction ParquetFragmentScanOptions._reconstruct at 0x7f2660177140>: it's not the same object as pyarrow._dataset_parquet.ParquetFragmentScanOptions._reconstruct

opt/conda/envs/arrow/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py:632: PicklingError

jorisvandenbossche · 2023-07-21T07:28:29Z

Not directly an idea. We use the same pattern in many other places. Can you try this patch:

--- a/python/pyarrow/_dataset_parquet.pyx
+++ b/python/pyarrow/_dataset_parquet.pyx
@@ -741,7 +741,7 @@ cdef class ParquetFragmentScanOptions(FragmentScanOptions):
             thrift_string_size_limit=self.thrift_string_size_limit,
             thrift_container_size_limit=self.thrift_container_size_limit,
         )
-        return type(self)._reconstruct, (kwargs,)
+        return ParquetFragmentScanOptions._reconstruct, (kwargs,)

(in theory it shouldn't matter for python, but maybe cython 3 has a bug that treats this different)

kou · 2023-07-21T07:33:55Z

Thanks! I've pushed your patch.

h-vetinari · 2023-07-21T08:19:31Z

If there are problems with compatibility between 0.29 and 3.0 (aside from the few intentional breaking changes), the Cython devs are usually quick to respond if you open an issue

danepitkin · 2023-08-10T14:41:06Z

The failing pickle test is only when using the cloudpickle module instead of the pickle5/pickle module. Perhaps they don't support cython3.0.0 yet? This is the only test in the entire codebase that tests cloudpickle.. not sure how important it is as a dependency to pyarrow.

@pytest.mark.pandas
@parametrize_legacy_dataset
def test_cloudpickle_dataset(tempdir, datadir, use_legacy_dataset):
    cp = pytest.importorskip('cloudpickle')
    dataset = _make_dataset_for_pickling(tempdir, use_legacy_dataset)
    _assert_dataset_is_picklable(
        dataset, pickler=cp, use_legacy_dataset=use_legacy_dataset)

pitrou · 2023-08-10T14:48:09Z

Well, first, this PR should be rebased, because there are conflicts.

danepitkin · 2023-08-10T15:00:53Z

I did rebase/squash to test things out here: #37097

I wanted to experiment without polluting this PR. I can merge any fixes I find back into this PR.

kou · 2023-08-11T08:07:27Z

Can we use #37097 instead of this?

danepitkin · 2023-08-11T18:57:40Z

Can we use #37097 instead of this?

Sure!

kou · 2023-08-11T20:46:57Z

I close this in favor of #37097.

kou marked this pull request as ready for review July 18, 2023 09:31

github-actions bot added Component: Python awaiting committer review Awaiting committer review labels Jul 18, 2023

apache deleted a comment from github-actions bot Jul 19, 2023

kou added 5 commits July 20, 2023 08:36

apacheGH-36730: [Python] Add support for Cython 3.0.0

6a6171a

Add missing const

ecad6bd

Suppress C4551

945ac7d

Follow generator change

978b47c

Unpin Cython

cb6f36d

kou force-pushed the python-cython-3 branch from 81493b1 to cb6f36d Compare July 19, 2023 23:39

kou requested review from assignUser and raulcd as code owners July 19, 2023 23:39

kou added 2 commits July 20, 2023 08:45

Add a missing :

7d9a1ed

Add missing NULL check

49b2e25

kou requested a review from lidavidm as a code owner July 20, 2023 07:38

github-actions bot added the Component: FlightRPC label Jul 20, 2023

jorisvandenbossche reviewed Jul 20, 2023

View reviewed changes

github-actions bot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Jul 20, 2023

pitrou reviewed Jul 20, 2023

View reviewed changes

lidavidm approved these changes Jul 20, 2023

View reviewed changes

github-actions bot added awaiting merge Awaiting merge and removed awaiting changes Awaiting changes labels Jul 20, 2023

Don't use __next__ for MapScalar

113f90c

github-actions bot added awaiting changes Awaiting changes and removed awaiting merge Awaiting merge labels Jul 20, 2023

Fix table format

001f562

github-actions bot added Component: Documentation awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Jul 20, 2023

Try

3e83b00

h-vetinari mentioned this pull request Jul 24, 2023

GH-35658: [Packaging] Sync conda recipes with feedstocks #35637

Merged

kou mentioned this pull request Jul 25, 2023

[Python] 12.0.1 FTBFS w/ Cython-3 on Fedora 39/rawhide #36857

Closed

kou mentioned this pull request Aug 7, 2023

GH-36642: [Python][CI] Configure warnings as errors during pytest #37018

Merged

danepitkin mentioned this pull request Aug 9, 2023

GH-36730: [Python] Add support for Cython 3.0.0 #37097

Merged

kou closed this Aug 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-36730: [Python] Add support for Cython 3.0.0 #36745

GH-36730: [Python] Add support for Cython 3.0.0 #36745

kou commented Jul 18, 2023 •

edited

Loading

github-actions bot commented Jul 18, 2023

pitrou commented Jul 18, 2023

kou commented Jul 18, 2023

jorisvandenbossche commented Jul 19, 2023 •

edited

Loading

jorisvandenbossche commented Jul 19, 2023

github-actions bot commented Jul 19, 2023

pitrou commented Jul 19, 2023

kou commented Jul 19, 2023

github-actions bot commented Jul 19, 2023

kou commented Jul 19, 2023

kou commented Jul 20, 2023

kou commented Jul 20, 2023

pitrou commented Jul 20, 2023

jorisvandenbossche Jul 20, 2023

kou Jul 20, 2023

pitrou Jul 20, 2023

jorisvandenbossche Jul 20, 2023

kou Jul 20, 2023

lidavidm left a comment

kou commented Jul 20, 2023

jorisvandenbossche commented Jul 21, 2023

kou commented Jul 21, 2023

h-vetinari commented Jul 21, 2023

danepitkin commented Aug 10, 2023 •

edited

Loading

pitrou commented Aug 10, 2023

danepitkin commented Aug 10, 2023 •

edited

Loading

kou commented Aug 11, 2023

danepitkin commented Aug 11, 2023

kou commented Aug 11, 2023

GH-36730: [Python] Add support for Cython 3.0.0 #36745

GH-36730: [Python] Add support for Cython 3.0.0 #36745

Conversation

kou commented Jul 18, 2023 • edited Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

github-actions bot commented Jul 18, 2023

pitrou commented Jul 18, 2023

kou commented Jul 18, 2023

jorisvandenbossche commented Jul 19, 2023 • edited Loading

jorisvandenbossche commented Jul 19, 2023

github-actions bot commented Jul 19, 2023

pitrou commented Jul 19, 2023

kou commented Jul 19, 2023

github-actions bot commented Jul 19, 2023

kou commented Jul 19, 2023

kou commented Jul 20, 2023

kou commented Jul 20, 2023

pitrou commented Jul 20, 2023

jorisvandenbossche Jul 20, 2023

Choose a reason for hiding this comment

kou Jul 20, 2023

Choose a reason for hiding this comment

pitrou Jul 20, 2023

Choose a reason for hiding this comment

jorisvandenbossche Jul 20, 2023

Choose a reason for hiding this comment

kou Jul 20, 2023

Choose a reason for hiding this comment

lidavidm left a comment

Choose a reason for hiding this comment

kou commented Jul 20, 2023

jorisvandenbossche commented Jul 21, 2023

kou commented Jul 21, 2023

h-vetinari commented Jul 21, 2023

danepitkin commented Aug 10, 2023 • edited Loading

pitrou commented Aug 10, 2023

danepitkin commented Aug 10, 2023 • edited Loading

kou commented Aug 11, 2023

danepitkin commented Aug 11, 2023

kou commented Aug 11, 2023

kou commented Jul 18, 2023 •

edited

Loading

jorisvandenbossche commented Jul 19, 2023 •

edited

Loading

danepitkin commented Aug 10, 2023 •

edited

Loading

danepitkin commented Aug 10, 2023 •

edited

Loading