[CI][Python] Cython 3.0 seems to make our build of pyarrow fail #36730

raulcd · 2023-07-17T16:39:35Z

Describe the bug, including details regarding any error messages, version, and platform.

As seen on some jobs for 13.0.0 on the maintenance branch:
https://github.com/ursacomputing/crossbow/actions/runs/5576946887/jobs/10192099702
and this other branch from main:
https://github.com/ursacomputing/crossbow/actions/runs/5577773946/jobs/10191016009

It seems that the new version of Cython being used (it was released ~1 hour ago at the moment of creation of the issue):

 Collecting cython>=0.29.31 (from -r /Users/runner/work/crossbow/crossbow/arrow/python/requirements-wheel-build.txt (line 1))
  Obtaining dependency information for cython>=0.29.31 from https://files.pythonhosted.org/packages/a7/9d/643b508fc18fe095a8c1efea5bd86054ac63974ecd566dba962de40ebaa6/Cython-3.0.0-cp39-cp39-macosx_10_9_x86_64.whl.metadata
  Downloading Cython-3.0.0-cp39-cp39-macosx_10_9_x86_64.whl.metadata (3.1 kB)

There is a build issue:

 [15/129] cd /Users/runner/work/crossbow/crossbow/arrow/python/build/temp.macosx-10.14-x86_64-cpython-39 && /Users/runner/work/crossbow/crossbow/build-env/bin/python -m cython --cplus --warning-errors --directive embedsignature=True --working /Users/runner/work/crossbow/crossbow/arrow/python --output-file /Users/runner/work/crossbow/crossbow/arrow/python/build/temp.macosx-10.14-x86_64-cpython-39/_flight.cpp /Users/runner/work/crossbow/crossbow/arrow/python/pyarrow/_flight.pyx
FAILED: CMakeFiles/_flight_pyx /Users/runner/work/crossbow/crossbow/arrow/python/build/temp.macosx-10.14-x86_64-cpython-39/CMakeFiles/_flight_pyx 
cd /Users/runner/work/crossbow/crossbow/arrow/python/build/temp.macosx-10.14-x86_64-cpython-39 && /Users/runner/work/crossbow/crossbow/build-env/bin/python -m cython --cplus --warning-errors --directive embedsignature=True --working /Users/runner/work/crossbow/crossbow/arrow/python --output-file /Users/runner/work/crossbow/crossbow/arrow/python/build/temp.macosx-10.14-x86_64-cpython-39/_flight.cpp /Users/runner/work/crossbow/crossbow/arrow/python/pyarrow/_flight.pyx

Error compiling Cython file:
------------------------------------------------------------
...

    cdef cppclass CFlightListing" arrow::flight::FlightListing":
        CResult[unique_ptr[CFlightInfo]] Next()

    cdef cppclass CSimpleFlightListing" arrow::flight::SimpleFlightListing":
        CSimpleFlightListing(vector[CFlightInfo]&& info)
                            ^
------------------------------------------------------------

pyarrow/includes/libarrow_flight.pxd:175:28: Rvalue-reference as function argument not supported

Error compiling Cython file:
------------------------------------------------------------
...
            # Text header values in gRPC (and HTTP/1, HTTP/2) are
            # required to be valid ASCII. Binary header values are
            # exposed as bytes.
            value = value.decode("ascii")
        headers.setdefault(header, []).append(value)
        postincrement(header_iter)
        ^
------------------------------------------------------------

pyarrow/_flight.pyx:2312:8: No 'operator++(int)' declared for postfix '++' (operand type is 'const_iterator')

Component(s)

Continuous Integration, Python

The text was updated successfully, but these errors were encountered:

…uild dependencies (#36743) ### Rationale for this change Although we already fixed some cython 3 build issues (#34726), some new have been introduced, which we are seeing now cython 3 is released (#36730) Adding an upper pin (<3) for the release, so we have more time (the full 14.0 release cycle) to iron out issues. * Closes: #36744 Authored-by: Joris Van den Bossche <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

…rrow build dependencies (apache#36743) ### Rationale for this change Although we already fixed some cython 3 build issues (apache#34726), some new have been introduced, which we are seeing now cython 3 is released (apache#36730) Adding an upper pin (<3) for the release, so we have more time (the full 14.0 release cycle) to iron out issues. * Closes: apache#36744 Authored-by: Joris Van den Bossche <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

danepitkin · 2023-09-15T17:14:27Z

FYI #37097 adds support for Cython 3 without enabling Cython 3 by default. Enablement will happen in a separate PR #37743. Both PRs are targeting 14.0 milestone.

### Rationale for this change Cython 3.0.0 is the latest release. PyArrow should work with Cython 3.0.0. **Cython 3 is not enabled in this diff.** ### What changes are included in this PR? * Don't use `vector[XXX]&&` * Add a declaration for `postincrement` * See also: https://cython.readthedocs.io/en/latest/src/userguide/migrating_to_cy30.html#c-postincrement-postdecrement-operator * Ignore `C4551` warning (function call missing argument list) with MSVC * See also: cython/cython#4445 * Add missing `const` to `CLocation`'s static methods. * Don't use `StopIteration` to stop generator * See also: https://cython.readthedocs.io/en/latest/src/userguide/migrating_to_cy30.html#python-3-syntax-semantics * non-extern `cdef` functions will now propagate python exceptions automatically unless explicitly labeled `noexcept` * Function binding in cython is now enabled by default. Class methods that are used as wrappers for pickling should be converted to staticmethods. * Numpydocs now validates more Cython 3 objects than Cython <3 * Enum types are now being validated, and some unhelpful validation checks on Enums are now ignored * Added a cython <3 nightly CI job Note: * Cython 3.0.0, 3.0.1, 3.0.2 has an issue when compiling with debug mode cython/cython#5552 ### Are these changes tested? Yes. ### Are there any user-facing changes? Yes. * Closes: #36730 Lead-authored-by: Dane Pitkin <[email protected]> Co-authored-by: Sutou Kouhei <[email protected]> Co-authored-by: Dane Pitkin <[email protected]> Signed-off-by: AlenkaF <[email protected]>

### Rationale for this change Cython 3.0.0 is the latest release. PyArrow should work with Cython 3.0.0. **Cython 3 is not enabled in this diff.** ### What changes are included in this PR? * Don't use `vector[XXX]&&` * Add a declaration for `postincrement` * See also: https://cython.readthedocs.io/en/latest/src/userguide/migrating_to_cy30.html#c-postincrement-postdecrement-operator * Ignore `C4551` warning (function call missing argument list) with MSVC * See also: cython/cython#4445 * Add missing `const` to `CLocation`'s static methods. * Don't use `StopIteration` to stop generator * See also: https://cython.readthedocs.io/en/latest/src/userguide/migrating_to_cy30.html#python-3-syntax-semantics * non-extern `cdef` functions will now propagate python exceptions automatically unless explicitly labeled `noexcept` * Function binding in cython is now enabled by default. Class methods that are used as wrappers for pickling should be converted to staticmethods. * Numpydocs now validates more Cython 3 objects than Cython <3 * Enum types are now being validated, and some unhelpful validation checks on Enums are now ignored * Added a cython <3 nightly CI job Note: * Cython 3.0.0, 3.0.1, 3.0.2 has an issue when compiling with debug mode cython/cython#5552 ### Are these changes tested? Yes. ### Are there any user-facing changes? Yes. * Closes: apache#36730 Lead-authored-by: Dane Pitkin <[email protected]> Co-authored-by: Sutou Kouhei <[email protected]> Co-authored-by: Dane Pitkin <[email protected]> Signed-off-by: AlenkaF <[email protected]>

raulcd added the Type: bug label Jul 17, 2023

github-actions bot added Component: Continuous Integration Component: Python labels Jul 17, 2023

raulcd added this to the 13.0.0 milestone Jul 17, 2023

raulcd added the Priority: Blocker Marks a blocker for the release label Jul 17, 2023

raulcd mentioned this issue Jul 17, 2023

WIP: Dummy PR to check maint-13.0.0 status #36616

Closed

This was referenced Jul 18, 2023

GH-36744: [Python][Packaging] Add upper pin for cython<3 to pyarrow build dependencies #36743

Merged

[Python][Packaging] Add upper pin for cython<3 to pyarrow build dependencies #36744

Closed

kou added a commit to kou/arrow that referenced this issue Jul 18, 2023

apacheGH-36730: [Python] Add support for Cython 3.0.0

685ecf7

github-actions bot mentioned this issue Jul 18, 2023

GH-36730: [Python] Add support for Cython 3.0.0 #36745

Closed

github-actions bot assigned kou Jul 18, 2023

raulcd modified the milestones: 13.0.0, 14.0.0 Jul 18, 2023

raulcd removed the Priority: Blocker Marks a blocker for the release label Jul 18, 2023

kou added a commit to kou/arrow that referenced this issue Jul 19, 2023

apacheGH-36730: [Python] Add support for Cython 3.0.0

6a6171a

raulcd mentioned this issue Jul 20, 2023

Issue in Pyarrow installation in Docker file #36785

Closed

kou mentioned this issue Jul 25, 2023

[Python] 12.0.1 FTBFS w/ Cython-3 on Fedora 39/rawhide #36857

Closed

danepitkin pushed a commit to danepitkin/arrow that referenced this issue Aug 9, 2023

apacheGH-36730: [Python] Add support for Cython 3.0.0

e8d03fe

danepitkin pushed a commit to danepitkin/arrow that referenced this issue Aug 9, 2023

apacheGH-36730: [Python] Add support for Cython 3.0.0

76f3d06

danepitkin pushed a commit to danepitkin/arrow that referenced this issue Aug 9, 2023

apacheGH-36730: [Python] Add support for Cython 3.0.0

b875cc9

danepitkin pushed a commit to danepitkin/arrow that referenced this issue Aug 9, 2023

apacheGH-36730: [Python] Add support for Cython 3.0.0

c9a25ea

github-actions bot mentioned this issue Aug 11, 2023

GH-36730: [Python] Add support for Cython 3.0.0 #37097

Merged

danepitkin pushed a commit to danepitkin/arrow that referenced this issue Aug 14, 2023

apacheGH-36730: [Python] Add support for Cython 3.0.0

dd5fc08

danepitkin pushed a commit to danepitkin/arrow that referenced this issue Aug 18, 2023

apacheGH-36730: [Python] Add support for Cython 3.0.0

e417313

h-vetinari mentioned this issue Aug 24, 2023

Unpin pytest conda-forge/arrow-cpp-feedstock#1145

Closed

danepitkin pushed a commit to danepitkin/arrow that referenced this issue Aug 28, 2023

apacheGH-36730: [Python] Add support for Cython 3.0.0

2a032e8

danepitkin pushed a commit to danepitkin/arrow that referenced this issue Aug 30, 2023

apacheGH-36730: [Python] Add support for Cython 3.0.0

2512174

danepitkin pushed a commit to danepitkin/arrow that referenced this issue Aug 31, 2023

apacheGH-36730: [Python] Add support for Cython 3.0.0

53fcf07

danepitkin pushed a commit to danepitkin/arrow that referenced this issue Sep 12, 2023

apacheGH-36730: [Python] Add support for Cython 3.0.0

ba77437

danepitkin pushed a commit to danepitkin/arrow that referenced this issue Sep 15, 2023

apacheGH-36730: [Python] Add support for Cython 3.0.0

93c61c7

danepitkin mentioned this issue Sep 15, 2023

[Python] Enable Cython 3 #37742

Closed

danepitkin assigned danepitkin and unassigned kou Sep 15, 2023

danepitkin added the Priority: Blocker Marks a blocker for the release label Sep 19, 2023

AlenkaF closed this as completed in #37097 Sep 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI][Python] Cython 3.0 seems to make our build of pyarrow fail #36730

[CI][Python] Cython 3.0 seems to make our build of pyarrow fail #36730

raulcd commented Jul 17, 2023

danepitkin commented Sep 15, 2023

[CI][Python] Cython 3.0 seems to make our build of pyarrow fail #36730

[CI][Python] Cython 3.0 seems to make our build of pyarrow fail #36730

Comments

raulcd commented Jul 17, 2023

Describe the bug, including details regarding any error messages, version, and platform.

Component(s)

danepitkin commented Sep 15, 2023