[Python] Error compiling Cython files on Windows during release verification

[amoeba]

Describe the bug, including details regarding any error messages, version, and platform.

I ran into this while doing release verification for 15.0.1 RC0. The code in the error message is relatively new (PyCapsule) but old enough that it may just be my setup.

On Windows 11 in cmd.exe prompt with Anaconda set up, I ran

dev\release\verify-release-candidate.bat 15.0.1 0

C++ compiled fine and then when it got to building PyArrow, I see:

-- Running cmake --build for PyArrow
cmake --build . --config release
Microsoft (R) Build Engine version 16.11.2+f32259642 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

  1>Checking Build System
  Compiling Cython CXX source for _acero...
  performance hint: pyarrow\_acero.pyx:511:67: Exception check after calling 'unwrap' will always require the GIL to be
   acquired. Declare 'unwrap' as 'noexcept' if you control the definition and you're sure you don't want the function t
  o raise exceptions.
  performance hint: pyarrow\_acero.pyx:511:36: Exception check after calling 'GetResultValue[shared_ptr[CTable]]' will
  always require the GIL to be acquired. Declare 'GetResultValue[shared_ptr[CTable]]' as 'noexcept' if you control the
  definition and you're sure you don't want the function to raise exceptions.
  Building Custom Rule C:/tmp/arrow-verify-release/apache-arrow-15.0.1/python/CMakeLists.txt
  Compiling Cython CXX source for lib...

  Error compiling Cython file:
  ------------------------------------------------------------
  ...
      if schema.release != NULL:
          schema.release(schema)

      free(schema)

  cdef object alloc_c_schema(ArrowSchema** c_schema) noexcept:
                            ^
  ------------------------------------------------------------

  pyarrow\types.pxi:5264:26: noexcept clause is ignored for function returning Python object

  Error compiling Cython file:
  ------------------------------------------------------------
  ...
      if array.release != NULL:
          array.release(array)

      free(array)

  cdef object alloc_c_array(ArrowArray** c_array) noexcept:
                           ^
  ------------------------------------------------------------

  pyarrow\types.pxi:5283:25: noexcept clause is ignored for function returning Python object

  Error compiling Cython file:
  ------------------------------------------------------------
  ...
      if stream.release != NULL:
          stream.release(stream)

      free(stream)

  cdef object alloc_c_stream(ArrowArrayStream** c_stream) noexcept:
                            ^
  ------------------------------------------------------------

  pyarrow\types.pxi:5302:26: noexcept clause is ignored for function returning Python object
C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(24
1,5): error MSB8066: Custom build for 'C:\tmp\arrow-verify-release\apache-arrow-15.0.1\python\build\temp.win-amd64-cpyt
hon-38\CMakeFiles\e1d0ace4d7a55ee7d9220e8a19e03e2f\lib_pyx.rule;C:\tmp\arrow-verify-release\apache-arrow-15.0.1\python\
CMakeLists.txt' exited with code 1. [C:\tmp\arrow-verify-release\apache-arrow-15.0.1\python\build\temp.win-amd64-cpytho
n-38\lib_pyx.vcxproj]

I noted that the release verification script pins Python 3.8 so right now I'm re-running verification with Python 3.11 in the errors are related to that.

Component(s)

Python, Release

[amoeba]

Using Python 3.11 instead didn't magically fix it.

[bkietz]

It looks like this warning was added to cython recently; seems to be present in cython 3.0.9. Since the warning indicates noexcept never did anything, I think it's reasonable to just remove the keyword

[felipecrv]

I just merged #40387 which fix the duplicate issue #40386. This one can be closed as well.

[bkietz]

Fixed by #40387

[mkumatag]

any workaround for this one?

[mkumatag]

and also it will be great if we can back port to older branch and cut a release to unblock folks

[jorisvandenbossche]

@mkumatag For now, you will have to ensure you are using cython<3.0.9 while installing pyarrow from source.
How are you running into this? Through a pip install pyarrow? (I would have to look up how you can pass constraints on build dependencies to pip)

[jorisvandenbossche]

Googling a bit, I think you should be able to specify "constraints" to pip:

# constraints.txt
cython<3.0.9

$ PIP_CONSTRAINT=constraints.txt pip install pyarrow

The other option is to disable the build-isolation, and ensure you have installed the build dependencies up front:

pip install cython<3.0.9 numpy setuptools_scm setuptools wheel
pip install pyarrow --no-build-isolation

[mkumatag]

The other option is to disable the build-isolation, and ensure you have installed the build dependencies up front:

My case is little different, I'm using the micropipenv tool to install the packages form the Pipfile.lock got this entry as an indirect dependency for this project. I tried the pinning the cython in the Pipfile itself but couldn't control this version pinning. Not sure what else I need to look into..

[jorisvandenbossche]

The PIP_CONSTRAINT approach doesn't work? ( I would assume pipenv is still using pip to actually install the packages, so I would hope it picks up that env variable)

[mkumatag]

The PIP_CONSTRAINT approach doesn't work? ( I would assume pipenv is still using pip to actually install the packages, so I would hope it picks up that env variable)

yes, this solution worked for me :) thanks.

PIP_CONSTRAINT=constraints.txt micropipenv install

[jorisvandenbossche]

FYI, 15.0.2 was released that includes the fix for this. So the latest pyarrow should again build fine with latest cython