Custom controller fails in cibuildwheel on macOS Python 3.8

[alugowski]

I'm building wheels that include a threadpoolctl custom controller. Everything works great, except for one highly specific scenario: the wheel test step of cibuildwheel on macOS Python 3.8.

The same tests pass on other Python versions, other OSes, in a regular GitHub Action (not cibuildwheel), or on my macOS laptop with Python 3.8. Same error with cibuildwheel from pip or via their pypa/[email protected] action step.

A workaround is to xfail that one scenario:

if sys.platform == "darwin" and sys.version_info.minor == 8 and os.environ.get("CIBUILDWHEEL", 0):
    pytest.xfail("threadpoolctl fails inside cibuildwheel macOS Python 3.8")

[cheetooooo]

بتاريخ ١٦‏/٠٩‏/٢٠٢٣ ٨:٥٢ ص، كتب Adam Lugowski ***@***.***>: I'm building wheels that include a threadpoolctl custom controller. Everything works great, except for one highly specific scenario: the wheel test step of cibuildwheel on macOS Python 3.8 (both x86 and arm). The same tests pass on other Python versions, or in a regular GitHub Action (not cibuildwheel), or on my macOS laptop with Python 3.8. I'm running on GitHub Actions, same error with cibuildwheel from pip or via their ***@***.*** action step. A workaround is to xfail that one scenario: if sys.platform == "darwin" and sys.version_info.minor == 8 and os.environ.get("CIBUILDWHEEL", 0): pytest.xfail("threadpoolctl fails inside cibuildwheel macOS Python 3.8") —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[jeremiedbb]

Thanks for the report @alugowski, could you provide more details about the failure ? a traceback ? It would help us a lot to figure out what's going on.

[alugowski]

I wish I could offer more than "Controller doesn't work in that context". No stack traces, just that the controller doesn't call the set_num_threads method as one would expect.

If I had time to debug it I'd start with verifying that cibuildwheel's relocation/repair doesn't have some weird interaction with threadpoolctl's library detection. It's a long shot since literally all other OSes/platforms/Python versions work, but cibuildhweel does do some location juggling (for good reason). I can now also verify that Conda's macOS Python 3.8 build works too.

I might be one of the early ones using this functionality in production, so let me at least offer what my code does. Maybe there's something obviously wrong.

My controller is simple. It sets a global variable that is read by the parallel methods:

    class FMMThreadPoolCtlController(threadpoolctl.LibController):
        user_api = "fast_matrix_market"
        internal_api = "fast_matrix_market"

        filename_prefixes = ("_fmm_core",)

        # noinspection PyMethodMayBeStatic
        def get_num_threads(self):
            global PARALLELISM
            return PARALLELISM

        # noinspection PyMethodMayBeStatic
        def set_num_threads(self, num_threads):
            global PARALLELISM
            PARALLELISM = num_threads

        # noinspection PyMethodMayBeStatic
        def get_version(self):
            return __version__

        def set_additional_attributes(self):
            pass

The test is correspondingly simple too:

    def test_threadpoolctl(self):
        import os
        import sys
        import pytest
        if sys.platform == "darwin" and sys.version_info.minor == 8 and os.environ.get("CIBUILDWHEEL", 0):
            pytest.xfail("threadpoolctl fails inside cibuildwheel macOS Python 3.8")
            # see https://github.com/joblib/threadpoolctl/issues/150
            return

        with threadpoolctl.threadpool_limits(limits=2, user_api='fast_matrix_market'):
            self.assertEqual(fmm.PARALLELISM, 2)
        with threadpoolctl.threadpool_limits(limits=4):
            self.assertEqual(fmm.PARALLELISM, 4)

The cibuildwheel action is fairly standard as well.

This is one Actions run that experiences the test fail. Obviously it's from before the xfail was added.

Note: basically this code is now part of SciPy main, but causes no issues there because they only support Python 3.9 and up.

[ogrisel]

For the record, here is the traceback:

  ________________________ TestModule.test_threadpoolctl _________________________
  
  self = <test_basic.TestModule testMethod=test_threadpoolctl>
  
      @unittest.skipIf(not threadpoolctl or not hasattr(threadpoolctl, "register"),
                       reason="no threadpoolctl or version too old")
      def test_threadpoolctl(self):
          with threadpoolctl.threadpool_limits(limits=2, user_api='fast_matrix_market'):
  >           self.assertEqual(fmm.PARALLELISM, 2)
  E           AssertionError: 0 != 2
  
  /Users/runner/work/fast_matrix_market/fast_matrix_market/python/tests/test_basic.py:26: AssertionError
  =============================== warnings summary ===============================
  test_basic.py::TestModule::test_threadpoolctl
    /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-h48wa04i/cp38-macosx_x86_64/venv-test/lib/python3.8/site-packages/threadpoolctl.py:1019: RuntimeWarning: libc not found. The ctypes module in Python 3.8 is maybe too old for this OS.
      warnings.warn(
  
  -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
  =========================== short test summary info ============================
  FAILED ../../../../../../../../../Users/runner/work/fast_matrix_market/fast_matrix_market/python/tests/test_basic.py::TestModule::test_threadpoolctl - AssertionError: 0 != 2

I think the problem is that ctypes cannot load the libc on this OS:

threadpoolctl.py:1019: RuntimeWarning: libc not found. The ctypes module in Python 3.8 is maybe too old for this OS

I think xfailing and not supporting Python 3.8 is fine if it works with more recent versions of Python.

I don't think there is anything to do at the threadpoolctl level.

[alugowski]

Ah, I'll try to summarize:

threadpoolctl uses methods from libc to get the list of loaded libraries that are then matched against registered controllers. If there is no compatible libc then no controllers.

For whatever reason _get_libc() does not find a libc on the Python 3.8 that cibuildwheel uses for macOS.