The macOS-14 failure is just a flaky test. Nothing to do with this PR.

Sorry for the probably noob questions, but here goes:

1. How many additional .whl files does this produce? For what platforms?
2. How many additional test runs / jobs are being executed?
3. Are additional unit tests run specifically for free-thread python? Or this just produces the .whl file without running tests?

How many additional .whl files does this produce? For what platforms?

This generates 1 additional wheel per axis in the build matrix so 7 additional wheels (2 for macOS, 2 for Windows, 3 for Linux). It only needed tweaking cibuildwheel configuration in order to build & tests those wheels.

How many additional test runs / jobs are being executed?
Are additional unit tests run specifically for free-thread python? Or this just produces the .whl file without running tests?

The existing set of tests is ran for those new wheels, except on Windows where the tests are not run pending pywin32 availability for free-threading python (see the comment in pyproject.toml). There are no new specific tests.

Happy to help out with adding support for free-threaded Python. I've been actively working on adding support in many projects. Also see https://py-free-threading.github.io with lots more resources and suggestions for porting code.

That said, because psutil uses ctypes, a lot of the information in there specific to libraries that use C extensions probably doesn't apply. I don't know offhand what the thread safety guarantees of ctypes are.

We're also hopeful that we'll be able to define a new "abi4" or similar mechanism that will support the free-threaded build and allow generating a single wheel that supports e.g. Python 3.8 or newer with both the free-threaded and "normal" ABI. That will hopefully arrive in time for either Python 3.14 or 3.15.

See python/cpython#127945. It looks there are lots of thread safety issues in ctypes under free-threading and this is being actively worked on upstream.

The tricky aspect of this is that the "normal" way people get warnings about the lack of thread safety in C code wrapped by Python is they'll see a RuntimeWarning that the GIL is re-enabled at runtime when Python imports a module with Py_MOD_GIL set to Py_MOD_GIL_USED (the default for C extensions).

psutil totally bypasses this mechanism, because it's talking to C code using ctypes:

Python 3.13.1 experimental free-threading build (main, Dec 10 2024, 14:07:41) [Clang 16.0.0 (clang-1600.0.26.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> # no warnings!

This is a little unfortunate.

ping @ZeroIntensity since it looks like you're working on the ctypes code in CPython. Do you have any idea what psutil should do to support free-threaded Python 3.13? Report thread safety bugs upstream? Or assume ctypes in 3.13 is unsafe and add lots of threading.Lock use?

It would be great if they reported bugs upstream, but right now 3.13 ctypes will stay thread-unsafe, because the changes are too complex to backport. You'll have to wait until 3.14 before it becomes properly thread safe. threading.Lock (or probably threading.RLock, because finalizers can cause problems) should be a fine workaround for now.

It might be worth marking ctypes as requiring the GIL for 3.13, so it gets re-enabled at runtime. Would that be preferable compared to adding locks everywhere?

It might be worth marking ctypes as requiring the GIL for 3.13, so it gets re-enabled at runtime. Would that be preferable compared to adding locks everywhere?

Unfortunately the ship might have sailed on that one. Many projects that are already shipping free-threaded wheels for 3.13 implicitly import ctypes, so making that change would cause those modules to also re-enable the GIL all of the sudden in a bugfix release of Python.

Hi

That said, because psutil uses ctypes, a lot of the information in there specific to libraries that use C extensions probably doesn't apply.

psutil does not use ctypes. It was only used in one place on Linux for Python 2.7, but it was removed after I dropped support for Python 2.7. The rest of the code is a mix of Python and C extension modules, which use Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros around those syscalls which are supposed to be blocking (e.g. psutil.disk_usage()). I assume (but I'm not sure) that in a free-threaded world these macros are no-ops?

We're also hopeful that we'll be able to define a new "abi4" or similar mechanism that will support the free-threaded build and allow generating a single wheel that supports e.g. Python 3.8 or newer with both the free-threaded and "normal" ABI. That will hopefully arrive in time for either Python 3.14 or 3.15.

Considering what @mayeut stated above, I'm afraid it's better to wait for abi4. Despite numerous efforts over the years (e.g. the retry_on_failure decorator or the tolerances), I've never managed to make the psutil test suite matrix really "stable". The OS often interferes with the unit tests by spawing new PIDs behind your back, or increasing the used memory for its own purposes, opening connections, etc. Things you just can't control. To get an idea, see how many red crosses we have. Adding 7 additional test runs to the matrix would mean getting even more failures, which is a problem especially when I have to make a new release. E.g. I never added a CI job for PYPY for this reason.

I assume (but I'm not sure) that in a free-threaded world these macros are no-ops?

No. In fact, they're quite important for free-threading. The critical section API is heavily reliant on Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS to temporarily release the thread's active locks, thus preventing re-entrancy and lock-ordering deadlocks.

psutil does not use ctypes

Oops! Clearly I need to look closer at the code. I'll try to understand this more and get back to you with more ideas about the CI issues you raised.

Question: what happens now when you pip install psutil with a free-thread cPython? Does it automatically pick up the .tar.gz and compile/install from sources?

Also, do tests pass (make test)?

Question: what happens now when you pip install psutil with a free-thread cPython? Does it automatically pick up the .tar.gz and compile/install from sources?

Yup! This is on my Mac with developer tools installed:

goldbaum at Mac in ~/Documents/bcrypt on support-free-threading
± python -VV
Python 3.13.1 experimental free-threading build (main, Dec 10 2024, 14:07:41) [Clang 16.0.0 (clang-1600.0.26.4)]

goldbaum at Mac in ~/Documents/bcrypt on support-free-threading
± pip install psutil
Collecting psutil
  Downloading psutil-6.1.1.tar.gz (508 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: psutil
  Building wheel for psutil (pyproject.toml) ... done
  Created wheel for psutil: filename=psutil-6.1.1-cp313-cp313t-macosx_14_0_arm64.whl size=250301 sha256=0fdab9283163c66473262788bfa53c161024602077584677a0877cc749f93b65
  Stored in directory: /Users/goldbaum/Library/Caches/pip/wheels/ff/fe/eb/59cac25690b1a9600e50b007a414ddabb88c04e3ca5df008d9
Successfully built psutil
Installing collected packages: psutil
Successfully installed psutil-6.1.1

Also, do tests pass (make test)?

I see four test failures on the same Mac:

FAILED psutil/tests/test_posix.py::TestSystemAPIs::test_disk_usage - assert 76481990656 < 4194304
FAILED psutil/tests/test_posix.py::TestSystemAPIs::test_users_started - AssertionError: assert 'Jan 06 13:47' == 'Jan 10 12:50'
FAILED psutil/tests/test_process_all.py::TestFetchAllProcesses::test_all - Failed: DID NOT RAISE <class 'psutil.NoSuchProcess'>
FAILED psutil/tests/test_scripts.py::TestInternalScripts::test_import_all - ModuleNotFoundError: No module named 'pypinfo'

Along with lots of warnings like:

 psutil-debug [psutil/arch/osx/proc.c:258]> task_for_pid() failed (pid=98270, err=5, errno=0, msg='(os/kern) failure'); setting AccessDenied()

Ah and now I understand why I don't see RuntimeWarnings - it looks like Sam Gross already did a pass over psutil back in June and declared that all the C extensions don't use the GIL! Sorry for missing this yesterday.

@ZeroIntensity

I assume (but I'm not sure) that in a free-threaded world these macros are no-ops?

No. In fact, they're quite important for free-threading. The critical section API is heavily reliant on Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS to temporarily release the thread's active locks, thus preventing re-entrancy and lock-ordering deadlocks.

Does it mean we have to replace this:

    Py_BEGIN_ALLOW_THREADS
    blocking_syscall();
    Py_END_ALLOW_THREADS

...with this?

    Py_BEGIN_CRITICAL_SECTION(self);
    Py_BEGIN_ALLOW_THREADS
    blocking_syscall();
    Py_END_ALLOW_THREADS
    Py_END_CRITICAL_SECTION();

No, you only need to lock self if you truly need it to be locked. Py_BEGIN_ALLOW_THREADS will just prevent deadlocks for any prior locked critical sections.

For example, if you locked self, then called a function that attempts to also lock self, then you won't end up with a deadlock because waiting on the mutex will also implicitly release the critical section. When Py_END_ALLOW_THREADS is reached, the previously locked critical sections are then restored.

Instead of thinking about when it's legal to use the C API in terms of acquiring and releasing the GIL, I find it better to think about attaching to and detaching from the runtime. It's only legal to call into the C API if a thread is attached to the runtime, but Python is only able to do global synchronization events like running the garbage collector when all threads are detached. So just like how on the GIL-enabled build you call Py_BEGIN_ALLOW_THREADS to prevent hangs and deadlocks, you still do it on the free-threaded build for similar reasons.

The main difference is it's no longer necessary to call Py_BEGIN_ALLOW_THREADS to allow other threads to execute Python code - many threads can be simultaneously attached to the runtime on the free-threaded build.

If pip install psutil automatically fetches the .tar.gz, installation (on Linux) just requires installing Python header files first (apt install python3-dev on Debian/Ubuntu). setup.py already suggests the exact command to run if headers are not installed. The real annoyance is on Windows, which requires Visual Studio. Still, considering that free-thread cPython is still experimental, I think just offering the tarball is good enough for now.

Let's provide wheels later when we have abi4.