Ability to include bare ffi methods in a #[pyclass]

[ariebovenberg]

As described in issues #3827 and #3843 there is some (unavoidable) overhead in calling PyO3 wrapped functions. An idea: the ability to fall back on lower level pyo3_ffi code for certain methods where you'd like to avoid this overhead.

Why is this important? Often, classes have small methods that are called a lot (e.g. magic methods). In my case, I'm porting Python classes with many such short methods.

related: this thread on discord

davidhewitt: Perhaps worth opening as a PyO3 issue? I could at least spitball some ideas on syntax and implementation and then maybe someone is interested in giving it a crack

[davidhewitt]

Sorry for being a bit slow to reply here. Overall, my take is that I can see why users would want this, and I'm open to supporting it if there's a good design and sufficient motivation. Though I think personally my priority would be to understand what the framework overheads are which affect users and work to eliminate those, so that reaching into unsafe Rust doesn't seems necessary.

So I guess that this could look something like this:

use pyo3::ffi;

#[pyclass]
struct Foo;

#[pymethods]
impl Foo {
    #[pyo3(raw_method = METH_VARARGS | METH_KWARGS)]  // specify the calling convention
    unsafe fn raw_method(args: *mut ffi::PyObject, kwargs: *mut ffi::PyObject) -> *mut ffi::PyObject {
        // insert code here
    }

    #[pyo3(raw_slot_method = tp_len)] // or specify a protocol slot
    unsafe fn tp_len(slf: *mut ffi::PyObject) -> ffi::Py_ssize_t {
        // insert code here
    }
}

There's a few things to consider here:

• We might want to make PyO3's argument extraction code public API too, as it's pretty useful. Though maybe a good reason to skip the framework would be to use bespoke extraction when you don't care about things like fancy error handling.
• While GIL refs are still around, setting up the GILPool is a necessary soundness mechanism, so skipping the framework requires that you'd need to type out this boilerplate.
• The other main PyO3 framework step is to update our "ReferencePool" which is used for PyObject drop/clone without the GIL held. There are perhaps ways that we can make this cheaper (e.g. maybe this problem goes away when Python 3.13 freethreaded Python lands)