Introduce capsule API

[fangerer]

This introduces an HPyCapsule API similar to PyCapsule.
While the capsule API isn't use a lot in the top4000 packages, it's needed for our NumPy migration efforts.

Main goals to achieve were:

• HPyCapsule should be interchangeable with PyCapsule. So, one can create an HPyCapsule, store into some attribute and a completely unrelated C API extension could use it via the PyCapsule API (and vice versa).
• Map HPyCapsule_* functions to PyCapsule_* functions in CPython ABI mode.
• Use just one context function to get and one to set all fields (to keep number of context functions low).

The implementation for CPython is mostly straight forward except of the destructor. This was the main problem is should probably be the main discussion point here.
The capsule destructor function has signature void (*PyCapsule_Destructor)(PyObject *) and is also exposed to the user with PyCapsule_Get/SetDestrutor.
I've introduced signature void (*HPyCapsule_Destructor)(const char *name, void *pointer, void *context) that will receive the contents of the capsule but not the object itself (and no HPy context, ofc).
However, we now cannot directly map the HPy destructor to a CPython one, we need a trampoline. This one needs to know the function pointer of the HPyCapsule_Destructor but PyCapsule does not give you the possibility to store this pointer somewhere. I've considered:

• subclassing and using a different C struct for the capsule object but the type is not base type.
• over-allocating the name field and storing the pointer before the name but the name is optional.

The only reliable solution I could think of was to have a global registry somewhere. I decided to put it into the interpreter state's dict (Python >= 3.8) or into the sys module (using PySys_Get/SetObject for Python < 3.8).

[mattip]

It seems this is required for the graal-team/hpy branch of numpy. Were there any more thoughts about how to do global state for the desctuctor?

[mattip]

Not connected to this PR, but I also see h_ComplexType in the numpy branch. Is there a PR or issue to add that to the HPyContext?

[mattip]

... and HPyTupleBuilder_IsNull as well...

[fangerer]

It seems this is required for the graal-team/hpy branch of numpy.

Yes.

Were there any more thoughts about how to do global state for the desctuctor?

Not sure if I understand your question.
In this PR, the problem was that I wanted HPyCapsule to be compatible with PyCapsule. The main problem is PyCapsule's destructor that receives the full capsule object (i.e. as PyObject *). We clearly cannot do that in HPy. So, we install a trampoline as PyCapsule destructor function that calls the HPy destructor function. Now, the tricky part is to store the pointer of the HPy destructor function somewhere for the trampoline.
The solution in this PR is to do that in the interpreter state's dict.

Anyway, I didn't think about other solutions since then. Ideas are welcome.

[mattip]

Adding this to the 0.9 milestone since it is needed for the NumPy port.

[mattip]

Needs a rebase

[fangerer]

sure, I'll do that

[mattip]

@fangerer waiting for a rebase to merge this

[fangerer]

@fangerer waiting for a rebase to merge this

Yes, still on my list. I'm sorry for the delay. I got sick (I'm still on leave) but will return tomorrow.

[mattip]

C++ is not happy with the capsule code:

/home/runner/work/hpy/hpy/hpy/devel/src/runtime/ctx_capsule.c:105:1: error: jump to label ‘error’
[214](https://github.com/hpyproject/hpy/actions/runs/3497785299/jobs/5857316290#step:7:215)
  105 | error:
[215](https://github.com/hpyproject/hpy/actions/runs/3497785299/jobs/5857316290#step:7:216)
      | ^~~~~

[mattip]

LGTM. I will merge once CI passes, if no-one objects.

[fangerer]

🥳 thank you, Matti

[antocuni]

(Sorry to chime in just few hours after the PR has been merged 🤦‍♂️ )

The only reliable solution I could think of was to have a global registry somewhere.

I might be missing something here, but I think that we could use the same technique that we used for HPyDef_METH:

hpy/hpy/devel/include/hpy/hpydef.h

Lines 21 to 27 in 414be8b

	typedef struct {
	const char *name; // The name of the built-in function/method
	HPyCFunction impl; // Function pointer to the implementation
	cpy_PyCFunction cpy_trampoline; // Used by CPython to call impl
	HPyFunc_Signature signature; // Indicates impl's expected the signature
	const char *doc; // The __doc__ attribute, or NULL
	} HPyMeth;

i.e., instead of passing a pointer to a function, we could pass a pointer to a struct which contains two function pointers (one to the user-provided code, and one to the autogenerated cpy trampoline). Something like this:

typedef void (*HPyFunc_capsule_dtor)(const char *name, void *pointer, void *context);

typedef struct {
    cpy_PyCapsule_Destructor cpy_trampoline;
    HPyFunc_capsule_dtor impl;
} HPyCapsule_Destructor;

Then the user will have to use a macro to generate the struct:

/* this automatically generate the trampoline and fills a statically allocated 
    struct called 'my_dtor' */
HPyCapsule_DESTRUCTOR(my_dtor, my_dtor_impl);
void my_dtor_impl(const char *name, void *pointer, void *context) {
    ...
}

void init() {
    ...
    HPyCapsule c = HPyCapsule_New(ctx, "foo.bar", myptr, &my_dtor);
}

[mattip]

It's not too late, this could be an add-on PR to enhance the current code. Would you want to make a PR?

[fangerer]

No, it's not too late. That's a good suggestion. Sorry, the PR is so old that I didn't spent too much thoughts on it recently.

[antocuni]

ok, I've opened an issue for it to ensure that we don't forget