dev call 20221006

Dev Call: 6 October 2022

Present

Antonio, Stepan, Petr V., Florian, Tim, Matti, Ronan, Mohaned

Agenda

• Discussion item: Mixed mode binary verification step? https://github.com/hpyproject/hpy/issues/366

• Discussion item: Limited contexts for tp_traverse and tp_destroy

  Mark has proposed limited contexts for these slots: https://github.com/markshannon/New-C-API-for-Python/blob/main/DesignRules.md#all-functions-to-take-an-opaque-context

  It would help with the numpy port if we could have e.g. ctx->Mem_Free available to tp_destroy; or ctx->Field_Load, ctx->Global_Load, ctx->Type and ctx->Is available to tp_traverse

• Discussion item: Clarify encoding used for C strings in the API. #355 How much should we start to deviate from CPython towards the "ideal" API already now?

• Discussion item: 1.0 release? Should we aim for one since we'll have the ABI version check very soon?

  • PyPy 0.0.4 seems ready. Review before merge? Can we add it to CI?

• HPyBytes_AsString/AS_STRING: should we remove them and just use the buffer protocol for that?

• Multi-phase module init

• Module state vs HPyGlobal

Misc

• HPyBytes_AsString/AS_STRING: * Currently, pointer is valid as long as the handle. The buffer protocol could be * There is consens that we should keep since there is not much difference to the buffer protocol and it is easier to use. * top4000 uses it 630 times * CPython might want to remove it because the underlying storage might be changed * Mark will probably remove it from his new API proposal
• PyPy/HPy 0.0.4 changes are ready to merge; Matti asked for review * Ronan pointed out the the mixed-mode seems to be broken
• Encoding in C strings: let's just do that and make it explicit. Everyone agreed on that.

Release 1.0

• What do we want to achieve? Visibility?
• Antonio's concern: what if we change the API/ABI soon after the release? This should be covered
• Matti: let's add milestones (and add issues)
• Antonio doesn't feel convinced that HPy is already there. He suggested to have NumPy fully ported (tests passing) before we do that.
• The idea of having a 1.0 release is to bootstrap everything.
• We should probably go for something like 0.9 by the end of the year.
• There is consens about having milestones.
• Matti suggests to have a fixed release schedule regardless of how many changes are in the release.

Binary verification

• Tim, Florian suggest to do a verification when creating the stub loader
• Antonio suggests to also allow mixed-mode binaries. But there would probably not be much difference to a C API extension.
• Antonio thinks that we should do that at the compilation level.
• Python.h has a guard against multiple inclusions; we could just define that
• We can have both: the compilation guard and the binary verfication.

Multi-phase module init

• Tim asks: how is multi-phase module init related to subinterpreters?
• Petr: all modules should be sub-interpreter ready. The multi-phase module init basically asks for that. Current doc says: you shouldn't use multi-phase module init if not subinterpreter ready.
  • What does subinterpreter ready mean? Modules might be initialized multiple times. Simplest way: just have a global flag that already initialized.
• Stepan: it looks like that multi-phase module init and subinterpreters are still orthogonal. We could just have multi-phase module init and have a flags for subinterpreter-readiness
• Petr strongly recommends to use multi-phase module init from the beginning

Minimal context for tp_traverse, tp_destroy

• Antonio thinks it opens a can of worms
• Q: What is PyMem_Malloc/Free actually good for? A: This is for memory allocation tracking. Victor should know more about it. CPython's debug build is doing some memory tracking and padding which allows to detect use-after-free and out-of-bounds checking.
• Are those functions meant to be used by extensions authors? Petr would recommend using it because it since it allows easier debugging.
• Numpy also uses those functions in some places (we don't remember exactly where).
• Antonio would be fine to have some separate context like HPyDestroyContext.
• Apart memory management, PyPy will have problems to support anything like HPyField_Load in such a context.