dev call 20230112

Dev Call: 12 January 2023

Present

Stepan, Florian, Mohaned, Ronan, Simon, Tim, Antonio, Du Toit

Agenda

• Next release 0.9: I think the major parts are mostly ready. Some important PRs still need to be reviewed and merged.
  • Multi-phase module init
  • Use fixed-width integers and clarify encoding of C strings
  • Support vectorcall protocol
  • Debug mode: Check correct context usage
  • Exact type checks with HPy_TypeCheck
  • Module-state support
• Merged PRs:
  • Fix microbenchmarks
  • Build static libraries of HPy devel sources for testing
  • HPy ABI version verification
  • Fix GitHub actions
  • Split valgrind tests
• Status updates

Multi-phase module init

• This is the way to go forward and we will drop single-phase init.
• Antonio is just slightly concerned about the more complicated migration path since most C API extensions still use single-phase init.
• We already have this problem with types since one needs to convert from static types to heap types.
• Migration might not be too hard since most of the PyInit_* function just goes into the HPy_mod_exec slot.
• Major difference: it is no longer possible to directly create a module instance since HPyModule_Create was removed and the runtime will create the module instance. As a consequence, the HPy_mod_create slot may also just create other objects like SimpleNamespace but there is anyhow not many use cases for HPy_mod_create.
• People are in favor of merging this PR. It may be possible to provide a few more helper macros to make migration easier.

Module state

• Stepan investigated and discussed (with CPython core devs) one flaw in the current CPython API: It is possible to create multiple instances of one module in the same interpreter with multi-phase init. In this case, using module state may be problematic when used in slots. This problem may be hit if one inherits from a type of a module with multiple instances. The module instance needs to be looked up with the corresponding PyModuleDef but and this will point to the old instance. So, it is possible to operate on the wrong module state.
• See Issue 286
• Antonio proposed: change convention such that you always get a module state in slots as well; then introduce a global flag that you don't care about this such that we can be fast on CPython.
• Antonio strongly asks for getting the API right even if we cannot implement it correctly or efficiently on CPython.
• Simon suggested to create a micro benchmark and measure the expensive lookup on CPython

Fixed-width integer in ABI

• After some discussion, everyone seems to be convinced that it is useful to have fixed-width integers in the ABI.
• The open PR still provides inline helper functions like HPyLong_FromLong that will decide at HPy extension compilation time what ABI function to use. Hence, the API doesn't change much but gets additional functions for HPyLong_FromInt32_t and such.
• We also discussed on how to deal with size_t and HPy_ssize_t. In particular HPy_ssize_t is a bit more problematic because it is also frequently used in function signatures for slots or varargs etc. Since HPy mainly targets for 64-bit systems, we might just use int64_t instead of HPy_ssize_t in those signatures. This may be a slight performance drawback on 32-bit systems but we don't even test on those.
• Antonio also suggested to replace HPy_ssize_t by ssize_t since that should be available on all C99 (and newer) compilers.

Misc

• Florian will kick out Issue 181 and Issue 214 since we preserve those for internship projects. They will mainly just add API and thus shouldn't be a problem to do after the next release.
• Florian also asked for a review of PR Support vectorcall protocol. Even if the vectorcall function signature is already very similar to HPy's varargs function signature, it gives one major advantage: you can have different functions per object.