dev call 20220303

Dev Call: 3 March 2022

Present

Antonio, Simon, Ronan, Matti, Tim, Mohaned, Stepan, Florian

Agenda

• HPyGlobal vs Module State: both or just one

  • HPyGlobal POC: https://github.com/hpyproject/hpy/pull/299
  • Module state PR: TBD soon
  • global state in MPL is needed, for example, in some slots where the module object is not easily accessible
  • related: new "get current context" approach described in https://github.com/hpyproject/hpy/issues/268#issuecomment-1042824498

• NumPy Progress and Plans

  • Metaclasses support for heap types in CPython.
  • Extend other builtin types with pure HPy types (e.g. dtype meta inherits PyType_Type, numpy scalars inherit Python builtins) for pure HPy universal mode.
  • Create binary that can run the e.g. all of the code in a [NumPy Game of Life](https://github.com/rougier/numpy-tutorial/blob/master/scripts/game-of-life-numpy.py)

• HPyErr_FromFormat and friends: https://github.com/hpyproject/hpy/issues/274

  • Not super-important but if we want the MPL port to look polished and finalized, we'll need to deal with those.

• Misc

  • Make final decision on "Should HPy_Close silently ignore HPy_NULL" https://github.com/hpyproject/hpy/issues/297
  • CPython 3.10 compatibility https://github.com/hpyproject/hpy/issues/292
  • Should we take over and finish the PEP 518 support PR https://github.com/hpyproject/hpy/pull/287
  • Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS: https://github.com/hpyproject/hpy/pull/276
  • Provide a HPyModuleFinder and HPyModuleLoader for some QoL for people trying out our HPy ports

• Next release?

  • matplotlib-hpy port HPy extensions and publication of the port?

HPyGlobal

• Possible implementation: every global is basically just a per-interpreter token, e.g., an index into an array. Each interpreter would have its own instance of this array.
• Allocation needs to be done per process (across all interpreters)
• Possible "fast" mode: does not allow sub-interpreters and multiple interpreters in the same process but directly store the object's pointer in the global.
• We would have some context functions to use the global
• Open issue: how to do allocation of globals (e.g. generating a unique array index per process) if multiple interpreters are present. In this case, we would probably need to make it part of the HPy ABI (e.g. a counter variable).
• Each interpreter then needs to initialize every global during module initialization.

Numpy Progress

• Stepan started to migrate dtype
• Main problem: HPy does not support meta types and CPython does also not support it for heap types.
• There is a CPython issue for meta type support for heap types (10y old). It was recently brought up again.
• We could ask if Victor could help us with this. Chances should be good since we now have concrete use cases.
• Possible alternative solution: HPyType_FromSpec could also create a static type (but it will leak the struct)
• For upstream changes, we would need backport to 3.8
• Further problem: pure inheriting from built-in types * We need to align the artificial struct that contains PyObject/the builtin struct + the HPy struct, this is now done using "union hack", but that is not going to scale for all builtins(?), alternative: there is some POSIX API for getting memory alignment

Misc

• Should HPy_Close silently ignore HPy_NULL ? To make a decision: we should change all Py_DECREF to Py_XDECREF and see what the impact is.
• Kiwi, MPL perf was something like 25% worse than with C API. We need to investigate that anyway.
• HPyContext.h_builtins: don't remove HPy_Repr and co; Florian will create an appropriate GH issue for that
• Create issue for adding HPy_Eval
• CPython 3.10 compatibility: we should address that. Moving most HPy flags to HPyType_Extra_t sounds good. But we still need a way to identify HPy types. We could ask CPython to reserve a type flag for us. An alternative would be to use e.g. a special tp_dealloc function.
• HPyErr_FromFormat: continue discussion on the issue.