c api next level manifesto

Taking the C API to the Next Level

Python's C extension API has historically been one of it's strengths. A huge ecosystem of high-performance scientific and numerical software has been built on top of it. It got us this far, but now it's holding us back. It's time to take it to the next level.

What needs to change

The existing C API exposes many details of how Python was implemented. This prevents both the evolution of the current Python implementation and the successful creation of new implementations.

In particular, the existing C API exposes the implementation of a Python objects rather than the semantics of a Python object.

In order to avoid exposing the details of the Python implementation, the next level API should:

1. Not return borrowed references.
2. Not steal references.
3. Not expose reference counting as part of the API.
4. Not expose the location of objects in memory.
5. Not rely on pointers for object identity.
6. Not expose the memory layout of Python objects as part of the API.
7. Not expose static types.
8. Expose Python (the language), not a specific Python implementation version.
9. Expose constructs generally useful in C, not constructs specific to a Python implementation version.
10. Provide an explicit execution context.

There are also many good things about the existing API and we don't want to change everything. The next level API should be a subset of the existing API as far as the changes above allow.

For a more detailed explanation of what needs to change, see What needs to change and why.

Constraints

1. The existing ecosystem of C extensions should continue to work.
2. Performance of existing C extensions run on CPython should be no worse.
3. There should be a sensible migration path for big C extensions (e.g. numpy).
4. There should be a way to easily pick up common errors introduced by migrating.

In short, we should avoid making the lives of maintainers and users of C extensions difficult.

Why do this?

If everything above is achievable -- and we believe it is! -- we'll arrive in a wonderful new future where Python implementations can experiment with all sorts of amazing new features:

• tracing garbage collectors
• nurseries for short-lived objects
• sub-interpreters with separate contexts
• specialised implementations of lists
• removing the GIL
• avoiding the boxing of primitive types
• just-in-time compilation

... and many other things you can imagine that we haven't!

No one can guarantee that a particular new idea will work out, but exposing fewer implementation details via the C API will make it possible to try many new things.

The good news

Excitingly, we believe all of these changes are achievable within the constraints. We're currently working on one way to achieve them and we'd love input and feedback and welcome contributions of all kinds!