Public C API for accessing code object fields

[Fidget-Spinner]

A user/maintainer of a popular library documented their use case for accessing fields in the code object. In 3.11 the fields in C are gone but still available as properties in Python. Accessing these properties can be made faster by exposing their functions in the C API. The current option is PyObject_GetAttrString(code, "attr") which is much slower.

https://discuss.python.org/t/getting-the-class-name-of-a-code-frame-object-in-cpython-3-11-c-api/17396

A similar issue is #92154.

I am marking this as a release blocker @pablogsal as it would be mucho bueno if we could get this before rc1.

[pablogsal]

Mucho bueno indeed :)

I am willing to consider it but how many new APIs are we talking about? Also, we should double check and agree that exposing these fields is not going to backfire on us if we decide to change the implementation or add new stuff that will invalidate the meaning of the current fields. Maybe we can consider adding them with prefixed underscores (probably a bad idea)?

[pablogsal]

CC @markshannon @brandtbucher @iritkatriel

[Fidget-Spinner]

3 getter functions. I've thought about whether they will restrict our code. Thing is we already are restricted because all three are already exposed as Python properties. So we have no choice but to maintain backwards compatiblity, even without the new C API changes.

[markshannon]

PyObject_GetAttrString(code, "attr") may be slower, but I doubt that PyObject_GetAttr(code, attr) is much slower given that most of the attributes of code objects are computed.

Calling PyObject_GetAttr(code, co_varnames) requires that the co_varnames tuple is created. Dispatching is not likely to be any slower than creating the tuple.

If performance is the issue, we should offer a faster API (for 3.12)
Something like:
PyObject *PyCode_GetLocalName(int n)

We should also offer the same API in Python.

[markshannon]

Not only that, but co_nlocals and co_varnames don't make much sense.

def f(a, b):
     c = a + b
    def d(): 
       return c
    return d

def g(a, b):
     c = a + b
    def d(): 
       return a + b
    return d

>>> f.__code__.co_nlocals
3
>>> g.__code__.co_nlocals
4
>>> f.__code__.co_varnames
('a', 'b', 'd')
>>> g.__code__.co_varnames
('a', 'b', 'c', 'd')

[Fidget-Spinner]

If performance is the issue, we should offer a faster API (for 3.12) Something like: PyObject *PyCode_GetLocalName(int n)

We should also offer the same API in Python.

Yeah I planned to lazily create the introspection information and cache them there (kind of like how debug information in frames are lazily created). However, that missed that for 3.11 so it should go in 3.12. Something like

struct PyCodeObject {
    ...
    PyCodeIntrospectInformation *introspect_information;
}

struct debug information {
    PyObject *co_code;
    PyObject *co_varnames;
    PyObject *co_freevars;
    ...
}

[gvanrossum]

Given that the user requesting this is writing a profiler and needs access to the frame as well, why not just recommend that they include internal headers and call _PyCode_GetVarnames(), _PyCode_GetCellvars and _PyCode_GetFreevars directly? Looking at #95008 (presumably a tentative fix?) it would seem that all the required functions already exist but aren't public any more. That seems prudent since there are some concerns about the future-preparedness of these functions anyway.

In the past I think we would never have considered such a late request for new APIs a blocker.

[Fidget-Spinner]

OK. I'll remove the blocker. However, I think we should consider adding a performant version of these APIs in 3.12.

[pablogsal]

After reading the discussion, I concur with Guido. I am removing the 3.11 label. This feature can only be considered from 3.12 onwards.

[vstinner]

See also #91248

[Fidget-Spinner]

Docs TODO.

[gvanrossum]

Docs TODO.

What more docs do you need?

[Fidget-Spinner]

Docs TODO.

What more docs do you need?

Woops I forgot that I already documented these. Sorry 😆 .

I'll leave this issue open for the next thing I plan to do in 3.12:

I want a better/more performant implementation for 3.12. It will cache the debug info in a lazy debug struct much like how we do it for PyInterpreterFrame and PyFrameObject now.

[vstinner]

IMO the problem of migrating existing C extensions to Python 3.11 is now solved. What's New in Python 3.11:

PyCodeObject no longer has the co_code, co_varnames, co_cellvars and co_freevars fields. Instead, use PyCode_GetCode(), PyCode_GetVarnames(), PyCode_GetCellvars() and PyCode_GetFreevars() respectively to access them via the C API. (Contributed by Brandt Bucher in bpo-46841 and Ken Jin in gh-92154 and gh-94936.)

I close the issue.

I'll leave this issue open for the next thing I plan to do in 3.12

Please open a separated issue for that.