Expose _PyInterpreterFrame_GetLine in the private API

[pablogsal]

In 3.11 we are using now an opaque pointer to _PyInterpreterFrame in the eval function. The problem is that for PEP 523's APIs, we changed the type of _PyFrameEvalFunction to be PyObject *(*_PyFrameEvalFunction)(PyThreadState *tstate, _PyInterpreterFrame *frame, int throwflag) but there is no way to do anything with _PyInterpreterFrame because is opaque. Even if the extension defines PY_BUILD_CORE the symbols are not exposed in the executable/libpython, which means that including the headers is insufficient.

This is a problem because it really limits what users of PEP 523 can do with the frame object that they receive, even if is opaque. Most profilers using PEP 523 API do it for speed reasons and the only thing they need is to get the line number, but that is now impossible because the pointer is opaque and we don't offer any exposed APIs.

Linked PRs

• GH-96803: Add three C-API functions to make _PyInterpreterFrame less opaque for users of PEP 523. #96849
• GH-96803: Document and test new unstable internal frame API functions #104211
• GH-96803: Move PyUnstable_InterpreterFrame_GetCode() to Python.h #107188
• [3.12] GH-96803: Move PyUnstable_InterpreterFrame_GetCode() to Python.h (GH-107188) #107195

[pablogsal]

CC: @markshannon @brandtbucher @iritkatriel @ericsnowcurrently

[ericsnowcurrently]

CC @mdboom

[markshannon]

I see your point, but I don't think that _PyInterpreterFrame_GetLine() is the right API.
I think the pair of _PyInterpreterFrame_GetCode() and _PyInterpreterFrame_GetLasti() would be better.
Getting the offset is much cheaper than computing the line number.
The line number can be computed, if needed.
Other information to be extracted from the code object, such as the filename and function name.

[pablogsal]

I started exposing _PyInterpreterFrame_GetCode and _PyInterpreterFrame_GetLasti but then I realized that neither _PyInterpreterFrame_GetCode and _PyInterpreterFrame_GetLasti are in 3.12 so exposing those is going to be more chaotic than just exposing _PyInterpreterFrame_GetLine. Notice that if the application defines PY_BUILD_CORE it can already access f_code and f_lasti manually from the struct.

As there can be arbitrary logic in _PyInterpreterFrame_GetLine I still think is a better idea to export _PyInterpreterFrame_GetLine.

[markshannon]

It still think that the pair of _PyInterpreterFrame_GetCode and _PyInterpreterFrame_GetLasti is a better interface.

If you think that doing that would be too disruptive for 3.11, we can add them for 3.12, and just expose _PyInterpreterFrame_GetLine for 3.11.

[pablogsal]

If you think that doing that would be too disruptive for 3.11, we can add them for 3.12, and just expose

In 3.12 they are already exposed if the user calls PY_BUILD_CORE because one is a macro and the other can be basically substituted by frame->f_code because the struct definition is available if PY_BUILD_CORE is defined. The problem here is having to re-implement _PyInterpreterFrame_GetLine.

Exposing _PyInterpreterFrame_GetCode and _PyInterpreterFrame_GetLasti doesn't give the user anything that cannot be achieved by just calling frame->f_code and frame->f_lasti after defining PY_BUILD_CORE

[markshannon]

_PyInterpreterFrame_GetLine is quite simple once we have the other two functions. There is no harm in exposing it as well.

the struct definition is available if PY_BUILD_CORE is defined.

All that does is expose the existence of a field call f_code. It doesn't say anything about its meaning or when it is valid.

In general, frame->f_code cannot be substituted for _PyInterpreterFrame_GetCode() as the meaning of frame->f_code is not defined. E.g. we might make it such that f_code is only valid for some frames faster-cpython/ideas#111

[pablogsal]

Ok, I am convinced. I will expose _PyInterpreterFrame_GetCode and _PyInterpreterFrame_GetLasti

[pablogsal]

Ah, turns out this has changed in ef6a482 and now is a macro on 3.11 as well. This means we can backport the 3.12 change

[pablogsal]

@markshannon I had another idea. Maybe we should expose the API to transform the interpreter frame into a python object. At that point you can do whatever you want with it. What do you think?

[markshannon]

Doesn't that undermine the whole point as using the _PyInterpreterFrame in the first place, i.e. efficiency?
Also, incomplete _PyInterpreterFrames have a code object and lasti, but we cannot safely produce a frame object for them

[erlend-aasland]

gh-96849 added new APIs, but no documentation was added. Mentioning it here, so we don't forget before we close this issue.

[encukou]

Note that even unstable API needs tests as well as documentation.