gh-119786: improve internal docs on co_linetable

[picnixz]

In #123168, I needed to play with artificial instructions where some positions are not available. It's trivial to have full positions information or no positions information, but it's hard to create co_linetable values where only some instructions have positions.

This PR aims to improve the internal docs so that we can easily remember what to do (and how to do it with an example).

• Issue: create an internals documentation folder in the cpython repo #119786

[markshannon]

Up until the section "Artificial constructions" this seems fine.

I don't think we want to be encouraging anyone to create position tables.

The reason for explaining the layout in detail is for anyone maintaining the code.
Even for out of process tools, like pyspy, we provide C functions for them to use, so they don't need to parse the tables.

To create tables for testing dis and such, I'd recommend parsing Python code, modifying the locations on the AST, then generating the code object with compile.
Like this:

import ast
import dis

code = """
def foo():
    a = 1
    b = 2
    c = 3
"""

tree = ast.parse(code)
func = tree.body[0]
b = func.body[1].targets[0]
b.lineno = 5
b.col_offset = 17
b.end_lineno = 6
b.end_col_offset = 18
foo = compile(tree,"test", "exec")
dis.dis(foo.co_consts[0], show_positions=True)

2:0-2:0              RESUME                   0

3:8-3:9              LOAD_CONST               1 (1)
3:4-3:5              STORE_FAST               0 (a)

4:8-4:9              LOAD_CONST               2 (2)

5:17-6:18            STORE_FAST               1 (b)
5:8-5:9              LOAD_CONST               3 (3)
5:4-5:5              STORE_FAST               2 (c)
5:4-5:5              RETURN_CONST             0 (None)

[picnixz]

To create tables for testing dis and such, I'd recommend parsing Python code, modifying the locations on the AST, then generating the code object with compile.

Oh, that's what I was searching for in #123168. Should I amend this PR and use your approach instead?

[iritkatriel]

Yes, make a new PR though as that one was merged.

[picnixz]

Actually, modifying the AST does not help because I can't remove some attributes that are expected to exist on the nodes. In particular, I cannot cancel positions (but I can easily change the values of the positions). So I think we need to stick with the co_linetable approach (unless there is something I'm missing?)

[markshannon]

I see. You can't set an expression to have no location, because the compiler will just give it one.
FTR, you could test the no columns case by setting node.end_lineno = node.end_col_offset = -1, and leaving the line number.

I think the test you wrote is fine. There is need to rewrite it.

[picnixz]

no columns case by setting node.end_lineno = node.end_col_offset = -1,

Yes, that's what I saw. But I'm not sure whether this is actually a bug or a feature of the compiler (namely, whether the -1 is really meant to indicate "no location").

[picnixz]

So I managed to use AST tricks only though I'm not really sure that setting the columns to -1 means that we are deleting the information. I'll remove the artificial constructions in these docs anyway.

[picnixz]

Friendly ping @iritkatriel (I'll fix the merge conflicts once I'm back, sorry missed that one)

[iritkatriel]

There's a merge conflict now.

[picnixz]

Since I'm on mobile I cannot resolve it but I'll take care of it on Friday (or maybe you can if you have time). Sorry for bothering you!

[picnixz]

@iritkatriel Ok, actually the locations.md file was gone! So I fixed the conflicts. By the way, I put everything under the "Source code locations" section (semantically speaking I think the way we encode integers, the meaning of locations and the short forms are all part of this topic).

[iritkatriel]

Thank you!