gh-116022: Improve repr() of AST nodes

[tomasr8]

closes #116022

Based on the original issue the new repr() has:

• limited depth: nodes which are to deep are represented as Node(...)
• compressed list/tuple fields (e.g. body): more than 2 items are shown as [repr(first), ..., repr(last)]

Examples (output is formatted to be more readable):

>>> import ast
>>> ast.parse("x = 3")
Module(
    body=[Assign(targets=[Name(...)], value=Constant(...), type_comment=None)],
    type_ignores=[]
)


>>> ast.parse('x=7; y=4; c,z=3,4')
Module(
    body=[
        Assign(targets=[Name(...)], value=Constant(...), type_comment=None),
        ...,
        Assign(targets=[Tuple(...)], value=Tuple(...), type_comment=None)
    ],
    type_ignores=[]
)


>>> from pathlib import Path
>>> import typing
>>> ast.parse(Path(typing.__file__).read_text())
Module(
    body=[
        Expr(value=Constant(...)),
        ...,
        FunctionDef(
            name="__getattr__",
            args=arguments(...),
            body=[Expr(...), ..., Return(...)],
            decorator_list=[],
            returns=None,
            type_comment=None,
            type_params=[]
        )
    ],
    type_ignores=[]
)

Happy to take feedback :)

• Issue: Improve repr() of AST nodes #116022

[JelleZijlstra]

Thank you, and sorry I missed this PR!

I think your approach makes sense but there's a number of small things missing in the C code, and we need good unit test coverage for the new code.

[tomasr8]

Thanks for taking a look!

I think your approach makes sense

If you had a different approach in mind, I'm happy to change it. This is what makes sense to me, but maybe it's not the best way to go about it..

I've been busy with work lately, but I'll try to add more tests to the PR in the coming days :)

[JelleZijlstra]

I think this should also be documented in the ast module (a versionchanged note saying that the repr() of AST nodes now includes the values of fields) and in the What's New for 3.13. (Or 3.14 if we don't get this ready before Tuesday.)

[tomasr8]

I moved the tests to a separate data folder and added a CLI toggle to automatically regenerate them with ./python Lib/test/test_ast.py --snapshot-update. This is basically snapshot testing similar to how it's done in pytest for example. unittest unfortunately doesn't have this capability so it must be implemented separately in each module :/

The advantage of snapshot testing is that we don't need to write the test cases manually - both the initial tests and whenever they need to be updated. You can just run ./python Lib/test/test_ast.py --snapshot-update and compare the diff.

@ambv In case you have some thoughts on this

[tomasr8]

^ fixed the conflicts in tests

[JelleZijlstra]

Looking pretty good!

[JelleZijlstra]

You can DECREF name and value earlier so you don't have to repeat this. Looks like value is no longer needed after line 1602, so you could DECREF it there. name could be DECREFed after line 1623.

[tomasr8]

Indeed, I move those DECREFs up

[JelleZijlstra]

There's some CI failure that looks like something needs to be tweaked in relation to the new file.

[tomasr8]

There's some CI failure that looks like something needs to be tweaked in relation to the new file.

Thanks for pointing that out, should be fixed now :)

[JelleZijlstra]

There's some CI failures related to conversion warnings, I think there are some changes landing soon around that so I'll wait for now.

[tomasr8]

Thanks @JelleZijlstra and everyone else for reviews and help on getting this done!