add full-parser, like full-lexer

[benmkw]

Currently it does not seem to be possible to get an AST with comments attached?

[youknowone]

That will be a good feature if you need one.
Since I don't think you will need NonLogicalNewline, maybe just comment feature?

[benmkw]

I tried looking into ways to add comments but could not quickly figure out how to best add it to the grammar.

Then I looked at ruff cause they do formatting now as well and use this parser (although it seems they vendor/ fork it) and here is some code which does it for them https://github.com/astral-sh/ruff/tree/9cb5ce750e9dead53eb8235e8cfb7cff653a1cd2/crates/ruff_python_formatter/src/comments
( https://github.com/astral-sh/ruff/blob/9cb5ce750e9dead53eb8235e8cfb7cff653a1cd2/crates/ruff_python_formatter/src/comments/mod.rs#L319-L332 ) but not in a way which integrates into the parser but it seems to be retrofitted after parsing as far as I can see (maybe adding it the grammar is not trivial after all?).

Since I don't think you will need NonLogicalNewline, maybe just comment feature?

Yes that would be enough for me, it does not have to be a perfect reproduction, but I'd want to preserve the meaning/ content of the comments.

[MichaReiser]

Fitting comments into the grammar will be difficult because comments can appear in almost all positions.

Ruff's comment handling is specific to the formatter and not a general purpose solution.

I recommend taking a look at how oxc handles comments

https://github.com/web-infra-dev/oxc/blob/main/crates/oxc_ast/src/trivia.rs

An entirely different approach is to use a cst similar to rome/biome/Roslyn/swift

https://boshen.github.io/javascript-parser-in-rust/blog/rome/