Update FormatStringContinuation to use the new f-string tokens

[dhruvmanila]

The FormatStringContinuation implementation uses the full lexer for comment handling. It expects only a limited set of tokens (String, Newline, NonLogicalNewline, Comment, Indent, Dedent) which is expected given that the lexing is performed only on the string literals.

Now, with PEP 701, the f-string literal will instead emit FStringStart, FStringMiddle, and FStringEnd. Not only that, it'll also emit other tokens for the expression part of the f-string. This poses a challenge as currently it panics for any unexpected token encountered:

ruff/crates/ruff_python_formatter/src/expression/string.rs

Line 277 in 4ae463d

token => unreachable!("Unexpected token {token:?}"),

[MichaReiser]

Is this blocking for landing PEP701?

[dhruvmanila]

It's one of the failing test in the CI: https://github.com/astral-sh/ruff/actions/runs/6236001637/job/16926357093?pr=7376#step:7:2341

It shouldn't block Ruff in general but the formatter will panic when a f-string is encountered in implicit string concatenation with comments in it. For example,

foo = (
    f"bar"
    # comment
    "foo"
)

[MichaReiser]

Yeah, I don't think we can break the formatter so significantly at this stage

[dhruvmanila]

I've started looking into it. A basic implementation of using the outermost f-string range and ignoring all tokens within a f-string works but I'll look at if there are any other cases which may break this.

[dhruvmanila]

Currently, this is completed in a way so as to not block merging PEP 701. Tracking issue for handling comments and quotes in f-string is at #7594