Tokenise suffixes on all literals #19103
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
It's not run, but this ensures that the code at least doesn't go out of date.
|
I made some changes to
However, I don't have antlr4 working properly: I downloaded it and it generated a java file (so my new grammar parses correctly), but building that just gave me a pile of "symbol not defined" errors. I don't know how to wrangle java & classpaths into doing what I want very well at all... |
This adds an optional suffix at the end of a literal token: `"foo"bar`. An actual use of a suffix in a expression (or other literal that the compiler reads) is rejected in the parser. This doesn't switch the handling of numbers to this system, and doesn't outlaw illegal suffixes for them yet.
This moves errors and all handling of numeric suffixes into the parser rather than the lexer.
This makes the formal lexical grammar (more closely) reflect the one implemented by the compiler.
This changes the stated grammar of literals to move all suffixes into the generic literal production.
huonw
force-pushed
the
literal-suffixes
branch
from
18fc033
to
a11078f
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Futureproof Rust for fancier suffixed literals. The Rust compiler tokenises a literal followed immediately (no whitespace) by an identifier as a single token: (for example) the text sequences
"foo"bar,1bazand1u1024are now a single token rather than the pairs"foo"bar,1bazand1u1024respectively.The compiler rejects all such suffixes in the parser, except for the 12 numeric suffixes we have now.
I'm fairly sure this will affect very few programs, since it's not currently legal to have
<literal><identifier>in a Rust program, except in a macro invocation. Any macro invocation relying on this behaviour can simply separate the two tokens with whitespace:foo!("bar"baz)becomesfoo!("bar" baz).This implements RFC 463, and so closes #19088.