diff --git a/writeup/fine_grained_tokens.md b/writeup/fine_grained_tokens.md index eb303c4..4543255 100644 --- a/writeup/fine_grained_tokens.md +++ b/writeup/fine_grained_tokens.md @@ -15,6 +15,7 @@ Each fine-grained token has a kind, and possibly also some attributes | `Identifier` | represented identifier | | `RawIdentifier` | represented identifier | | `LifetimeOrLabel` | name | +| `RawLifetimeOrLabel` | name | | `CharacterLiteral` | represented character, suffix | | `ByteLiteral` | represented byte, suffix | | `StringLiteral` | represented string, suffix | diff --git a/writeup/pretokens.md b/writeup/pretokens.md index 3fb4099..bc6573b 100644 --- a/writeup/pretokens.md +++ b/writeup/pretokens.md @@ -14,6 +14,7 @@ Each pretoken has a *kind*, and possibly also some attributes, as described in t | `Identifier` | identifier | | `RawIdentifier` | identifier | | `LifetimeOrLabel` | name | +| `RawLifetimeOrLabel` | name | | `SingleQuoteLiteral` | prefix, literal content, suffix | | `DoubleQuoteLiteral` | prefix, literal content, suffix | | `RawDoubleQuoteLiteral` | prefix, literal content, suffix | diff --git a/writeup/reprocessing_cases.md b/writeup/reprocessing_cases.md index 67ba342..c9e06de 100644 --- a/writeup/reprocessing_cases.md +++ b/writeup/reprocessing_cases.md @@ -8,6 +8,7 @@ [`Identifier`](#identifier)\ [`RawIdentifier`](#rawidentifier)\ [`LifetimeOrLabel`](#lifetimeorlabel)\ +[`RawLifetimeOrLabel`](#rawlifetimeorlabel)\ [`SingleQuoteLiteral`](#singlequoteliteral)\ [`DoubleQuoteLiteral`](#doublequoteliteral)\ [`RawDoubleQuoteLiteral`](#rawdoublequoteliteral)\ @@ -163,6 +164,20 @@ A `LifetimeOrLabel` pretoken is always accepted. > See [NFC normalisation for lifetime/label]. +#### `RawLifetimeOrLabel` { .rcase } + +Fine-grained token kind produced: +`RawLifetimeOrLabel` + +A `RawLifetimeOrLabel` pretoken is always accepted. + +##### Attributes +name: copied + +> Note that the name is not NFC-normalised. +> See [NFC normalisation for lifetime/label]. + + #### `SingleQuoteLiteral` { .rcase } The pretokeniser guarantees the pretoken's prefix attribute is one of the following: diff --git a/writeup/rules.md b/writeup/rules.md index 7b4fd1d..cfb1472 100644 --- a/writeup/rules.md +++ b/writeup/rules.md @@ -6,6 +6,8 @@ [Unterminated block comment](#unterminated-block-comment)\ [Punctuation](#punctuation)\ [Single-quoted literal](#single-quoted-literal)\ +[Raw lifetime or label (Rust 2021)](#raw-lifetime-or-label-rust-2021)\ +[Reserved lifetime or label prefix (Rust 2021)](#reserved-lifetime-or-label-prefix-rust-2021)\ [Lifetime or label](#lifetime-or-label)\ [Double-quoted non-raw literal (Rust 2015 and 2018)](#double-quoted-non-raw-literal-rust-2015-and-2018)\ [Double-quoted non-raw literal (Rust 2021)](#double-quoted-non-raw-literal-rust-2021)\ @@ -190,6 +192,43 @@ when `character_sequence` represents an iterator over the sequence of characters | suffix | captured characters | +#### Raw lifetime or label (Rust 2021) { .rule } + +##### Pattern +``` +' r \# +(? + [ \p{XID_Start} _ ] + \p{XID_Continue} * +) +``` + +##### Pretoken kind +`RawLifetimeOrLabel` + +##### Attributes +| | | +|:----------------|:--------------------| +| name | captured characters | + + +#### Reserved lifetime or label prefix (Rust 2021) { .rule } + +##### Pattern +``` +' +[ \p{XID_Start} _ ] +\p{XID_Continue} * +\# +``` + +##### Pretoken kind +`Reserved` + +##### Attributes +(none) + + #### Lifetime or label { .rule } ##### Pattern diff --git a/writeup/rustc_oddities.md b/writeup/rustc_oddities.md index 08d3354..53ee8e1 100644 --- a/writeup/rustc_oddities.md +++ b/writeup/rustc_oddities.md @@ -11,7 +11,7 @@ But this doesn't happen for lifetimes or labels, so `'Kelvin` and `'Kelvin` ar For example, [this][playground-lifetime] compiles without warning in Rust 1.80, while [this][playground-ident] doesn't. In this writeup, the represented identifier attribute of `Identifier` and `RawIdentifier` fine-grained tokens is in NFC, -and the name attribute of `LifetimeOrLabel` tokens isn't. +and the name attribute of `LifetimeOrLabel` and `RawLifetimeOrLabel` tokens isn't. I think this behaviour is a promising candidate for provoking the "Wait...that's what we currently do? We should fix that."