Commits on Apr 11, 2021

1. Implement token-based handling of attributes during expansion …

   This PR modifies the macro expansion infrastructure to handle attributes
   in a fully token-based manner. As a result:
   
   * Derives macros no longer lose spans when their input is modified
     by eager cfg-expansion. This is accomplished by performing eager
     cfg-expansion on the token stream that we pass to the derive
     proc-macro
   * Inner attributes now preserve spans in all cases, including when we
     have multiple inner attributes in a row.
   
   This is accomplished through the following changes:
   
   * New structs `AttrAnnotatedTokenStream` and `AttrAnnotatedTokenTree` are introduced.
     These are very similar to a normal `TokenTree`, but they also track
     the position of attributes and attribute targets within the stream.
     They are built when we collect tokens during parsing.
     An `AttrAnnotatedTokenStream` is converted to a regular `TokenStream` when
     we invoke a macro.
   * Token capturing and `LazyTokenStream` are modified to work with
     `AttrAnnotatedTokenStream`. A new `ReplaceRange` type is introduced, which
     is created during the parsing of a nested AST node to make the 'outer'
     AST node aware of the attributes and attribute target stored deeper in the token stream.
   * When we need to perform eager cfg-expansion (either due to `#[derive]` or `#[cfg_eval]`),
   we tokenize and reparse our target, capturing additional information about the locations of
   `#[cfg]` and `#[cfg_attr]` attributes at any depth within the target.
   This is a performance optimization, allowing us to perform less work
   in the typical case where captured tokens never have eager cfg-expansion run.

   

   Aaron1011 committed Apr 11, 2021
   Configuration menu

   • View commit details
   • Copy full SHA for a93c4f0
   • Browse repository at this point

   Copy the full SHA

   a93c4f0 View commit details

   Browse the repository at this point in the history