Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize parse+extend to make only one proc macro bridge call #203

Merged
merged 1 commit into from
Dec 28, 2021
Merged

Conversation

dtolnay
Copy link
Owner

@dtolnay dtolnay commented Dec 28, 2021

proc_macro::TokenStream provides both Extend<TokenTree> and Extend<TokenStream>. They are implemented in libproc_macro as:

impl Extend<TokenTree> for TokenStream {
    fn extend<I: IntoIterator<Item = TokenTree>>(&mut self, trees: I) {
        self.extend(trees.into_iter().map(TokenStream::from));
    }
}

impl Extend<TokenStream> for TokenStream {
    fn extend<I: IntoIterator<Item = TokenStream>>(&mut self, streams: I) {
        *self = iter::once(mem::replace(self, Self::new())).chain(streams).collect();
    }
}

For our use case in quote, Extend<TokenTree> is worse because it involves taking the TokenStream we parse (which is the impl IntoIterator<Item = TokenTree> passed into the Extend impl), iterating it to get a sequence of TokenTrees, and converting those TokenTrees back to a TokenStream. That's circuitous! We had a TokenStream already.

This PR tweaks quote to use Extend<TokenStream> instead, which skips the needless round trip through TokenTree.

This optimization is also important for dtolnay/proc-macro2#320. If the TokenStream holds its contents as a CallSite-spanned string buffer instead of trees, using Extend<TokenTree> forces proc-macro2 to reparse that buffered representation in order to iterate its tokens, whereas Extend<TokenStream> allows the whole buffer to be efficiently appended to the lhs all at once.

@dtolnay dtolnay merged commit 0a04bae into master Dec 28, 2021
@dtolnay dtolnay deleted the parse branch December 28, 2021 03:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant