-
Notifications
You must be signed in to change notification settings - Fork 12.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove TokenStreamBuilder
#102692
Remove TokenStreamBuilder
#102692
Conversation
`TokenTree::Punct` is handled outside the `match`. This commits moves it inside the `match`, avoiding the need for the `return`s and making it easier to read.
`TokenStreamBuilder` exists to concatenate multiple `TokenStream`s together. This commit removes it, and moves the concatenation functionality directly into `TokenStream`, via two new methods `push_tree` and `push_stream`. This makes things both simpler and faster. `push_tree` is particularly important. `TokenStreamBuilder` only had a single `push` method, which pushed a stream. But in practice most of the time we push a single token tree rather than a stream, and `push_tree` avoids the need to build a token stream with a single entry (which requires two allocations, one for the `Lrc` and one for the `Vec`). The main `push_tree` use arises from a change to one of the `ToInternal` impls in `proc_macro_server.rs`. It now returns a `SmallVec` instead of a `TokenStream`. This return value is then iterated over by `concat_trees`, which does `push_tree` on each element. Furthermore, the use of `SmallVec` avoids more allocations, because there is always only one or two token trees. Note: the removed `TokenStreamBuilder::push` method had some code to deal with a quadratic blowup case from rust-lang#57735. This commit removes the code. I tried and failed to reproduce the blowup from that PR, before and after this change. Various other changes have happened to `TokenStreamBuilder` in the meantime, so I suspect the original problem is no longer relevant, though I don't have proof of this. Generally speaking, repeatedly extending a `Vec` without pre-determining its capacity is *not* quadratic. It's also incredibly common, within rustc and many other Rust programs, so if there were performance problems there you'd think it would show up in other places, too.
The return type can only appear once.
Hey! It looks like you've submitted a new PR for the library teams! If this PR contains changes to any Examples of
|
Best reviewed one commit at a time. cc @pnkfelix Because of the parts relating to #57735. |
@bors try @rust-timer queue |
Awaiting bors try build completion. @rustbot label: +S-waiting-on-perf |
⌛ Trying commit ed6f481 with merge 168fa6991448154472709aac01ef2a0051741518... |
☀️ Try build successful - checks-actions |
Queued 168fa6991448154472709aac01ef2a0051741518 with parent d4846f9, future comparison URL. |
Finished benchmarking commit (168fa6991448154472709aac01ef2a0051741518): comparison URL. Overall result: ✅ improvements - no action neededBenchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf. @bors rollup=never Instruction countThis is a highly reliable metric that was used to determine the overall result at the top of this comment.
Max RSS (memory usage)ResultsThis is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.
CyclesResultsThis is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.
Footnotes |
r=me, assuming that there are no objections from any of the other mentioned reviewers. |
It has been a week since the PR was filed, and I don't think it's controversial, so I will r+. It'll take a while to get merged, if anyone objects they can still r- in the meantime. @bors r=Aaron1011 |
☀️ Test successful - checks-actions |
Finished benchmarking commit (2b91cbe): comparison URL. Overall result: ✅ improvements - no action needed@rustbot label: -perf-regression Instruction countThis is a highly reliable metric that was used to determine the overall result at the top of this comment.
Max RSS (memory usage)ResultsThis is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.
CyclesResultsThis is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.
Footnotes |
…ron1011 Remove `TokenStreamBuilder` `TokenStreamBuilder` is used to combine multiple token streams. It can be removed, leaving the code a little simpler and a little faster. r? `@Aaron1011`
TokenStreamBuilder
is used to combine multiple token streams. It can be removed, leaving the code a little simpler and a little faster.r? @Aaron1011