Make SyncDroplessArena allocations thread-local in the fast path #61873

Zoxc · 2019-06-15T20:03:26Z

This works by having a thread local chunk to allocate from and a global list of chunks for in_arena to use.

rust-highfive · 2019-06-15T22:26:57Z

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_time:end:138e0ab4:start=1560636922330806312,finish=1560636924688547365,duration=2357741053
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
$ export AWS_ACCESS_KEY_ID=AKIA46X5W6CZEJZ6XT55
---
[00:07:20]    Compiling rustc_data_structures v0.0.0 (/checkout/src/librustc_data_structures)
[00:07:22] error[E0308]: mismatched types
[00:07:22]    --> src/librustc_data_structures/sync.rs:229:35
[00:07:22]     |
[00:07:22] 229 |                 SharedWorkerLocal(OneThread::new(f(0)))
[00:07:22]     |                                   ^^^^^^^^^^^^^^^^^^^^ expected type parameter, found struct `sync::OneThread`
[00:07:22]     = note: expected type `T`
[00:07:22]     = note: expected type `T`
[00:07:22]                found type `sync::OneThread<T>`
[00:07:23] error: aborting due to previous error
[00:07:23] 
[00:07:23] For more information about this error, try `rustc --explain E0308`.
[00:07:23] error: Could not compile `rustc_data_structures`.
---
travis_time:end:0301eff5:start=1560637392091412071,finish=1560637392096760232,duration=5348161
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:03a6275a
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:2f9866ec
travis_time:start:2f9866ec
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:014422e0
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

eddyb · 2019-06-16T18:00:44Z

I don't know how any of this concurrent code works, sorry! (Or rather, why it might be safe)
Maybe if @aturon gets back, he can help with reviewing it?

cc @rust-lang/compiler Is anyone else comfortable with concurrent data structures and willing to review?

nikomatsakis · 2019-06-18T12:06:15Z

I'll review it and/or get @aturon to do so =)

Dylan-DPC-zz · 2019-07-08T14:13:30Z

ping from triage @nikomatsakis @aturon any updates on this?

aturon · 2019-07-08T18:12:30Z

Hey @Dylan-DPC, I plan to review this one, however at the moment I'm getting up to speed on the concurrency approach in general.

Dylan-DPC-zz · 2019-07-09T07:57:12Z

thanks. No issues.

eddyb · 2019-07-11T13:03:14Z

Also pinging @gankro and @stjepang, in case they want to review as well.

Gankra · 2019-07-11T19:12:13Z

I do not

hdhoang · 2019-07-26T01:33:48Z

ping from triage @stjepang, could you provide comment on this PR?

Centril · 2019-08-08T14:47:40Z

src/libarena/lib.rs

-    pub fn clear(&mut self) {
-        self.lock.get_mut().clear();
+    fn grow(&self, n: usize, chunks: &mut Vec<TypedArenaChunk<T>>) {
+        unsafe {


Would be great if these large unsafe { .. } blocks would have comments justifying their soundness.
(Applies throughout the PR.)

wirelessringo · 2019-08-16T06:55:12Z

Ping from triage. @aturon any updates on this? Thanks

JohnCSimon · 2019-08-24T01:58:29Z

Pinging again from triage
This PR has sat idle waiting for review for eight days.
@Centril @aturon

Thanks.

Dylan-DPC-zz · 2019-08-24T11:11:53Z

Waiting for a review from @rust-lang/compilers

oli-obk · 2019-08-26T00:55:39Z

src/libarena/lib.rs

+impl<T> CurrentChunk<T> {
+    #[inline]
+    fn align(&self, align: usize) {
+        let final_address = ((self.ptr.get() as usize) + align - 1) & !(align - 1);


use https://doc.rust-lang.org/std/primitive.pointer.html#method.align_offset instead of handrolling the computation

oli-obk · 2019-08-26T00:57:07Z

src/libarena/lib.rs

-        self.lock.get_mut().clear();
+    fn grow(&self, n: usize, chunks: &mut Vec<TypedArenaChunk<T>>) {
+        unsafe {
+            let (chunk, mut new_capacity);


move chunk declaration to initialization site

oli-obk · 2019-08-26T01:00:11Z

src/libarena/lib.rs

+    fn grow(&self, n: usize, chunks: &mut Vec<TypedArenaChunk<T>>) {
+        unsafe {
+            let (chunk, mut new_capacity);
+            if let Some(last_chunk) = chunks.last_mut() {


The statements in the Some block of this if let deserve some documenting comments.

oli-obk · 2019-08-26T01:17:57Z

src/libarena/lib.rs

+
+            current.align(align);
+
+            let future_end = intrinsics::arith_offset(current.ptr.get(), bytes as isize);


why not use https://doc.rust-lang.org/std/primitive.pointer.html#method.wrapping_offset here? Intrinsics should not be used except in their wrapper code

Alternatively you can use offset_from to compute the offset between end and ptr and see if that is bigger than bytes. I think I'd prefer it that way around.

Oh, we stabilized this intrinsic? That seems unfortunate as it can be used to create invalid pointers (which are UB in C).

It's not stable. and it's still unsafe, so if you think there are problems with it, please raise them on the tracking issue (#41079). wrapping_offset_from is safe.

oli-obk · 2019-08-26T01:20:05Z

src/libarena/lib.rs

+            let ptr = current.ptr.get();
+            // Set the pointer past ourselves
+            current.ptr.set(
+                intrinsics::arith_offset(current.ptr.get(), bytes as isize) as *mut u8,


since we just grew the thing, we can now use ptr::offset, right?

JohnCSimon · 2019-09-07T16:01:38Z

Ping from triage -
@Zoxc @oli-obk @rust-lang/compilers
This has sat idle for two weeks. What needs to happen for this pr?

Thanks

joelpalmer · 2019-09-16T10:38:13Z

Ping from Triage: closing due to inactivity. Please re-open if there are updates. @Zoxc

rust-highfive assigned eddyb Jun 15, 2019

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jun 15, 2019

Zoxc force-pushed the tls-arena branch from 21f3a94 to 5b5cea8 Compare June 15, 2019 22:14

Zoxc added 2 commits June 16, 2019 00:37

Add SharedWorkerLocal

f1961bc

Make SyncDroplessArena allocations thread-local in the fast path

512a1cd

Zoxc force-pushed the tls-arena branch from 5b5cea8 to 512a1cd Compare June 15, 2019 22:37

nikomatsakis assigned nikomatsakis and aturon and unassigned eddyb Jun 18, 2019

Dylan-DPC-zz unassigned nikomatsakis Jul 9, 2019

eddyb mentioned this pull request Jul 11, 2019

[WIP] Use a sharded dep node to dep node index map #61845

Closed

nagisa added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-team Status: Awaiting decision from the relevant subteam (see the T-<team> label). labels Aug 8, 2019

Centril reviewed Aug 8, 2019

View reviewed changes

oli-obk requested changes Aug 26, 2019

View reviewed changes

oli-obk assigned oli-obk and unassigned aturon Aug 26, 2019

oli-obk added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 8, 2019

joelpalmer closed this Sep 16, 2019

joelpalmer added S-inactive Status: Inactive and waiting on the author. This is often applied to closed PRs. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Sep 16, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make SyncDroplessArena allocations thread-local in the fast path #61873

Make SyncDroplessArena allocations thread-local in the fast path #61873

Zoxc commented Jun 15, 2019

rust-highfive commented Jun 15, 2019

eddyb commented Jun 16, 2019

nikomatsakis commented Jun 18, 2019

Dylan-DPC-zz commented Jul 8, 2019

aturon commented Jul 8, 2019

Dylan-DPC-zz commented Jul 9, 2019

eddyb commented Jul 11, 2019

Gankra commented Jul 11, 2019

hdhoang commented Jul 26, 2019

Centril Aug 8, 2019

wirelessringo commented Aug 16, 2019

JohnCSimon commented Aug 24, 2019

Dylan-DPC-zz commented Aug 24, 2019

oli-obk Aug 26, 2019

oli-obk Aug 26, 2019

oli-obk Aug 26, 2019

oli-obk Aug 26, 2019

oli-obk Aug 26, 2019

Zoxc Sep 25, 2019

oli-obk Sep 25, 2019

oli-obk Aug 26, 2019

JohnCSimon commented Sep 7, 2019

joelpalmer commented Sep 16, 2019


		current.align(align);

		let future_end = intrinsics::arith_offset(current.ptr.get(), bytes as isize);

Make SyncDroplessArena allocations thread-local in the fast path #61873

Make SyncDroplessArena allocations thread-local in the fast path #61873

Conversation

Zoxc commented Jun 15, 2019

rust-highfive commented Jun 15, 2019

eddyb commented Jun 16, 2019

nikomatsakis commented Jun 18, 2019

Dylan-DPC-zz commented Jul 8, 2019

aturon commented Jul 8, 2019

Dylan-DPC-zz commented Jul 9, 2019

eddyb commented Jul 11, 2019

Gankra commented Jul 11, 2019

hdhoang commented Jul 26, 2019

Choose a reason for hiding this comment

wirelessringo commented Aug 16, 2019

JohnCSimon commented Aug 24, 2019

Dylan-DPC-zz commented Aug 24, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JohnCSimon commented Sep 7, 2019

joelpalmer commented Sep 16, 2019