rustc: Allow changing the default allocator #27400

alexcrichton · 2015-07-30T16:51:52Z

This commit is an implementation of RFC 1183 which allows swapping out
the default allocator on nightly Rust. No new stable surface area should be
added as a part of this commit.

Two new attributes have been added to the compiler:

#![needs_allocator] - this is used by liballoc (and likely only liballoc) to
indicate that it requires an allocator crate to be in scope.
#![allocator] - this is a indicator that the crate is an allocator which can
satisfy the needs_allocator attribute above.

The ABI of the allocator crate is defined to be a set of symbols that implement
the standard Rust allocation/deallocation functions. The symbols are not
currently checked for exhaustiveness or typechecked. There are also a number of
restrictions on these crates:

An allocator crate cannot transitively depend on a crate that is flagged as
needing an allocator (e.g. allocator crates can't depend on liballoc).
There can only be one explicitly linked allocator in a final image.
If no allocator is explicitly requested one will be injected on behalf of the
compiler. Binaries and Rust dylibs will use jemalloc by default where
available and staticlibs/other dylibs will use the system allocator by
default.

Two allocators are provided by the distribution by default, alloc_system and
alloc_jemalloc which operate as advertised.

Closes #27389

rust-highfive · 2015-07-30T16:51:57Z

r? @pcwalton

(rust_highfive has picked a reviewer for you, use r? to override)

alexcrichton · 2015-07-30T16:52:07Z

r? @brson

cc @nnethercote

pnkfelix · 2015-07-30T17:44:53Z

src/librustc/metadata/creader.rs

@@ -49,15 +49,16 @@ impl<'a, 'v> visit::Visitor<'v> for CrateReader<'a> {
 }

 fn dump_crates(cstore: &CStore) {
-    debug!("resolved crates:");


Was the switch from debug! to info! (and likewise from log::DEBUG to log::INFO) deliberate, or an accident?

I would have thought we could (and should) leave it as it was, under debug!

Yeah this was currently intentional but I wouldn't mind changing it back. Whenever someone's having weird behavior with loading crates I typically ask them to set RUST_LOG=rustc::metadata::creader,rustc::metadata::loader to get info like this, but debug! means it's all compiled out by default.

Along those lines I'd somewhat prefer to have it be info! but I don't have much of a preference either way

nnethercote · 2015-07-30T23:27:42Z

@alexcrichton: can I ask what the code would look like for a program (such as Servo) to implement allocation wrappers? I know we talked about this at Whistler but I don't remember the details and now that you have an implementation to reference it might be easier to describe. Thank you.

alexcrichton · 2015-07-30T23:38:20Z

Certainly! You'd basically just create a copy of src/liballoc_jemalloc/lib.rs, filling in different implementations for the various allocation functions. You'd then just say extern crate foo to link in your allocator and you'd be good to go!

retep998 · 2015-07-30T23:46:50Z

src/liballoc_system/lib.rs

+mod imp {
+    use core::mem::size_of;
+    use libc::{BOOL, DWORD, HANDLE, LPVOID, SIZE_T, INVALID_HANDLE_VALUE};
+    use libc::{WriteFile};


WriteFile and INVALID_HANDLE_VALUE aren't needed.

retep998 · 2015-07-31T01:11:43Z

src/liballoc_system/lib.rs

+
+#[cfg(windows)]
+mod imp {
+    use core::mem::size_of;


size_of is also unused it seems.

brson · 2015-07-31T18:14:46Z

This feels very complex, makes me want to remove jemalloc.

retep998 · 2015-07-31T18:21:46Z

Considering how jemalloc is responsible for #26647 and is also really brittle when used under conemu #14600 (comment).

You know, this reminds me of the time we had green threaded IO, then split it using virtual dispatch between a green and a native portion, then later dropped the green part entirely.

steveklabnik · 2015-07-31T18:28:06Z

jemalloc tends to have _way_ better performance than libc's malloc or the system malloc, though. At least, in my understanding...

alexcrichton · 2015-07-31T18:40:36Z

@brson

This feels very complex, makes me want to remove jemalloc.

Could you elaborate a bit on this? I can certainly more aggressively document various aspects and I certainly wouldn't mind rewriting various code paths. I don't think that we want to just look at this and jettison jemalloc, though, as there are very real other use cases for swapping out for custom allocators (such as Servo's memory tracking, embedded systems with different allocators, etc).

If we followed reasoning like this I would also say that supporting both dylibs/rlibs at the same time is pretty complex, but I don't feel the need to remove that distinction. There's concrete use cases for both so we just need to support both in as robust a manner as possible.

@retep998

You know, this reminds me of the time we had green threaded IO, then split it using virtual dispatch between a green and a native portion, then later dropped the green part entirely.

I don't think that this is the same situation as here most notably because there's no virtual dispatch here. This is a static decision which is made that has no costs associated with it (unlike the virtual dispatch).

Like I said above, complexity is not a reason to jettison something, it's a reason to rethink why it's so complex and consider other options, but in this case I don't consider removing jemalloc an option.

If there are bugs on Windows caused by our usage of jemalloc (I was unaware of these), then we should track them down and fix them, but removing it from all platforms is a pretty heavy hammer.

alexcrichton · 2015-07-31T21:35:19Z

I'm currently running a crater build for this PR

alexcrichton · 2015-08-01T01:42:16Z

Crater reports one regression but I have confirmed locally that it was spurious.

nnethercote · 2015-08-02T23:04:42Z

A few comments...

The ability to provide custom allocators is very important and useful.
In the context of Rust, I've heard it said many times that jemalloc is faster, but I haven't seen any measurements showing this. (Hmm, now that I think of it, somebody may have mentioned that using jemalloc sped up rustc significantly? Not sure.)
Servo uses a mixture of jemalloc (for Rust code) and the system heap (for C/C++ code, e.g. SpiderMonkey). Servo's memory reporting code thus has machinery for dealing with both: it reports the total size of both heaps, and it uses je_malloc_usable_size to measure jemalloc heap blocks and malloc_usable_size (on Linux) or malloc_size (on Mac) to measure system heap blocks.

bors · 2015-08-03T23:49:59Z

☔ The latest upstream changes (presumably #27210) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2015-08-05T03:55:20Z

☔ The latest upstream changes (presumably #27393) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2015-08-05T12:10:48Z

☔ The latest upstream changes (presumably #27458) made this pull request unmergeable. Please resolve the merge conflicts.

alexcrichton · 2015-08-13T23:40:25Z

@bors: r=brson abd839

bors · 2015-08-13T23:40:25Z

🙀 abd839 is not a valid commit SHA. Please try again with adbd839.

alexcrichton · 2015-08-13T23:40:34Z

@bors: r=brson adbd839

bors · 2015-08-14T01:06:36Z

🔒 Merge conflict

bors · 2015-08-14T01:06:57Z

☔ The latest upstream changes (presumably #27684) made this pull request unmergeable. Please resolve the merge conflicts.

alexcrichton · 2015-08-14T18:05:29Z

@bors: r=brson 9d21851

bors · 2015-08-14T18:13:32Z

⌛ Testing commit 9d21851 with merge 7959211...

bors · 2015-08-14T19:19:22Z

💔 Test failed - auto-linux-64-x-android-t

alexcrichton · 2015-08-14T20:45:44Z

@bors: retry

On Fri, Aug 14, 2015 at 12:19 PM, bors [email protected] wrote:

[image: 💔] Test failed - auto-linux-64-x-android-t
http://buildbot.rust-lang.org/builders/auto-linux-64-x-android-t/builds/6061

—
Reply to this email directly or view it on GitHub
#27400 (comment).

bors · 2015-08-14T21:02:29Z

⌛ Testing commit 9d21851 with merge d5f5dca...

bors · 2015-08-14T22:07:33Z

💔 Test failed - auto-linux-64-x-android-t

This commit is an implementation of [RFC 1183][rfc] which allows swapping out the default allocator on nightly Rust. No new stable surface area should be added as a part of this commit. [rfc]: rust-lang/rfcs#1183 Two new attributes have been added to the compiler: * `#![needs_allocator]` - this is used by liballoc (and likely only liballoc) to indicate that it requires an allocator crate to be in scope. * `#![allocator]` - this is a indicator that the crate is an allocator which can satisfy the `needs_allocator` attribute above. The ABI of the allocator crate is defined to be a set of symbols that implement the standard Rust allocation/deallocation functions. The symbols are not currently checked for exhaustiveness or typechecked. There are also a number of restrictions on these crates: * An allocator crate cannot transitively depend on a crate that is flagged as needing an allocator (e.g. allocator crates can't depend on liballoc). * There can only be one explicitly linked allocator in a final image. * If no allocator is explicitly requested one will be injected on behalf of the compiler. Binaries and Rust dylibs will use jemalloc by default where available and staticlibs/other dylibs will use the system allocator by default. Two allocators are provided by the distribution by default, `alloc_system` and `alloc_jemalloc` which operate as advertised. Closes rust-lang#27389

alexcrichton · 2015-08-14T22:15:08Z

@bors: r=brson 45bf1ed

bors · 2015-08-14T22:23:14Z

⌛ Testing commit 45bf1ed with merge 290cf6b...

alexcrichton · 2015-08-14T22:52:09Z

@bors: retry force

This commit is an implementation of [RFC 1183][rfc] which allows swapping out the default allocator on nightly Rust. No new stable surface area should be added as a part of this commit. [rfc]: rust-lang/rfcs#1183 Two new attributes have been added to the compiler: * `#![needs_allocator]` - this is used by liballoc (and likely only liballoc) to indicate that it requires an allocator crate to be in scope. * `#![allocator]` - this is a indicator that the crate is an allocator which can satisfy the `needs_allocator` attribute above. The ABI of the allocator crate is defined to be a set of symbols that implement the standard Rust allocation/deallocation functions. The symbols are not currently checked for exhaustiveness or typechecked. There are also a number of restrictions on these crates: * An allocator crate cannot transitively depend on a crate that is flagged as needing an allocator (e.g. allocator crates can't depend on liballoc). * There can only be one explicitly linked allocator in a final image. * If no allocator is explicitly requested one will be injected on behalf of the compiler. Binaries and Rust dylibs will use jemalloc by default where available and staticlibs/other dylibs will use the system allocator by default. Two allocators are provided by the distribution by default, `alloc_system` and `alloc_jemalloc` which operate as advertised. Closes #27389

bors · 2015-08-14T22:52:12Z

⌛ Testing commit 45bf1ed with merge ab450ef...

bors · 2015-08-15T01:32:04Z

☀️ Test successful - auto-linux-32-nopt-t, auto-linux-32-opt, auto-linux-64-nopt-t, auto-linux-64-opt, auto-linux-64-x-android-t, auto-mac-32-opt, auto-mac-64-nopt-t, auto-mac-64-opt, auto-win-gnu-32-nopt-t, auto-win-gnu-32-opt, auto-win-gnu-64-nopt-t, auto-win-gnu-64-opt, auto-win-msvc-32-opt, auto-win-msvc-64-opt

rust-highfive assigned pcwalton Jul 30, 2015

rust-highfive assigned brson and unassigned pcwalton Jul 30, 2015

alexcrichton force-pushed the less-jemalloc branch 2 times, most recently from 3e5fd87 to 8517834 Compare July 30, 2015 16:54

pnkfelix reviewed Jul 30, 2015
View reviewed changes

retep998 reviewed Jul 30, 2015
View reviewed changes

alexcrichton force-pushed the less-jemalloc branch from 8517834 to 6b95b66 Compare July 30, 2015 23:59

retep998 reviewed Jul 31, 2015
View reviewed changes

src/liballoc_system/lib.rs

#[cfg(windows)]

mod imp {

use core::mem::size_of;

Copy link

Member

retep998 Jul 31, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

size_of is also unused it seems.

alexcrichton force-pushed the less-jemalloc branch from 6b95b66 to 5dbf9be Compare July 31, 2015 02:30

alexcrichton force-pushed the less-jemalloc branch from 5dbf9be to 31b083f Compare August 4, 2015 00:25

alexcrichton force-pushed the less-jemalloc branch 2 times, most recently from 9e1e50d to 49632c0 Compare August 5, 2015 05:11

alexcrichton force-pushed the less-jemalloc branch from 49632c0 to 234b1bc Compare August 5, 2015 16:07

alexcrichton force-pushed the less-jemalloc branch from adbd839 to 9d21851 Compare August 14, 2015 18:05

alexcrichton force-pushed the less-jemalloc branch from 9d21851 to 45bf1ed Compare August 14, 2015 22:14

bors merged commit 45bf1ed into rust-lang:master Aug 15, 2015

bors mentioned this pull request Aug 15, 2015

SIMD groundwork part 1 #27169

Merged

5 tasks

alexcrichton deleted the less-jemalloc branch August 17, 2015 19:50

geofft mentioned this pull request Aug 31, 2015

Update patches for current Rust japaric-archived/std-with-cargo#14

Closed

apasel422 mentioned this pull request Sep 4, 2015

Valgrind doesn't report allocations with jemalloc on nightly #28224

Closed

dherman mentioned this pull request Sep 21, 2015

default to system rust when stable is ready neon-bindings/neon-cli#2

Closed

SimonSapin mentioned this pull request Apr 18, 2018

Tracking issue for changing the global, default allocator (RFC 1974) #27389

Closed

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rustc: Allow changing the default allocator #27400

rustc: Allow changing the default allocator #27400

alexcrichton commented Jul 30, 2015

rust-highfive commented Jul 30, 2015

alexcrichton commented Jul 30, 2015

pnkfelix Jul 30, 2015

alexcrichton Jul 30, 2015

nnethercote commented Jul 30, 2015

alexcrichton commented Jul 30, 2015

retep998 Jul 30, 2015

retep998 Jul 31, 2015

brson commented Jul 31, 2015

retep998 commented Jul 31, 2015

steveklabnik commented Jul 31, 2015 via email

alexcrichton commented Jul 31, 2015

alexcrichton commented Jul 31, 2015

alexcrichton commented Aug 1, 2015

nnethercote commented Aug 2, 2015

bors commented Aug 3, 2015

bors commented Aug 5, 2015

bors commented Aug 5, 2015

alexcrichton commented Aug 13, 2015

bors commented Aug 13, 2015

alexcrichton commented Aug 13, 2015

bors commented Aug 14, 2015

bors commented Aug 14, 2015

alexcrichton commented Aug 14, 2015

bors commented Aug 14, 2015

bors commented Aug 14, 2015

alexcrichton commented Aug 14, 2015

bors commented Aug 14, 2015

bors commented Aug 14, 2015

alexcrichton commented Aug 14, 2015

bors commented Aug 14, 2015

alexcrichton commented Aug 14, 2015

bors commented Aug 14, 2015

bors commented Aug 15, 2015

rustc: Allow changing the default allocator #27400

rustc: Allow changing the default allocator #27400

Conversation

alexcrichton commented Jul 30, 2015

rust-highfive commented Jul 30, 2015

alexcrichton commented Jul 30, 2015

pnkfelix Jul 30, 2015

Choose a reason for hiding this comment

alexcrichton Jul 30, 2015

Choose a reason for hiding this comment

nnethercote commented Jul 30, 2015

alexcrichton commented Jul 30, 2015

retep998 Jul 30, 2015

Choose a reason for hiding this comment

retep998 Jul 31, 2015

Choose a reason for hiding this comment

brson commented Jul 31, 2015

retep998 commented Jul 31, 2015

steveklabnik commented Jul 31, 2015 via email

alexcrichton commented Jul 31, 2015

alexcrichton commented Jul 31, 2015

alexcrichton commented Aug 1, 2015

nnethercote commented Aug 2, 2015

bors commented Aug 3, 2015

bors commented Aug 5, 2015

bors commented Aug 5, 2015

alexcrichton commented Aug 13, 2015

bors commented Aug 13, 2015

alexcrichton commented Aug 13, 2015

bors commented Aug 14, 2015

bors commented Aug 14, 2015

alexcrichton commented Aug 14, 2015

bors commented Aug 14, 2015

bors commented Aug 14, 2015

alexcrichton commented Aug 14, 2015

bors commented Aug 14, 2015

bors commented Aug 14, 2015

alexcrichton commented Aug 14, 2015

bors commented Aug 14, 2015

alexcrichton commented Aug 14, 2015

bors commented Aug 14, 2015

bors commented Aug 15, 2015