Valgrind doesn't report allocations with jemalloc on nightly

[apasel422]

Compiling the following code with opt-level=0, Valgrind reports 26 allocs for rustc 1.2.0 (082e47636 2015-08-03), but 5 allocs for rustc 1.4.0-nightly (cd138dc44 2015-09-02):

fn main() {
    let mut t = 0;

    for i in 0..10 {
        t += *Box::new(i);
    }

    assert_eq!(t, 45);
}

Perhaps this has something to do with #27400? In any case, it's harder to use Valgrind to debug Rust programs because of this.

[alexcrichton]

This is actually an optimization that LLVM is doing, it realizes that the allocation isn't necessary so it's elided entirely. As a result I believe this is working as intended, so closing.

[apasel422]

I'm not sure that's the case. Consider this program:

#![feature(test)]

extern crate test;

fn main() {
    for i in 0..100 {
        let foo = Box::new(i);
        test::black_box(foo);
    }
}

The assembly clearly contains calls to exchange_malloc, and running the code under gdb indicates that exchange_malloc is called 100 times:

(gdb) break _ZN4heap15exchange_malloc20he8623aa1084055ffJfaE
Breakpoint 1 at 0x4a90: file ../src/liballoc/heap.rs, line 123.
(gdb) ignore 1 1000
Will ignore next 1000 crossings of breakpoint 1.
(gdb) run
Starting program: /home/andrew/foo 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Inferior 1 (process 1890) exited normally]
(gdb) info breakpoints
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   0x0000555555558a90 in foo::heap::exchange_malloc at ../src/liballoc/heap.rs:123
    breakpoint already hit 100 times
    ignore next 900 hits

[apasel422]

But Valgrind reports:

==1899== Memcheck, a memory error detector
==1899== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==1899== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==1899== Command: ./foo
==1899== 
==1899== 
==1899== HEAP SUMMARY:
==1899==     in use at exit: 0 bytes in 0 blocks
==1899==   total heap usage: 5 allocs, 5 frees, 976 bytes allocated
==1899== 
==1899== All heap blocks were freed -- no leaks are possible
==1899== 
==1899== For counts of detected and suppressed errors, rerun with: -v
==1899== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

[alexcrichton]

Hm something fishy may indeed be going on here, it looks like valgrind is no longer tracking jemalloc? If you use alloc_system for now you'll get accurate tracking via valgrind, I'll reopen for more investigation into jemalloc though.

(also tweaking the title a bit)

[apasel422]

Using alloc_system indeed causes valgrind to report the correct information.

[shepmaster]

Sorry to jump in here, but how did you ever get valgrind to report jemalloc stats in the first place? On OS X, I've never successfully gotten any useful stats.

[apasel422]

I think the issue is that Valgrind can't track calls to jemalloc's mallocx-family of functions. There is a way to get Valgrind to track them, but it involves some custom headers: http://valgrind.org/docs/manual/mc-manual.html#mc-manual.mempools

[apasel422]

See also https://github.com/jemalloc/jemalloc/blob/master/include/jemalloc/internal/valgrind.h.

[apasel422]

jemalloc is apparently not being built with --enable-valgrind (which makes sense for performance reasons):

foo.c:

#include <stdio.h>
#include <stdlib.h>

extern void je_mallctl(const char *, void *, size_t *, void *, size_t);

int main() {
    int valgrind = 6;
    size_t len = 1;
    je_mallctl("config.valgrind", &valgrind, &len, NULL, 0);
    printf("%d\n", valgrind);
}

> gcc foo.c -L/usr/local/lib/rustlib/x86_64-unknown-linux-gnu/lib/ -l:liballoc_jemalloc-35017696.rlib -lpthread
> ./a.out 
0

[apasel422]

Building nightly with

JEMALLOC_FLAGS='--enable-valgrind' make

results in the correct Valgrind behavior.

Not sure how we want to address this, but providing multiple versions of liballoc_jemalloc seems like a possibility.

[shepmaster]

FWIW, I ran your example program with rustc 1.5.0-dev (5f06607e2 2015-09-29) and valgrind-3.11.0.SVN on OS X, and it seems to track allocations:

==28285== HEAP SUMMARY:
==28285==     in use at exit: 35,221 bytes in 426 blocks
==28285==   total heap usage: 611 allocs, 185 frees, 42,405 bytes allocated

To my knowledge, I have not specified any custom flags when building. I also see that massif reports heap allocations over time, so I have some hope that things are just magically working together now!

[apasel422]

Valgrind on Ubuntu still indicates that this code, compiled with opt-level=0 on nightly, has only 5 allocations:

#![feature(alloc_jemalloc)]
#![feature(test)]

extern crate alloc_jemalloc;
extern crate test;

fn main() {
    for i in 0..100 {
        let foo = Box::new(i);
        test::black_box(foo);
    }
}

Removing the allocator has the same effect, while changing to alloc_system reports 112 allocations.

[Hansyperman]

I think I reported this same issue on reddit: rust 1.8 stable lost valgrind support
https://www.reddit.com/r/rust/comments/4fddre/i_lost_valgrind_in_rust_18/

I have a mixed rust/extern "C" program. I know it leaks memory and use valgrind to verify behaviour. When I upgraded to rust stable 1.8, it proudly claims not to leak anymore, and even stops allocating memory altogether! While I'm impressed by rusts memory safety guarantees, this is a bit too much for me ;-)

Anyone knows what happens and how to make it un-happen? I read somewhere they integrated jemalloc, so maybe that broke the monitoring allocator from valgrind?

An example on debian 8.4, x86-64:

extern "C" {
  fn malloc(len: usize) -> *mut u8;
}

fn main() {
  println!("Allocated at {:?}",unsafe{malloc(100)});
}

On rust 1.8:

me@aruba:~/atest$ rustc --version
rustc 1.8.0 (db2939409 2016-04-11)
me@aruba:~/atest$ cargo build
me@aruba:~/atest$ valgrind  target/debug/atest
==2921== Memcheck, a memory error detector
==2921== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==2921== Using Valgrind-3.10.0 and LibVEX; rerun with -h for copyright info
==2921== Command: target/debug/atest
==2921== 
Allocated at 0x6035000
==2921== 
==2921== HEAP SUMMARY:
==2921==     in use at exit: 0 bytes in 0 blocks
==2921==   total heap usage: 0 allocs, 0 frees, 0 bytes allocated
==2921== 
==2921== All heap blocks were freed -- no leaks are possible
==2921== 
==2921== For counts of detected and suppressed errors, rerun with: -v
==2921== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

On rust 1.7

me@aruba:~/atest$ rustc --version
rustc 1.7.0 (a5d1e7a59 2016-02-29)
me@aruba:~/proggies/atest$ cargo build
Compiling atest v0.1.0 (file:///home/hans/proggies/atest)
me@aruba:~/atest$ valgrind  target/debug/atest
==5975== Memcheck, a memory error detector
==5975== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==5975== Using Valgrind-3.10.0 and LibVEX; rerun with -h for copyright info
==5975== Command: target/debug/atest
==5975== 
Allocated at 0x5e00570
==5975== 
==5975== HEAP SUMMARY:
==5975==     in use at exit: 100 bytes in 1 blocks
==5975==   total heap usage: 7 allocs, 6 frees, 1,124 bytes allocated
==5975== 
==5975== LEAK SUMMARY:
==5975==    definitely lost: 100 bytes in 1 blocks
==5975==    indirectly lost: 0 bytes in 0 blocks
==5975==      possibly lost: 0 bytes in 0 blocks
==5975==    still reachable: 0 bytes in 0 blocks
==5975==         suppressed: 0 bytes in 0 blocks
==5975== Rerun with --leak-check=full to see details of leaked memory
==5975== 
==5975== For counts of detected and suppressed errors, rerun with: -v
==5975== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

[alexcrichton]

The tools team discussed this issue during triage yesterday and the conclusion was that while not high priority we would very much like to see this issue fixed! The --enable-valgrind option mentioned by @apasel422 above seems like the best fix here, and we'd just want to investigate if that has any runtime impact or affect on runtime dependencies.

[michaelwoerister]

It seems that Valgrind support in jemalloc is going away:
jemalloc/jemalloc#369

[lilith]

I have a mixed C/Rust project targeting windows, linux, & mac. Lots of codecs are involved, and valgrind is an essential tool for verifying that integration tests aren't exposing a new bug in libpng or another dependency (this is a very common occurrence).

Does this change mean that I should move all of my integration tests to C++? Or is a custom build of rust required?

[ssokolow]

I believe you just need to ask nightly rust to build your project with the system allocator rather than the bundled jemalloc.

[shepmaster]

Or if you produce a shared library, it will use the system allocator instead of jemalloc by default, which should allow valgrind to work.

[lilith]

@ssokolow You mean the 'official' nightly build would do this? What flags would I provide? (Building nightly from source means I can't do this in Travis, which is where I valgrind as part of CI).

[shepmaster]

The documentation covers specifying allocators.

[lilith]

Thank you! The "--enable-valgrind" context led me to misunderstand @ssokolow.

So to clarify, there's no compilation flag that can change this; I would need to inject source code #[feature(alloc_system)] extern crate alloc_system; into my tests, but only as a permutation for the nightly channel?

[ssokolow]

Yes. As I understand the situation, compiling jemalloc with --enable-valgrind either is or was a temporary measure before support will be/was dropped entirely.

[lilith]

There is the problem that I'm running valgrind against a totally different executable, compiled by nightly instead of stable. Essentially this means I can't valgrind products of a stable rust build.

[ssokolow]

@nathanaeljones I'm no expert, but you could look into making a custom rust build with an older jemalloc and/or --enable-valgrind (as appropriate) while you wait for alloc_system support to be in stable.

[lilith]

@ssokolow Thanks! For now it looks like nightly will be a requirement for other reasons anyway :)

Valgrind has been crucial for my work with unsafe Rust. As far as I can tell there's no way to make the generated test harnesses Valgrind-friendly; a standalone executable is required.

[ssokolow]

As a quick update, since this is getting linked to, here's how I make the system allocator opt-in for nightly:

First, add these lines to the top of main.rs:

#![cfg_attr(feature="nightly", feature(alloc_system))]
#[cfg(feature="nightly")]
extern crate alloc_system;

Then, add these to Cargo.toml...

[features]
nightly = []

And, finally, use these commands for setup and building:

rustup toolchain add nightly-x86_64-unknown-linux-gnu
cargo +nightly build --features=nightly

For records purposes:

• I pieced that together for an automation "script" I haven't yet released, where I want to crunch down the size of a musl build.
• I first posted those instructions here on reddit.
• I later cleaned up the reusable bits (so far) of the project I'd developed them in and published this CLI project boilerplate, which incorporates that code and splits it between the just install-rustup-deps and just channel=nightly build-release commands.

[alexcrichton]

We're unlikely to really do anything to fix this, so I'm going to close this in favor of #27389. It's highly likely that all programs will start to link to jemalloc the system allocator by default once we stabilize that feature.