Stop compressing metadata

[brson]

Decompressing metadata takes time. I have a branch that turned off compression but it was causing dynamic linker crashes.

[brson]

cc #8752

[alexcrichton]

Closing, this is done. Rlibs don't have compressed metadata, but dylibs still do (due to linker restrictions)

We prefer reading rlibs, so we're getting all the benefits anyway.

[eddyb]

What was the perf gain from not compressing metadata?

I am trying to create a dynamically linked snapshot and I would rather not increase its size significantly.

Initial increase was from 49MB to 108MB which is unacceptable IMO. Disabling this rather wasteful functionality brought it down to 96MB.
On top of not having field names output in RBML (which that DEBUG is controlling), compressing libcore's 14MB rust.metadata.bin with gzip gives me 2MB, while xz takes it to 1.1MB, though adding xz support to the compiler is not exactly trivial.

I understand compression can add to the compilation time, but I think it's faster without field names and I only really need it turned on for snapshots.

@alexcrichton Any ideas? Perhaps I should open another issue for shrinking binary sizes in general, I believe #14527 might help with that - also, bundling all the crates behind the std facade within a dynamically linked std and something similar for librustc_llvm - but that's really offtopic by now.

[eddyb]

Turns out I completely forgot the snapshot is compressed, and with xz -9 -e (thanks @thestinger), I get 26MB for the dynamically linked snapshot and 12MB for the old one.
Even just gz is less than twice that, so I'm less worried about it (for xz support we would need this python library).

[alexcrichton]

I'm a bit hazy on the details at this point, so I decided to re-run the numbers:

extern crate alloc;
extern crate arena;
extern crate collections;
extern crate core;
extern crate debug;
extern crate flate;
extern crate getopts;
extern crate glob;
extern crate graphviz;
extern crate green;
extern crate libc;
extern crate log;
extern crate native;
extern crate num;
extern crate rand;
extern crate rbml;
extern crate regex;
extern crate rlibc;
extern crate rustrt;
extern crate semver;
extern crate serialize;
extern crate sync;
extern crate term;
extern crate test;
extern crate time;
extern crate unicode;
extern crate url;
#[cfg(so)] extern crate rustc;
#[cfg(so)] extern crate rustdoc;
#[cfg(so)] extern crate uuid;

fn main() {}

Reading rlib metadata

$ RUST_LOG=rustc::metadata::loader rustc foo.rs -Lrlib -Lcommon 2>&1 | grep 'reading.*=>'  
INFO:rustc::metadata::loader: reading libstd-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading liballoc-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading libcore-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading liblibc-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading libunicode-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading libcollections-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading librand-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading libsync-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading librustrt-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading libnative-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading libarena-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading libdebug-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading libflate-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading libgetopts-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading libglob-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading libgraphviz-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading libgreen-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading liblog-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading libregex-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading libnum-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading librbml-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading libserialize-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading librlibc-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading libsemver-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading libterm-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading libtest-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading libtime-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading liburl-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading libuuid-4e7c5e5c.rlib => 0ms

Reading dylib metadata

$ RUST_LOG=rustc::metadata::loader rustc foo.rs --cfg so -Lso -Lcommon 2>&1 | grep 'reading.*=>'
INFO:rustc::metadata::loader: reading libstd-4e7c5e5c.so => 16ms
INFO:rustc::metadata::loader: reading liballoc-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading libcore-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading liblibc-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading libunicode-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading libcollections-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading librand-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading libsync-4e7c5e5c.so => 5ms
INFO:rustc::metadata::loader: reading librustrt-4e7c5e5c.so => 2ms
INFO:rustc::metadata::loader: reading libnative-4e7c5e5c.so => 1ms
INFO:rustc::metadata::loader: reading libarena-4e7c5e5c.so => 0ms
INFO:rustc::metadata::loader: reading libdebug-4e7c5e5c.so => 0ms
INFO:rustc::metadata::loader: reading libflate-4e7c5e5c.so => 0ms
INFO:rustc::metadata::loader: reading libgetopts-4e7c5e5c.so => 0ms
INFO:rustc::metadata::loader: reading libglob-4e7c5e5c.so => 1ms
INFO:rustc::metadata::loader: reading libgraphviz-4e7c5e5c.so => 0ms
INFO:rustc::metadata::loader: reading libgreen-4e7c5e5c.so => 0ms
INFO:rustc::metadata::loader: reading liblog-4e7c5e5c.so => 0ms
INFO:rustc::metadata::loader: reading libregex-4e7c5e5c.so => 1ms
INFO:rustc::metadata::loader: reading libregex_macros-4e7c5e5c.so => 0ms
INFO:rustc::metadata::loader: reading libnum-4e7c5e5c.so => 3ms
INFO:rustc::metadata::loader: reading librbml-4e7c5e5c.so => 1ms
INFO:rustc::metadata::loader: reading libserialize-4e7c5e5c.so => 6ms
INFO:rustc::metadata::loader: reading librlibc-4e7c5e5c.rlib => 0ms
INFO:rustc::metadata::loader: reading libsemver-4e7c5e5c.so => 0ms
INFO:rustc::metadata::loader: reading libterm-4e7c5e5c.so => 0ms
INFO:rustc::metadata::loader: reading libtest-4e7c5e5c.so => 2ms
INFO:rustc::metadata::loader: reading libtime-4e7c5e5c.so => 0ms
INFO:rustc::metadata::loader: reading liburl-4e7c5e5c.so => 0ms
INFO:rustc::metadata::loader: reading librustc_llvm-4e7c5e5c.so => 1ms
INFO:rustc::metadata::loader: reading librustc_back-4e7c5e5c.so => 0ms
INFO:rustc::metadata::loader: reading librustc-4e7c5e5c.so => 43ms
INFO:rustc::metadata::loader: reading libsyntax-4e7c5e5c.so => 34ms
INFO:rustc::metadata::loader: reading libfmt_macros-4e7c5e5c.so => 0ms
INFO:rustc::metadata::loader: reading librustc_llvm-4e7c5e5c.so => 1ms
INFO:rustc::metadata::loader: reading librustc_back-4e7c5e5c.so => 0ms
INFO:rustc::metadata::loader: reading librustdoc-4e7c5e5c.so => 10ms
INFO:rustc::metadata::loader: reading libuuid-4e7c5e5c.so => 0ms

Note that reading compressed metadata adds an extra 22ms to all rust compilations, and as we continue to proliferate libraries through cargo I suspect that this will only get worse. The benefit of an uncompressed metadata file is that it can be mmap'd directly by LLVM which is basically an instantaneous operation (as you can see).

I had no idea we had some sort of "debug" flag turned on for rbml, we should definitely turn that off if it's just bloating metadata!

Also, as you seem to have figured out, metadata is super compressable, so I would only worry about the gzip sizes. I wouldn't worry too much about the distribution size of the snapshot as it's not like it's downloaded on every rust compile!