Rustc does not warn when .so/.rlib files are not in sync

[alexcrichton]

Right now the logic for reading metadata for a crate is to read it the first time the crate's dylib or rlib file is found. On the second file that's found, the metadata isn't read at all because metadata reading is one of the known slow portions of the compiler.

This implies that if an rlib file is regenerated, but the dylib isn't, then the two are out of sync and rustc will read the old metadata from the dylib and link to the rlib, resulting in linker errors.

I don't want to slow down rustc by reading metadata from all rlibs and dylibs, but this situation is far less than optimal. Perhaps this is a job for a rustpkg-like tool?

[emberian]

@alexcrichton is this still relevant? we don't do this checking, but the metadata isn't pulled from the dylib.

[alexcrichton]

This has an interesting development:

$ RUST_LOG=rustc::metadata::loader=info rustc foo.rs 2>&1 | grep '^reading'
reading libstd-aad93cea-0.11-pre.rlib => 0ms
reading libstd-aad93cea-0.11-pre.so => 47ms
reading liblibc-8f65d5a8-0.10-pre.rlib => 0ms
reading liblibc-8f65d5a8-0.10-pre.so => 0ms
reading libstd-aad93cea-0.11-pre.rlib => 0ms
reading libstd-aad93cea-0.11-pre.so => 46ms
reading liblibc-8f65d5a8-0.10-pre.rlib => 0ms
reading liblibc-8f65d5a8-0.10-pre.so => 0ms
reading libnative-83574243-0.11-pre.rlib => 0ms
reading libnative-83574243-0.11-pre.so => 1ms

This issue is supposed to be relevant, but it looks like we're reading metadata of rlibs and dylibs. This is not necessary, and it looks like we could shave ~100ms off compiling hello world if it were taken care of.

In short, I'd like to keep this issue open.

[reem]

Triage: I see that some progress has been made on this issue in the above commits. Does that re-enable this issue?

[steveklabnik]

$ RUST_LOG=rustc::metadata::loader=info rustc hello.rs 2>&1 | grep reading
INFO:rustc::metadata::loader: rlib reading metadata from: /home/steve/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-4e7c5e5c.rlib
INFO:rustc::metadata::loader: reading libstd-4e7c5e5c.rlib => 0ms

just rlibs now, apparently. I'm giving this a close, @alexcrichton , feel free to reopen.

[alexcrichton]

I think this is still an issue (reopening)

In this test case below a blank file is compiled to an rlib/dylib pair, then the rlib is recompiled with a new function. A later executable is then linked against the dylib (using the rlib metadata), succeeding all the way until the linker actually fails.

$ touch foo.rs
$ rustc --crate-type dylib,rlib foo.rs
$ ls
foo.rs       libfoo.dylib libfoo.rlib
$ echo 'pub fn foo() {}' > foo.rs
$ rustc --crate-type rlib foo.rs
$ echo 'extern crate foo; fn main() { foo::foo(); }' > bar.rs
$ rustc bar.rs -L. -C prefer-dynamic
error: linking with `cc` failed: exit code: 1
note: cc '-m64' '-L' '/Users/acrichton/code/rust/lib/rustlib/x86_64-apple-darwin/lib' '-o' 'bar' 'bar.o' '-Wl,-force_load,/Users/acrichton/code/rust/lib/rustlib/x86_64-apple-darwin/lib/libmorestack.a' '-Wl,-dead_strip' '-nodefaultlibs' '-L' '/Users/acrichton/test' '-lfoo' '-L' '/Users/acrichton/code/rust/lib/rustlib/x86_64-apple-darwin/lib' '-lstd-4e7c5e5c' '-L' '.' '-L' '/Users/acrichton/code/rust/lib/rustlib/x86_64-apple-darwin/lib' '-L' '/Users/acrichton/test/.rust/lib/x86_64-apple-darwin' '-L' '/Users/acrichton/test/lib/x86_64-apple-darwin' '-L' '/Users/acrichton/.rust/lib/x86_64-apple-darwin' '-lSystem' '-lpthread' '-lc' '-lm' '-lcompiler-rt'
note: ld: warning: directory not found for option '-L/Users/acrichton/test/.rust/lib/x86_64-apple-darwin'
ld: warning: directory not found for option '-L/Users/acrichton/test/lib/x86_64-apple-darwin'
ld: warning: directory not found for option '-L/Users/acrichton/.rust/lib/x86_64-apple-darwin'
Undefined symbols for architecture x86_64:
  "foo::ha37403d109b4e734eaa", referenced from:
      main::h7e1981c1a43d0b7afaa in bar.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

error: aborting due to previous error

[steveklabnik]

Ahh, I forgot the crate-types. :(

[steveklabnik]

Triage: the error is slightly different:

error: linking with `cc` failed: exit code: 1
note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/home/steve/.multirust/toolchains/nightly/lib/rustlib/x86_64-unknown-linux-gnu/lib" "bar.0.o" "-o" "bar" "-Wl,--gc-sections" "-pie" "-nodefaultlibs" "-L" "." "-L" "/home/steve/.multirust/toolchains/nightly/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "-Wl,-Bdynamic" "-L" "/home/steve/tmp" "-l" "foo" "-L" "/home/steve/.multirust/toolchains/nightly/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "std-fd663c41" "-l" "dl" "-l" "pthread" "-l" "gcc_s" "-l" "pthread" "-l" "c" "-l" "m" "-l" "rt" "-l" "compiler-rt"
note: bar.0.o: In function `main::h7afac322a1b32d81faa':
bar.0.rs:(.text._ZN4main20h7afac322a1b32d81faaE+0x2): undefined reference to `foo::hfe26232f3a6cc9a1eaa'
collect2: error: ld returned 1 exit status

error: aborting due to previous error

[jonas-schievink]

Triage: This still happens, and seems like a bug with metadata loading and linkage, rather than just a diagnostics issue.

The compiler should not load metadata from one file and then link against another while assuming its contents match.

[gilescope]

Would #75594 help here? If we use libcrate_name-svh.rmeta as the metadata filename then we could check if the metadata was up to date without having to read more than the filename in the archive.

[tmandry]

I just got bit by this, except instead of a Rust rlib and dylib, the compiler found a Rust rlib and a .so file of the same name compiled by clang. (The Rust library wraps the C++ one which is why they have the same name.)

It then proceeded as if the .so was the dynamic version of the Rust library, leading to a linker error. Definitely confusing.