Constantly receiving Undefined symbols for architecture x86_64

[jkelleyrtp]

I'm constantly receiving Undefined symbols for architecture x86_64 when running cargo build/test/check/run. It happens in projects with a variety of targets (wasm/arm/x86/musl). I can make it temporarily go away when switching my default from stable to nightly, but the issues will resume and I'll have to switch back.

The stack traces I'm receiving are especially nasty:

  = note: Undefined symbols for architecture x86_64:
            "syn::punctuated::Pair$LT$T$C$P$GT$::into_tuple::h7650409a1648a723", referenced from:
                syn::gen::visit::visit_angle_bracketed_generic_arguments::h17c66b0edf1406c7 in libwasm_bindgen_macro_support-62eb06893cdd5d5c.rlib(wasm_bindgen_macro_support-62eb06893cdd5d5c.wasm_bindgen_macro_support.6s2lo5z5-cgu.0.rcgu.o)
            "syn::punctuated::Pair$LT$T$C$P$GT$::into_tuple::h9e3a72cf01e3775d", referenced from:
                syn::gen::visit::visit_lifetime_def::h12b1c6ddc1818ba7 in libwasm_bindgen_macro_support-62eb06893cdd5d5c.rlib(wasm_bindgen_macro_support-62eb06893cdd5d5c.wasm_bindgen_macro_support.6s2lo5z5-cgu.0.rcgu.o)
                syn::gen::visit::visit_predicate_lifetime::h90dd0a0408d5ad94 in libwasm_bindgen_macro_support-62eb06893cdd5d5c.rlib(wasm_bindgen_macro_support-62eb06893cdd5d5c.wasm_bindgen_macro_support.6s2lo5z5-cgu.0.rcgu.o)
....
....
"_$LT$alloc..raw_vec..RawVec$LT$T$C$A$GT$$u20$as$u20$core..ops..drop..Drop$GT$::drop::hdf6534d4c8747194", referenced from:
                core::ptr::drop_in_place::h3e5e73dbe7eaf673 in libwasm_bindgen_macro_support-62eb06893cdd5d5c.rlib(wasm_bindgen_macro_support-62eb06893cdd5d5c.wasm_bindgen_macro_support.6s2lo5z5-cgu.6.rcgu.o)
          ld: symbol(s) not found for architecture x86_64
          clang: error: linker command failed with exit code 1 (use -v to see invocation)
         

Unfortunately, cargo clean doesn't help, nor does switching from debug to release.

[jyn514]

This looks like it might be caused by wasm-bindgen. Have you asked those maintainers whether they recognize the issue?

[Aaron1011]

I think this is actually an incremental compilation bug.

@jkelleyrtp: What versions of Rust are you using?

[jkelleyrtp]

I think this is actually an incremental compilation bug.

@jkelleyrtp: What versions of Rust are you using?

@jyn514 not a wasm bug, this happens with all sorts of crates. In this case, it just so happens that wasm-bindgen got caught by the issue.

projects with a variety of targets (wasm/arm/x86/musl)

@Aaron1011

I think it has to do with incremental compilation. Often times I close an editor or Rust Analyzer locks the build directory, and I could imagine that it might crash rustc while building.

My stable is:

❯ rustc --version
rustc 1.48.0 (7eac88abb 2020-11-16)

My nightly is

❯ rustc +nightly --version
rustc 1.50.0-nightly (1c389ffef 2020-11-24)

If rustc gets killed while compiling, would it corrupt build artifacts? Though, I'm not entirely sure, because running cargo clean (which I assumes kills build artifacts) doesn't fix the problem.

[Aaron1011]

The nightly build that you're using includes #76859. I had hoped that that PR would solve this kind of issue once and for all - unfortunately, that doesn't seem to be the case.

@jkelleyrtp Can you share one of the projects that this happens with? It would be extremely helpful if you could provide precise reproduction instructions (e.g. do you need to modify any files)?

[tgnottingham]

I've been hitting spurious errors while building rustc itself. They're different from yours (more along the lines seen in #67802), but there are some commonalities. Incremental seems to play a role, ./x.py clean doesn't help, and depending on what I'm doing in my editor, I think it starts/stops rust-analyzer instances while I'm compiling.

[jyn514]

@tgnottingham can you paste the full errors you're running into?

[tgnottingham]

/home/tgnottingham/Projects/rust/rust/library/core/src/option.rs:386: undefined reference to `anon.04b56f81f8d51f30b8720ce746e686ac.0.llvm.4938662292890509205'
/usr/bin/ld: /tmp/rustcTyeVKl/librustc_metadata-4849b55e661cd891.rlib(rustc_metadata-4849b55e661cd891.2rvbnxcl0w6plmot.rcgu.o): relocation R_X86_64_PC32 against undefined hidden symbol `anon.04b56f81f8d51f30b8720ce746e686ac.0.llvm.4938662292890509205' can not be used when making a shared object
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status

They're varied, but generally along those lines. And they're different from @jkelleyrtp's, as I said, so maybe totally unrelated (apologies for derailment).

[jyn514]

Heh, I just ran into this myself:

  = note: /usr/bin/ld: /tmp/rustcQ1odj9/librustc_middle-b63a4b3fe5a6b279.rlib(rustc_middle-b63a4b3fe5a6b279.1bp2u94sew41xpja.rcgu.o): relocation R_X86_64_PC32 against undefined hidden symbol `anon.e4eedcdcf9dce0f77011b23711e1662a.196.llvm.162878321601332620' can not be used when making a shared object
          /usr/bin/ld: /tmp/rustcQ1odj9/librustc_middle-b63a4b3fe5a6b279.rlib(rustc_middle-b63a4b3fe5a6b279.1k24h0loog84q6w.rcgu.o): relocation R_X86_64_PC32 against undefined hidden symbol `anon.e4eedcdcf9dce0f77011b23711e1662a.196.llvm.162878321601332620' can not be used when making a shared object
          /usr/bin/ld: /tmp/rustcQ1odj9/librustc_middle-b63a4b3fe5a6b279.rlib(rustc_middle-b63a4b3fe5a6b279.2trl2z5frku3xuzy.rcgu.o): relocation R_X86_64_PC32 against undefined hidden symbol `anon.e4eedcdcf9dce0f77011b23711e1662a.196.llvm.162878321601332620' can not be used when making a shared object
          /usr/bin/ld: /tmp/rustcQ1odj9/librustc_middle-b63a4b3fe5a6b279.rlib(rustc_middle-b63a4b3fe5a6b279.31arrfea00oazgke.rcgu.o): relocation R_X86_64_PC32 against undefined hidden symbol `anon.e4eedcdcf9dce0f77011b23711e1662a.393.llvm.162878321601332620' can not be used when making a shared object
          /usr/bin/ld: /tmp/rustcQ1odj9/librustc_middle-b63a4b3fe5a6b279.rlib(rustc_middle-b63a4b3fe5a6b279.38hexmempyfoakzt.rcgu.o): relocation R_X86_64_PC32 against undefined hidden symbol `anon.e4eedcdcf9dce0f77011b23711e1662a.393.llvm.162878321601332620' can not be used when making a shared object
          /usr/bin/ld: final link failed: nonrepresentable section on output
          clang: error: linker command failed with exit code 1 (use -v to see invocation)

and now I think I know what's going wrong - @tgnottingham do you have an IDE set up to run x.py check in the background? I think it's conflicting with x.py build because there's no build lock: #76661.

[tgnottingham]

@jyn514, I don't think anything is running x.py check in the background in my case. I just have a vim plugin that invokes rust-analyzer.

[jyn514]

@tgnottingham rust analyzer invokes either cargo check or x.py depending on how you set it up: https://rustc-dev-guide.rust-lang.org/building/suggested.html?highlight=Rust-analyzer#configuring-rust-analyzer-for-rustc. So I expect that's what's going wrong.

Do you want to move to zulip so we're not spamming the issue? I think this is different from the wasm error.

[Aaron1011]

I just ran into this. I'm not running anything in the background, so this is definitely a rustc or LLVM bug.

[emeryc]

I just ran into this issue as well, building from scratch, no background build processes happening.

Is there any information I can give to help with this?

[jkelleyrtp]

Found a clue... I turned on rpath = true for a project and then tried disabling it. Clearing target and registry files are unable to recover the project. Using +nightly doesn't have the same issues.

[darnuria]

In case it helps somebody I encountered the issue in a build at work on CI using rustlang:nightly-[debname] images.

  = note: /usr/bin/ld: /usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libprofiler_builtins-50086f81dc2d4d6f.rlib(InstrProfilingPlatformLinux.o): relocation R_X86_64_PC32 against undefined hidden symbol `__ehdr_start' can not be used when making a shared object
          /usr/bin/ld: final link failed: nonrepresentable section on output
          collect2: error: ld returned 1 exit status

Plateforms impacted (for us)

• rustlang:nightly-buster -> KO
• rustlang:nightly-bulleyes -> OK

What we used

We used grcov with configuration for gitlab CI inspired from the README.md of grcov.

So we configure something like this:

Setup

export CARGO_INCREMENTAL=0
export RUSTFLAGS="-Zprofile -Cinstrument-coverage -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zpanic_abort_tests -Cpanic=abort"
export RUSTDOCFLAGS="-Cpanic=abort"
rustup component add llvm-tools-preview
info: downloading component 'llvm-tools'
info: installing component 'llvm-tools'
cargo install --version 0.8.18 grcov

The fail occurs while building grcov on thiserror-impl and serde_derive linking (see stacktrace)

stacktrace

error: linking with `cc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/usr/local/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" VSLANG="1033" "cc" "-Wl,--version-script=/tmp/rustco1AkJk/list" "-Wl,--no-undefined-version" "-m64" "/tmp/rustco1AkJk/symbols.o" "/tmp/cargo-installR9oiqm/release/deps/thiserror_impl-4f744515f9cb4819.thiserror_impl.ac9971b9d8603b5c-cgu.0.rcgu.o" "/tmp/cargo-installR9oiqm/release/deps/thiserror_impl-4f744515f9cb4819.461oo03yg4smhdm9.rcgu.rmeta" "/tmp/cargo-installR9oiqm/release/deps/thiserror_impl-4f744515f9cb4819.2z5rthv354yodzxf.rcgu.o" "-Wl,--as-needed" "-L" "/tmp/cargo-installR9oiqm/release/deps" "-L" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/tmp/cargo-installR9oiqm/release/deps/libsyn-4077fd9350529457.rlib" "/tmp/cargo-installR9oiqm/release/deps/libquote-e42fdfbb74cb4817.rlib" "/tmp/cargo-installR9oiqm/release/deps/libproc_macro2-6ba8af40d919ba07.rlib" "/tmp/cargo-installR9oiqm/release/deps/libunicode_ident-a505ddbb66acf82e.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libprofiler_builtins-50086f81dc2d4d6f.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libproc_macro-b02085de848d0686.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-19964a87984cce8a.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_abort-27be7e902ae40178.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-37a5cbc9c5c20c04.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libmemchr-e8cedd2960ff146a.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-5deb081ad96d9130.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-22c964fd8f2dfaf9.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-6971442fd1c2cfbb.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-ec7906ee77b1d540.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-4307a3ebc7f0bfcf.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-c2ab156d57b93da1.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-763bbb8717f828b7.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-eb9ba65f17380c5d.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-4286d0bf3addfc05.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-fa17a4d002f91ad9.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-ff4ae84067796be3.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-6f47c72c85a10103.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-cd2f2bc505f56f50.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-ec02dd343723da85.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-4df0073ab5846ddb.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/tmp/cargo-installR9oiqm/release/deps/libthiserror_impl-4f744515f9cb4819.so" "-shared" "-Wl,-z,relro,-z,now" "-nodefaultlibs" "-u" "__llvm_profile_runtime"
  = note: /usr/bin/ld: /usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libprofiler_builtins-50086f81dc2d4d6f.rlib(InstrProfilingPlatformLinux.o): relocation R_X86_64_PC32 against undefined hidden symbol `__ehdr_start' can not be used when making a shared object
          /usr/bin/ld: final link failed: nonrepresentable section on output
          collect2: error: ld returned 1 exit status
          
error: could not compile `thiserror-impl` (lib) due to previous error
warning: build failed, waiting for other jobs to finish...
error: linking with `cc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/usr/local/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" VSLANG="1033" "cc" "-Wl,--version-script=/tmp/rustctpHzqt/list" "-Wl,--no-undefined-version" "-m64" "/tmp/rustctpHzqt/symbols.o" "/tmp/cargo-installR9oiqm/release/deps/serde_derive-f0beeb3c8c1c4dec.serde_derive.b21f8c48fc810b11-cgu.0.rcgu.o" "/tmp/cargo-installR9oiqm/release/deps/serde_derive-f0beeb3c8c1c4dec.2fxq5lwx6clvk4ro.rcgu.rmeta" "/tmp/cargo-installR9oiqm/release/deps/serde_derive-f0beeb3c8c1c4dec.4hkmz16s2bgqz7ni.rcgu.o" "-Wl,--as-needed" "-L" "/tmp/cargo-installR9oiqm/release/deps" "-L" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/tmp/cargo-installR9oiqm/release/deps/libsyn-4077fd9350529457.rlib" "/tmp/cargo-installR9oiqm/release/deps/libquote-e42fdfbb74cb4817.rlib" "/tmp/cargo-installR9oiqm/release/deps/libproc_macro2-6ba8af40d919ba07.rlib" "/tmp/cargo-installR9oiqm/release/deps/libunicode_ident-a505ddbb66acf82e.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libprofiler_builtins-50086f81dc2d4d6f.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libproc_macro-b02085de848d0686.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-19964a87984cce8a.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_abort-27be7e902ae40178.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-37a5cbc9c5c20c04.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libmemchr-e8cedd2960ff146a.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-5deb081ad96d9130.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-22c964fd8f2dfaf9.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-6971442fd1c2cfbb.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-ec7906ee77b1d540.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-4307a3ebc7f0bfcf.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-c2ab156d57b93da1.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-763bbb8717f828b7.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-eb9ba65f17380c5d.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-4286d0bf3addfc05.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-fa17a4d002f91ad9.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-ff4ae84067796be3.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-6f47c72c85a10103.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-cd2f2bc505f56f50.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-ec02dd343723da85.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-4df0073ab5846ddb.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/tmp/cargo-installR9oiqm/release/deps/libserde_derive-f0beeb3c8c1c4dec.so" "-shared" "-Wl,-z,relro,-z,now" "-nodefaultlibs" "-u" "__llvm_profile_runtime"
  = note: /usr/bin/ld: /usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libprofiler_builtins-50086f81dc2d4d6f.rlib(InstrProfilingPlatformLinux.o): relocation R_X86_64_PC32 against undefined hidden symbol `__ehdr_start' can not be used when making a shared object
          /usr/bin/ld: final link failed: nonrepresentable section on output
          collect2: error: ld returned 1 exit status
          
error: could not compile `serde_derive` (lib) due to previous error
error: failed to compile `grcov v0.8.18`, intermediate artifacts can be found at `/tmp/cargo-installR9oiqm`.
To reuse those artifacts with a future compilation, set the environment variable `CARGO_TARGET_DIR` to that path.