Rust binaries require a too recent glibc

[konstin]

When rustc builds on a system with a recent glibc version, the binaries require a recent glibc version (2.18). When built on an old system however, they will only require glibc 2.3.

To reproduce we can use a simple hello world example, building e.g. on Ubuntu 18.04:

cargo new hello-world
cd hello-world
cargo build --release

Let's look at the required symbols, filtering out glibc 2.2 and 2.3:

$ readelf -Ws target/release/hello-world | grep GLIBC | grep -v GLIBC_2.[23]
    11: 0000000000000000     0 FUNC    WEAK   DEFAULT  UND __cxa_thread_atexit_impl@GLIBC_2.18 (5)
    47: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND memcpy@GLIBC_2.14 (12)
   537: 0000000000000000     0 FUNC    WEAK   DEFAULT  UND __cxa_thread_atexit_impl@@GLIBC_2.18
   644: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND memcpy@@GLIBC_2.14

We see that 2.14 and 2.18 seem to be used. We can confirm that those symbols are indeed required using a cent os 5 docker container:

$ docker run --rm -v $(pwd)/target:/root/target -it quay.io/pypa/manylinux1_x86_64 /root/target/release/hello-world
/root/target/release/hello-world: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /root/target/release/hello-world)
/root/target/release/hello-world: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /root/target/release/hello-world)

This happens even with cent os 7:

$ docker run --rm -it -v $(pwd):/io centos:7 /io/target/release/hello-world
/io/target/release/hello-world: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /io/target/release/hello-world)

When building inside the cent os 5 container on the other hand, none of the above symbols are used and the binary runs just fine in the container.

Why does this matter?

For my case, it's because I want to produce manylinux compliant libraries and binaries. manylinux is a policy defined by python designed to be compatible with virtual any used linux system, which requires that only symbols from glibc <= 2.5 are used. All packages in the python package index must follow that policy, even though the glibc part is currently not automatically enforced. This affects e.g. milksnake and pyo3. A newer requirement, called manylinux2010 is worked on, but it stills require glibc <= 2.12.

In general it means that you can't use any modern ci server to produce widely compatible linux-gnu binaries. You always have to build in some ancient docker container instead.

Versions

• rustc -vV: rustc 1.33.0-nightly (8e2063d02 2019-01-07)
• cargo -V: cargo 1.33.0-nightly (2cf1f5dda 2018-12-11)
• The quay.io/pypa/manylinux1_x86_64 docker container run cent os 5.11 (Final) with glibc 2.5
• The cent os 7 container is cent os 7.6.1810 with glibc 2.17

This issue a bit of a follow-up to #36826.

[sfackler]

AFAIK this is just how linking to glibc works - we talk to the linker the same way you would when building C code. If you want to target old glibc versions, you need to build against an old glibc version.

[konstin]

Afaik rust is manually linking __cxa_thread_atexit_impl. The relevant snippet even talks about backwards compatibility:

rust/src/libstd/sys/unix/fast_thread_local.rs

Lines 4 to 34 in 79d8a0f

	// Since what appears to be glibc 2.18 this symbol has been shipped which
	// GCC and clang both use to invoke destructors in thread_local globals, so
	// let's do the same!
	//
	// Note, however, that we run on lots older linuxes, as well as cross
	// compiling from a newer linux to an older linux, so we also have a
	// fallback implementation to use as well.
	//
	// Due to rust-lang/rust#18804, make sure this is not generic!
	#[cfg(any(target_os = "linux", target_os = "fuchsia", target_os = "hermit"))]
	pub unsafe fn register_dtor(t: *mut u8, dtor: unsafe extern fn(*mut u8)) {
	use libc;
	use mem;
	use sys_common::thread_local::register_dtor_fallback;
	extern {
	#[linkage = "extern_weak"]
	static __dso_handle: *mut u8;
	#[linkage = "extern_weak"]
	static __cxa_thread_atexit_impl: *const libc::c_void;
	}
	if !__cxa_thread_atexit_impl.is_null() {
	type F = unsafe extern fn(dtor: unsafe extern fn(*mut u8),
	arg: *mut u8,
	dso_handle: *mut u8) -> libc::c_int;
	mem::transmute::<*const libc::c_void, F>(__cxa_thread_atexit_impl)
	(dtor, t, &__dso_handle as *const _ as *mut _);
	return
	}
	register_dtor_fallback(t, dtor);
	}

[sfackler]

There's nothing particularly special about that import (other than being declared weak to support linkage against glibcs without it) - it's still going to be handled by the linker the same way that a reference to read or open or whatever would be.

[ishitatsuyuki]

I also disagree that this is something actionable on our side. Unlike Go, Rust depends on libc and the classic ABI problem needs to be handled just as C programs. And yes, you'll need to build things on an older distro, and that's what we do for rustup. See https://github.com/rust-lang/rust/tree/master/src/ci/docker/dist-x86_64-linux for a reference.

[konstin]

It's totally possible that rust can't do anything about that, but I'm a bit confused by the following: When I remove __cxa_thread_atexit_impl from std, I get binaries that work on cent os 7, even when I compile on ubuntu 18.04. As far as I've understood it, __cxa_thread_atexit_impl is an optional optimization, so wouldn't it be possible to only use that symbol when it's actually present on the platform running the binary, independent of the compiling platform?

diff --git a/src/libstd/sys/unix/fast_thread_local.rs b/src/libstd/sys/unix/fast_thread_local.rs
index d48d701dd5..27d3cae5f6 100644
--- a/src/libstd/sys/unix/fast_thread_local.rs
+++ b/src/libstd/sys/unix/fast_thread_local.rs
@@ -12,23 +12,11 @@
 // Due to rust-lang/rust#18804, make sure this is not generic!
 #[cfg(any(target_os = "linux", target_os = "fuchsia", target_os = "hermit"))]
 pub unsafe fn register_dtor(t: *mut u8, dtor: unsafe extern fn(*mut u8)) {
-    use libc;
-    use mem;
     use sys_common::thread_local::register_dtor_fallback;
 
     extern {
         #[linkage = "extern_weak"]
         static __dso_handle: *mut u8;
-        #[linkage = "extern_weak"]
-        static __cxa_thread_atexit_impl: *const libc::c_void;
-    }
-    if !__cxa_thread_atexit_impl.is_null() {
-        type F = unsafe extern fn(dtor: unsafe extern fn(*mut u8),
-                                  arg: *mut u8,
-                                  dso_handle: *mut u8) -> libc::c_int;
-        mem::transmute::<*const libc::c_void, F>(__cxa_thread_atexit_impl)
-            (dtor, t, &__dso_handle as *const _ as *mut _);
-        return
     }
     register_dtor_fallback(t, dtor);
 }

$ readelf -Ws main | grep GLIBC | grep -v GLIBC_2.[23]
    22: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __stack_chk_fail@GLIBC_2.4 (8)
    48: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND memcpy@GLIBC_2.14 (12)
   693: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __stack_chk_fail@@GLIBC_2.4
   804: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND memcpy@@GLIBC_2.14
$ docker run --rm -it -v $(pwd):/io centos:7 /io/main
Hello, world!

[sfackler]

That's not how extern_weak linkage works, though. The nullability decision is made at link time, not run time:

extern_weak
The semantics of this linkage follow the ELF object file model: the symbol is weak until linked, if not linked, the symbol becomes null instead of being an undefined reference.

https://llvm.org/docs/LangRef.html#linkage-types

[sanxiyn]

I agree this ideally should be fixed, but I consider this a bug on general Linux ecosystem. macOS doesn't have this problem because you compile with -mmacosx-version-min and appropriate older SDK is used. Linux needs to catch up to macOS.

[konstin]

That's not how extern_weak linkage works, though. The nullability decision is made at link time, not run time

I already understood it thus far. What I hope is that there is something like std::is_x86_feature_detected that allows to check the availability of the symbol at runtime.

[sfackler]

Rust programs can link to hundreds of glibc symbols, any of which could have versions that require newer glibcs. There's nothing particularly special about __cxa_thread_atexit_impl except that it happens to be the only one for that specific program.

[konstin]

Oh that's too bad; I had been under the impression that std links a fixed set of symbols and that we'd only need to work around __cxa_thread_atexit_impl and memcpy. But checking other programs confirmed that there are many more symbols that we'd need to care about. But since I don't even know how to find all relevant symbols there's not much sense in pursuing this further. Thanks for all the information provided!

[mckravchyk]

Best bet is to use Docker to compile the release binary. I had this exact problem and luckily, the official rust docker image did it.

There's even a command in the readme:
docker run --rm --user "$(id -u)":"$(id -g)" -v "$PWD":/usr/src/myapp -w /usr/src/myapp rust cargo build --release

If you need a version of libc lower than the version the official docker image has, you will have to build your own image, unfortunately.

[ismell]

Would something like https://github.com/wheybags/glibc_version_header help?

It seems like we need to emit a .semver op for the linker to handle?

It looks like gcc also implements a __symver__ attribute: InBetweenNames/gentooLTO#459, though it looks like llvm doesn't support the attribute yet: llvm/llvm-project#59438.

[konstin]

It's been a while and this issue is still getting shared, so i'd like to promote cargo-zigbuild, which can build for arbitrary glibc versions. For those who don't need to load dynamic libraries, you can build for *-unknown-linux-musl instead and get a static binary that works on both glibc and musl systems.