Enabling static-pie for musl

[haraldh]

and make it the default for the x86_64-unknown-linux-musl target

This is a quick implementation for #70693

Opening it as a draft PR to gather some feedback, before I put more work in it.

❯ cat hello.rs
fn main() {
    println!("main = {:#x}", &main as *const _ as usize);
}


❯  /tmp/rust-musl/bin/rustc  --target x86_64-unknown-linux-musl  ~/hello.rs

❯ ldd hello
	statically linked

❯ file hello
hello: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=fec5cdc170f503a712a63a6958691ce5ce433654, with debug_info, not stripped

❯ ./hello
main = 0x7f233ca30008

❯ ./hello
main = 0x7f9ddc529008

❯ ./hello
main = 0x7f1e5a224008

❯ ./hello
main = 0x7f4485c7c008

❯ /tmp/rust-musl/bin/rustc  --target x86_64-unknown-linux-musl  -Z print-link-args  ~/hello.rs
"cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-Wl,--eh-frame-hdr" "-m64" "-nostdlib" "/tmp/rust-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/rcrt1.o" "/tmp/rust-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/crti.o" "-L" "/tmp/rust-musl/lib/rustlib/x86_64-unknown-linux-musl/lib" "hello.hello.7rcbfp3g-cgu.0.rcgu.o" "hello.hello.7rcbfp3g-cgu.1.rcgu.o" "hello.hello.7rcbfp3g-cgu.2.rcgu.o" "hello.hello.7rcbfp3g-cgu.3.rcgu.o" "hello.hello.7rcbfp3g-cgu.4.rcgu.o" "hello.hello.7rcbfp3g-cgu.5.rcgu.o" "-o" "hello" "hello.1nxjf9so94czdgcz.rcgu.o" "-Wl,--gc-sections" "-static-pie" "-Wl,-zrelro" "-Wl,-znow" "-nodefaultlibs" "-L" "/tmp/rust-musl/lib/rustlib/x86_64-unknown-linux-musl/lib" "-Wl,--start-group" "-Wl,-Bstatic" "/tmp/rust-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-0f9cb7646f9e2c34.rlib" "/tmp/rust-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-ba857f2f2e4e7187.rlib" "/tmp/rust-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libhashbrown-58ba5e25bbdf9d29.rlib" "/tmp/rust-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_std_workspace_alloc-886bfe43afa847dc.rlib" "/tmp/rust-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libbacktrace-fbfb8fe99f19a67b.rlib" "/tmp/rust-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libbacktrace_sys-85fa859e7d364cc9.rlib" "/tmp/rust-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_demangle-07ab026cd3ec0d82.rlib" "/tmp/rust-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libunwind-a8ec5932d92ea864.rlib" "/tmp/rust-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcfg_if-0ba4cc2f38a198d5.rlib" "/tmp/rust-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/liblibc-c1bb2b3ce4f78b7c.rlib" "/tmp/rust-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/liballoc-0ff673c1cf0d451a.rlib" "/tmp/rust-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_std_workspace_core-c8ff2001db856926.rlib" "/tmp/rust-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcore-2ae14177140eeca2.rlib" "-Wl,--end-group" "/tmp/rust-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcompiler_builtins-4fd81b5ce1b08a9c.rlib" "-static" "-Wl,-Bdynamic" "/tmp/rust-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/crtn.o"

Closes #70693
Closes #53968

[rust-highfive]

r? @estebank

(rust_highfive has picked a reviewer for you, use r? to override)

[haraldh]

r? @alexcrichton

[Amanieu]

Note that the -static-pie flag is only supported in recent versions of GCC (since GCC 8).

cc @rust-lang/release Do we have any requirements for the GCC version that we depend on for linking?

[cuviper]

I don't think we have any stated policy on GCC support. ISTR there was some other flag that we try and fallback if not supported, but I don't remember what that was.

[Mark-Simulacrum]

I agree that we don't have a stated policy, but GCC 8 seems too recent; based on https://gcc.gnu.org/releases.html it was released May 2, 2018 -- that's just under a year ago. I assume that means that it's not available in long term stable distros, though maybe those sorts of distributions don't support musl anyway?

It would be helpful to determine what configurations this drops support for it (i.e. where is GCC prior to 8 the latest available, I guess?)

[cuviper]

The other flag I was thinking of is -no-pie, added in #48076.

[elahn]

GCC 7 is the default on Ubuntu 18.04 LTS and derivatives. GCC 8 is the default on Debian 10, RHEL/CentOS 8, Alpine 3.11. GCC 9 is the default on Fedora 31 and Ubuntu 20.04 LTS. (source)

With the same retry on unknown argument strategy as -no-pie, the worst it'll do is slow-down linking of musl builds on older distros.

[bors]

☔ The latest upstream changes (presumably #70826) made this pull request unmergeable. Please resolve the merge conflicts.

[alexcrichton]

Sorry I unfortunately don't have the bandwidth to review a change like this, @cuviper would you mind taking over?

[cuviper]

I can review, although I have no particular stake in musl support. But @Amanieu already assigned himself too.

@haraldh This needs a rebase, and I'd like to see that fallback possibility, ideally sharing that code with -no-pie. Do you have a checklist that you know needs doing for your draft?

[haraldh]

I can review, although I have no particular stake in musl support. But @Amanieu already assigned himself too.

@haraldh This needs a rebase, and I'd like to see that fallback possibility, ideally sharing that code with -no-pie. Do you have a checklist that you know needs doing for your draft?

yeah, replacing -static-pie with -static needs a little bit more than the no-pie strategy.. let me see what I can do there.

[haraldh]

@cuviper do you have a CI for some of the musl targets?

[cuviper]

@haraldh look in src/ci/docker -- dist-i586-gnu-i586-i686-musl/Dockerfile tests i686-musl, and test-various/Dockerfile tests x86_64-musl.

[haraldh]

Lol... the no-pie strategy only works for LANG=C or english. If you dare to set it to some other language it will horribly fail.

[haraldh]

All right. First draft on resetting the arguments, if -static-pie is not understood.
Passes the test with -static-pies instead of -static-pie. Still have to try on a distribution, which really does not know -static-pie.

[darbysauter]

Is there any way of doing a static PIE no-std? I want to make a relocatable kernel, people keep telling me it isn't possible with rust.

[npmccallum]

Is there any way of doing a static PIE no-std? I want to make a relocatable kernel, people keep telling me it isn't possible with rust.

I'm working on this with @joshtriplett. Stay tuned!

[Amanieu]

I actually have a working static PIE Linux binary with no_std. The source code is not public, but I can share the part that performs self-relocation early at startup. It uses Linux's auxvec to find the address of the ELF program headers but you should be able to adapt this to work for your kernel.

https://gist.github.com/Amanieu/588e3f9d330019c5d39f3ce60e8e0aae

[haraldh]

It is possible... we have the pendant to Amanieu's gist here and a static pie kernel here and the kvm boot loader here.

I guess --target x86_64-unknown-linux-musl, -C relocation-model=pic, -C -nostartfiles and a self written _start are the recipe.

[petrochenkov]

Perhaps I'm missing something, but why does static PIE requires any special actions with no_std?
CRT objects are still linked in no_std mode, and that's exactly where the self-relocation code running at the start reside.

Do you have to avoid linking in CRT objects for some other reasons, or have custom _start? Then it's not exactly a "vanilla" no_std mode.

[npmccallum]

@petrochenkov Yes. You want to link nothing and provide your own _start. If you're writing your own kernel, you likely can't depend on a crt0 stack or anything from libc. This includes builtins with any CPU features since you likely don't want to have to save/clear/restore extended CPU state during context switches.

@joshtriplett will be writing up an MCP for a new architecture called x86_64-unknown-elf which does precisely this: it turns on no CPU features and links nothing (but probably provides core and builtins for available linking if you want to opt-in; maybe even alloc).

[darbysauter]

@npmccallum Yes you explained it well. Currently my config is:
-Crelocation-model=pic
build-std = ["core", "compiler_builtins", "alloc"]
x86_64-unknown-none

The issue I am running into is the absolute addresses for GOT or in .data.rel.ro, Technically I could translate these at runtime but I would prefer not too and my .data.rel.ro contains 0x10 and 0x18 sized entries so there isn't really a good way of parsing

@haraldh The issue is not loading, but actually getting rust to compile it as a static PIE. I already have #![no_main] in my code, but I will try some of the suggested flags.

I almost feel like the -pic option shouldn't be the only option. Would it make more sense to have -pie and not compile when it is not possible rather than just "upgrading" from pic to pie essentially hidden from the user?

[Amanieu]

The absolute address in the GOT are fixed-up at runtime by processing the relocations with the code I posted. This needs to be done very early during program startup before the code tries to perform GOT accesses.

[darbysauter]

@Amanieu I am already doing this, but why couldn't these just be relative addresses from the start of a section? And sure I can do the GOT but what about .data.rel.ro which contains vtables, string references, and other stuff which contains absolute addresses which is translatable... but for some reason mine contains 0x10 byte and 0x18 byte entries (rel and rela)

[petrochenkov]

@darbysauter

The issue I am running into is the absolute addresses for GOT or in .data.rel.ro, Technically I could translate these at runtime

but actually getting rust to compile it as a static PIE

This is a static PIE, or at least the standard way to implement it in current toolchains (C/C++ including).
It indeed contains unresolved relocations, and it must contain a piece of dynamic linker statically linked into the exectuable that resolves them very early at runtime.

(I can't answer right now why more relative addresses are not introduced during codegen instead, perhaps there are technical reasons, or perhaps it's just a to avoid "one more way to codegen things".)

[petrochenkov]

That said, when we are generating code for a rlib we don't know that the rlib will be used for executables only, so codegen will effectively use -fPIC instead of -fPIE.
Perhaps if we provide an option to codegen rlibs with -fPIE, it will help to reduce the amount of relocation work that needs to be performed at runtime.

[darbysauter]

@petrochenkov
What about this here?: https://wiki.osdev.org/Stivale#Makefile
pie and no dynamic linker.

Realistically if string refs, vtables, GOT, and whatever else that is currently absolute could simply be changed to relative and the instructions that use them be relative, arguably there might be some additional overhead but really nothing much.

But also lets say the only way of doing this is manually translating at runtime how can I even do that when .data.rel.ro is labeled SHT_PROGBITS (not SHT_REL or SHT_RELA) and contains both 0x10 and 0x18 sized entries. There is no way to tell the size of these so no good way of parsing this. Theoretically I could determine the start of the entry by checking if the 8 byte value is in the range of valid addresses, but its definitely a bad way of handling this.

[darbysauter]

@petrochenkov
I think the issue is as you mentioned because the rlibs are PIC. so therefore the final output would be PIC right? Also this is likely why the final output has a mixed .data.rel.ro.

[petrochenkov]

Looks like there's already an open issue about the rlib problem - #87934.
Adding -C relocation-model=pie (that will be equivalent to -fPIE and will produce code that may be unusable when building shared libraries) looks like a reasonable solution.

[darbysauter]

that looks to be what I am looking for. Thanks!