linker: More systematic handling of CRT objects #71769

petrochenkov · 2020-05-01T20:00:48Z

Document which kinds of crt0.o-like objects we link and in which cases, discovering bugs in process.
src/librustc_target/spec/crt_objects.rs is the place to start reading from.

This PR also automatically contains half of the -static-pie support (#70740), because that's one of the six cases that we need to consider when linking CRT objects.

This is a breaking change for custom target specifications that specify CRT objects.

Closes #30868

rust-highfive · 2020-05-01T20:00:51Z

r? @eddyb

(rust_highfive has picked a reviewer for you, use r? to override)

petrochenkov · 2020-05-01T20:08:13Z

src/librustc_codegen_ssa/back/link.rs

-        let mut user_defined_link_args = sess.opts.cg.link_args.iter().chain(attr_link_args);
-        if sess.relocation_model() == RelocModel::Pic
-            && !sess.crt_static(Some(crate_type))
-            && !user_defined_link_args.any(|x| x == "-static")


The only case of rustc interpreting user-provided raw linker arguments is removed here.
First, -static wins over -pie so we could still pass -pie if the user passed -static with the same effect.
Second, it was a hack, this static guessing wasn't done consistently in other places, equivalent cases like -Wl,-static weren't recognized, and we shouldn't be interpreting raw linker args in general.

petrochenkov · 2020-05-01T20:10:10Z

src/librustc_target/spec/fuchsia_base.rs

+            (CrtObjectKind::DynamicPicExe, &["Scrt1.o"]),
+            (CrtObjectKind::StaticNoPicExe, &["Scrt1.o"]),
+            (CrtObjectKind::StaticPicExe, &["Scrt1.o"]),
+        ]),


No idea which libc flavor Fuchsia uses, just kept the status quo behavior.

src/librustc_target/spec/wasm32_wasi.rs

src/librustc_codegen_ssa/back/link.rs

bors · 2020-05-03T03:13:51Z

☔ The latest upstream changes (presumably #71807) made this pull request unmergeable. Please resolve the merge conflicts.

src/librustc_target/spec/crt_objects.rs

src/librustc_target/spec/linux_musl_base.rs

petrochenkov · 2020-05-10T13:43:45Z

@smaeul @shizmob @mixi (people who participated in #40113 and related PRs), which of you I can ping on PRs/issues like this one?

I have a question.
From the paper trail (#71586 (comment)) it looks like it's strongly preferable to use a musl-aware system gcc as a linker if it's available, and rely on it for pulling CRT objects. (And system libc.a as well, but it's a separate question.)
At the same time, for backward compatibility in some cases we have to fall back to gcc with -nostdlib and manually linking CRT objects distributed with rustc.

What would be a good and safe approximation for "musl-aware system gcc is available" that rustc could use to apply the fallback only when it's really necessary?

host == target (both musl) seems like a good heuristic
anything else?

smaeul · 2020-05-11T01:20:47Z

Checking to see if $GCC -dumpmachine contains "-musl" may be somewhat less fragile, as it would handle musl-to-musl cross compiling, and would handle the user overriding the linker (say, to a real musl cross toolchain when cross compiling glibc-to-musl).

petrochenkov · 2020-05-11T09:16:29Z

Checking to see if $GCC -dumpmachine contains "-musl" may be somewhat less fragile

Clarification: without expensive environment probing like running processes (but checking files is ok).

Ideally such check is performed by a build system, and the result is cached, and then passed to the compiler.
Cargo cannot do such things unfortunately, but it would be useful for other things as well (e.g. detecting support of -static-pie and -no-pie).

and would handle the user overriding the linker

Explicitly passed -C linker may be a good heuristic actually.

petrochenkov · 2020-05-11T22:44:46Z

So, I've measured the slowdown from the gcc -dumpmachine probing on a subset of UI tests in rustc (./x.py test --bless --stage 1 src/test/ui --test-args f, that's at least 3206 invocations of rustc, some don't do linking though).
The measurements were done on Windows, which is known for its slow process creation, so the slowdown on Linux should be smaller.

No probing (3 runs):

real    1m46.719s
real    1m47.304s
real    1m49.113s

Probing (3 runs):

real    1m50.650s
real    1m51.125s
real    1m50.946s

That's ~3% slower.

Should be pretty acceptable if it only happens in setups in which other heuristics fail.
I'll try to implement it in a separate PR.

In this PR some FIXMEs and comments need to be updated before it's ready.

petrochenkov · 2020-05-13T20:08:59Z

r? @cuviper
eddyb is nowhere in sight since april

cuviper · 2020-05-20T00:27:34Z

I don't have deep knowledge of all of these targets, but it looks reasonable to me.

@bors r+

bors · 2020-05-20T00:27:36Z

📌 Commit 49eb35c has been approved by cuviper

bors · 2020-05-20T05:47:16Z

⌛ Testing commit 49eb35c with merge 64ad709...

bors · 2020-05-20T09:28:16Z

☀️ Test successful - checks-azure
Approved by: cuviper
Pushing 64ad709 to master...

rustc_target: Remove `pre_link_args_crt` To regain some more control over the definition of `+crt-static` (rust-lang#71586). After rust-lang#71769 this target option wasn't used anywhere except for VxWorks, and I suspect that for VxWorks its use may be redundant as well.

tchebb · 2020-07-30T03:16:16Z

src/librustc_codegen_ssa/back/link.rs

+    match kind {
+        LinkOutputKind::DynamicPicExe if !pic_exe_supported => LinkOutputKind::DynamicNoPicExe,
+        LinkOutputKind::StaticPicExe if !static_pic_exe_supported => LinkOutputKind::StaticNoPicExe,
+        LinkOutputKind::StaticDylib if !static_dylib_supported => LinkOutputKind::DynamicDylib,


Am I missing something, or is this logic inverted from what it should be? If crt_static() returns true, aren't we only allowed to create a DynamicDylib when crt_static_allows_dylibs is also true?

This case is here for consistency, but we should never hit it right now because if invalid_output_for_target is true then we skip the crate type and never get to linking.
But if we didn't skip it (like we don't skip PIC executables when +crt-static is enabled but not supported), then we would fall back from StaticDylib to DynamicDylib.

rust-highfive assigned eddyb May 1, 2020

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label May 1, 2020

This was referenced May 1, 2020

Enabling static-pie for musl #70740

Merged

De-stabilize target spec JSON #71009

Open

petrochenkov commented May 1, 2020

View reviewed changes

src/librustc_target/spec/wasm32_wasi.rs Outdated Show resolved Hide resolved

petrochenkov commented May 1, 2020

View reviewed changes

src/librustc_codegen_ssa/back/link.rs Show resolved Hide resolved

petrochenkov mentioned this pull request May 2, 2020

linker: Support -static-pie and -static -shared #71804

Merged

petrochenkov force-pushed the crto branch from 9f0ebf8 to cda4990 Compare May 3, 2020 07:28

This was referenced May 9, 2020

Custom target files don't support (pre|post)_link_objects_(dll|exe) #30868

Closed

Link using the linker directly #11937

Open

petrochenkov commented May 9, 2020

View reviewed changes

src/librustc_target/spec/crt_objects.rs Outdated Show resolved Hide resolved

petrochenkov commented May 10, 2020

View reviewed changes

src/librustc_target/spec/linux_musl_base.rs Outdated Show resolved Hide resolved

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 12, 2020

petrochenkov force-pushed the crto branch from cda4990 to ef06a73 Compare May 13, 2020 20:07

rust-highfive assigned cuviper and unassigned eddyb May 13, 2020

petrochenkov added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels May 13, 2020

linker: More systematic handling of CRT objects

49eb35c

petrochenkov force-pushed the crto branch from ef06a73 to 49eb35c Compare May 14, 2020 07:55

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 20, 2020

bors added the merged-by-bors This PR was explicitly merged by bors. label May 20, 2020

bors merged commit 64ad709 into rust-lang:master May 20, 2020

bors mentioned this pull request May 20, 2020

rustc_target: Avoid an inappropriate use of post_link_objects #72304

Merged

petrochenkov mentioned this pull request May 23, 2020

provide a way of overriding the MinGW library path #68887

Closed

petrochenkov mentioned this pull request May 30, 2020

rustc_target: Remove pre_link_args_crt #72782

Merged

petrochenkov mentioned this pull request Jun 15, 2020

linking error: unknown agrument -no-pie #73370

Closed

cuviper mentioned this pull request Jun 16, 2020

linker: Never pass -no-pie to non-gnu linkers #73384

Merged

tchebb reviewed Jul 30, 2020

View reviewed changes

petrochenkov mentioned this pull request Feb 26, 2021

add dynamically-linked musl targets #82556

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

linker: More systematic handling of CRT objects #71769

linker: More systematic handling of CRT objects #71769

petrochenkov commented May 1, 2020 •

edited

Loading

rust-highfive commented May 1, 2020

petrochenkov May 1, 2020 •

edited

Loading

petrochenkov May 1, 2020

bors commented May 3, 2020

petrochenkov commented May 10, 2020 •

edited

Loading

smaeul commented May 11, 2020

petrochenkov commented May 11, 2020

petrochenkov commented May 11, 2020 •

edited

Loading

petrochenkov commented May 13, 2020

cuviper commented May 20, 2020

bors commented May 20, 2020

bors commented May 20, 2020

bors commented May 20, 2020

tchebb Jul 30, 2020 •

edited

Loading

petrochenkov Jul 30, 2020

linker: More systematic handling of CRT objects #71769

linker: More systematic handling of CRT objects #71769

Conversation

petrochenkov commented May 1, 2020 • edited Loading

rust-highfive commented May 1, 2020

petrochenkov May 1, 2020 • edited Loading

Choose a reason for hiding this comment

petrochenkov May 1, 2020

Choose a reason for hiding this comment

bors commented May 3, 2020

petrochenkov commented May 10, 2020 • edited Loading

smaeul commented May 11, 2020

petrochenkov commented May 11, 2020

petrochenkov commented May 11, 2020 • edited Loading

petrochenkov commented May 13, 2020

cuviper commented May 20, 2020

bors commented May 20, 2020

bors commented May 20, 2020

bors commented May 20, 2020

tchebb Jul 30, 2020 • edited Loading

Choose a reason for hiding this comment

petrochenkov Jul 30, 2020

Choose a reason for hiding this comment

petrochenkov commented May 1, 2020 •

edited

Loading

petrochenkov May 1, 2020 •

edited

Loading

petrochenkov commented May 10, 2020 •

edited

Loading

petrochenkov commented May 11, 2020 •

edited

Loading

tchebb Jul 30, 2020 •

edited

Loading