Use lld by default on x64 Ubuntu 20.04 LTS

[Gankra]

This is a metabug, constraining the unbound scope of #39915.

What is lld

A linker that's part of the llvm project, which is desirable for two reasons:

• it's very friendly to cross compilation (hence its emphasis for embedded targets)
• it's very fast (often runs in half the time as Gold -- linking can take several minutes for big projects (rustc, servo, etc.) and linking can be a huge % of the compile with incremental builds, so halving this runtime is a Big Deal.)

Rust currently ships its own copy of lld which it calls rust-lld. This is used by default to link bare-metal targets and wasm.

Goal

The goal of this metabug is to use rust-lld by default on a major x64 linux distro. I have arbitrarily chosen Ubuntu 20.04 LTS as our target. With all likelihood, this will incidentally get it working/enabled on many other linux distros, BSDs, and other similar platforms, but I am purposefully constraining the scope to one distro for the sake of focusing the effort.

My understanding is that the ELF backend is quite well maintained. Quoting lld's own landing page:

We are currently working closely with the FreeBSD project to make LLD default system linker in future versions of the operating system, so we are serious about addressing compatibility issues. As of February 2017, LLD is able to link the entire FreeBSD/amd64 base system including the kernel. With a few work-in-progress patches it can link approximately 95% of the ports collection on AMD64. For the details, see FreeBSD quarterly status report.

Hopefully this has continued to improve, and using lld will be a slam dunk. Although I specify that we should use rust-lld, it may be necessary/desirable to detect and use system copies of lld.

Blocking Issues

• Support feeding rust-lld into gcc (and clang) #71519 - support making gcc use rust-lld
• Coverage is not generated when using lld as linker #71233 - coverage information not generated when using lld

[cuviper]

As a distro maintainer (Fedora and RHEL), I would definitely want this to use the system lld, at least when we're talking about the system rustc that I compile.

[est31]

There is no such thing as a Ubuntu 20.04 LTS target platform (unlike wasm or the bare embedded targets you mention), but restricting it to x86_64-unknown-linux-gnu would be a good idea.

[Gankra]

yes, doing a good job with Ubuntu will likely get a lot of x86_64-unknown-linux-gnu working. but i'd rather get one specific (major) distro working first, since we're talking about integrating with system toolchains here.

[est31]

@Gankra I assumed you meant a target based check because the two other cases you mention (wasm and bare metal) are target based. Now I'm even more confused. What do you want to do? A host based check? As in: cargo or rustc trying to invoke lsb_release -a or something at its startup and enable LLD if it's Ubuntu 20.04? What if you cross compile to Windows or Mac? Or do you just want to talk to Ubuntu's rustc packagers so that they make it use LLD in the Ubuntu package?

[Gankra]

I don't intend to explicitly check for the distro/release, but rather am willing to have toolchain checks which succeed on vanilla Ubuntu 20.04 but e.g. don't succeed on the latest Arch or older Ubuntus.

So we don't need to worry about supporting using rust-lld with older versions of gcc/clang or whatever other tools, as long as our checks are solid enough to detect those things and fall back to using the native ld.

The starting point of this would be to not have this detection set up. We would instead blindly try to use rust-lld on x64 linux gnu targets if it's explicitly requested with the -C linker or -C linker-flavor options. This means using those options could lead to catastrophic failures in many configurations, but we will do our best to ensure they work on typical Ubuntu 20.04 installs (e.g. no older/weird toolchains installed).

In all likelihood this will incidentally get most major modern distro releases working well, but I would regard that as a Happy Accident, and not a priority. It would however provide a tool for distro maintainers to check if rust-lld mode works for them, which is good and useful.

After that we would add the detection or expand support until we could have the compiler try to automatically enable rust-lld without an explicit request on x64 linux gnu targets (possibly this step would be done by Cargo and not rustc). Once this is in good enough shape (everything working, not necessarily using rust-lld), we would enable this behaviour in production.

From there we will have a solid foundation for folks to push forward rust-lld support on their platforms by either making our toolchain detection more sophisticated, or by updating their distros to meet our requirements.

[est31]

@Gankra thanks for the explainer. Have there been any "please test LLD" threads? Those were done e.g. for pipelined compilation and might be helpful in this instance as well.

Also maybe one should think about adding an LLD label for issues so that they can be tracked.

[Gankra]

we should have an mvp of #71519 before we ask folks to try out linux lld support

[Gankra]

(CC to @rui314 in case they're interested in the fact that we're investigating this -- see also #71520)

[sledgehammervampire]

@Gankra I've made an MVP of #71519. What's next?

Edit: It should work for pure Rust projects. It doesn't always work when C libraries are dynamically linked.

[Logarithmus]

@1000teslas provided that C libraries have been fixed in #86740, what's the next thing to do to finally use lld by default? It's more than a year since this issue was created!

[sledgehammervampire]

@Logarithmus I have no idea to be honest. I was pretty much just following what @Gankra suggested.

[Gankra]

@1000teslas oh wow, great work!

All that's left to do is advertise the new flag and provide some instructions for people to test out if it works correctly/and how it performs. I know there was some discussion about the final flag name, could you post instructions on how to use it here? Then someone more familiar with community rallying can encourage people to check it out.

[sledgehammervampire]

@Gankra It is used like this: -Z gcc-ld=lld as part of RUSTFLAGS.

Where should I post the instructions?

[Gankra]

I've prodded the compiler folks to do a writeup, we should be able to take care of the rest for you. I'll post a link the the internals/users thread when it's ready (hopefully very soon, sorry for the slow response on your initial ping).

[sledgehammervampire]

Unfortunately, it seems like my PR only works for a stage1 toolchain. I tried to dogfood my PR using a nightly toolchain from rustup on one of my pure-Rust projects and I get some weird errors. When I use ldd on the rust-lld from the nightly toolchain provided by rustup, I see: libLLVM-12-rust-1.55.0-nightly.so => not found, whereas I see no such dependency in my stage1 toolchain.

Then again, I'm using a "weird" distro (NixOS), so maybe it'll work on a more conventional Linux distro.

[bjorn3]

$ ls $(rustc --print sysroot)/lib
libLLVM-12-rust-1.55.0-nightly.so  librustc_driver-8922750e6d18b0c7.so  libstd-37fdd654777c296f.so  libtest-08487f8e282f19e0.so  rustlib

libLLVM-12-rust-1.55.0-nightly.so should be available in $(rustc --print sysroot)/lib. You may need to set LD_LIBRARY_PATH. Alternatively you can run rustup run nightly ldd /path/to/rust-lld.

[sledgehammervampire]

@bjorn3 Are you saying that even if my system's ldd can't find the location of libLLVM-12-rust-1.55.0-nightly.so, when rust-lld is invoked by rustc, it knows where to find libLLVM-12-rust-1.55.0-nightly.so?

Anyway, I get errors that look like

error: linking with `cc` failed: exit status: 1
<bunch of cc gobbledygook>
  = note: collect2: fatal error: execvp: No such file or directory

[bstrie]

@1000teslas I'd like to help get the word out about testing this more broadly, would you be able to join us in the compiler team's Zulip so that we can hash out the details? https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/Need.20users.2Finternals.20thread.20for.20gcc-lld/near/245177775

[pnkfelix]

Visited during T-compiler backlog bonanza.

The work that @lqd is doing for rust-lang/compiler-team#510 are meant to be the final steps to unblocking turning lld on by default for Linux.

@rustbot label: S-tracking-impl-incomplete

[bstrie]

Now that rust-lang/compiler-team#510 is finished, I believe the next step here is to do some performance investigation to see if it's worth turning this on by default. If anyone would like to chime in with their results of turning this on for their Rust projects, now's the time to share your results. :)

[Nemo157]

I just tried this on https://github.com/rust-lang/docs.rs, to get the link time I used cargo rustc --bin cratesfyi -- -Zgcc-ld=lld -Ztime-passes -Ztime-passes-format=json |& rg '^time:' | cut -d' ' -f 2 | jq 'select(.pass == "link") | .time' and alternated running this with and without -Zgcc-ld=lld¹ 10 times each, after a few warmup runs, giving timings of:

cargo rustc --bin cratesfyi -- -Zgcc-ld=lld
  mean = 1.00, sd = 0.01
cargo rustc --bin cratesfyi
  mean = 5.46, sd = 0.04

So lld is taking <20% of the time for linking in this case. The overall build time after touch src/lib.rs with the default linker is ~11s (but with pretty big variance) so that's almost a 50% re-build time reduction.

Footnotes

1. while the MCP is closed the PRs implementing those new flags still appears to be open ↩

[tgnottingham]

Is it still the case that lld lacks jobserver support? Is that going to be a blocker to turning it on by default?

[Firstyear]

Now that rust-lang/compiler-team#510 is finished, I believe the next step here is to do some performance investigation to see if it's worth turning this on by default. If anyone would like to chime in with their results of turning this on for their Rust projects, now's the time to share your results. :)

We've seen about 40% reduction in build times at SUSE for projects that use LLD over the default system linker. I don't have the full metrics on hand right now however.

[and-reas-se]

Now that rust-lang/compiler-team#510 is finished, I believe the next step here is to do some performance investigation to see if it's worth turning this on by default. If anyone would like to chime in with their results of turning this on for their Rust projects, now's the time to share your results. :)

My current project is a pretty typical website backend. Pretty heavy on dependencies: Web server, database pool, templating engine, etc. The build time on my local machine is pretty much cut in half, from two minutes to one minute for a clean build, when I change the linker from the system default (the gnu one I assume) to lld.

However, in CI the difference wasn't nearly as dramatic. Possibly due to the version of llvm / lld in CI being several llvm versions behind the one on my local machine.

[kamulos]

Now that rust-lang/compiler-team#510 is finished, I believe the next step here is to do some performance investigation to see if it's worth turning this on by default. If anyone would like to chime in with their results of turning this on for their Rust projects, now's the time to share your results. :)

For me, this is mostly about development experience. When building a big project incrementally, the time to build is dominated by the linking time.

[Tergeltengis]

what about aarch64-unknown-linux-musl-clang

[Nemo157]

[...] but I am purposefully constraining the scope to one distro for the sake of focusing the effort

This issue is purely about a single target (and even a single distro within that target), any other targets would fall under the more general issue linked in the OP or their own tracking issues.

[lqd]

status update: as of #124129, rustup-distributed rustc (not distros') on x86_64-unknown-linux-gnu uses the self-contained lld linker by default, (rust-lld, itself also distributed with rustup), on nightly only for now and starting from tomorrow's nightly.

[bstrie]

status update: as of #124129, rustup-distributed rustc (not distros') on x86_64-unknown-linux-gnu uses the self-contained lld linker by default, (rust-lld, itself also distributed with rustup), on nightly only for now and starting from tomorrow's nightly.

What's the next step here? Now that we have some experience with LLD on nightly, in order to proceed further we'd need to start identifying LLD-related bugs and determining which of them are blockers. I think the A-linkage tag is the appropriate tag to watch, but AFAICT there's a lot of stuff there unrelated to LLD specifically. Should we have an A-lld tag to keep better track?

And even beyond blocking bugs, surely we can think about the steps after that and start pursuing them concurrently. For example, are all the rustc command line flags properly designed, bikeshedded, implemented, and stabilized? What about the Cargo-level config options? (Following the history of how the linker is intended to be configured seems to be surprisingly difficult!)

And even once we have no more blocking bugs and we've solved the user interface question, have we determined if we're allowed to change the default? In other words, is changing the default linker a breaking change? (IME linker scripts are incompatible between LD and LLD, which is an example of something that's sure to break.) If so, then we'd need to wait until the next edition to change this over to be the default.