Use BOLT in CI to optimize LLVM

[Kobzol]

This PR adds an optimization step in the Linux dist CI pipeline that uses BOLT to optimize the libLLVM.so library built by bootstrap.

Steps:

• Use LLVM 15 as a bootstrap compiler and use it to build BOLT
• Compile LLVM with support for relocations (-DCMAKE_SHARED_LINKER_FLAGS="-Wl,-q")
• Gather profile data using instrumented LLVM
• Apply profile to LLVM that has already been PGOfied
• Run with BOLT profiling on more benchmarks
• Decide on the order of optimization (PGO -> BOLT?)
• Decide how we should get bolt (currently we use the host bolt)
• Clean up

The latest perf results can be found here. The current CI build time with BOLT applied is around 1h 55 minutes.

[rust-highfive]

r? @Mark-Simulacrum

(rust-highfive has picked a reviewer for you, use r? to override)

[nikic]

FYI, you can test this locally by running sudo src/ci/docker/run.sh dist-x86_64-linux.

To fix the builder failure, there's a couple of options. a) set COMPILER_RT_BUILD_XRAY=OFF, b) update the GCC version used to build clang or c) upstream and backport rust-lang/llvm-project@75fef2e. Option a) is probably easiest.

[Kobzol]

Thanks for the hint! I was preparing for upgrading GCC, but this indeed seems easier.

[Kobzol]

-DCOMPILER_RT_BUILD_XRAY=OFF doesn't seem to help.

[nikic]

Looks like -DCOMPILER_RT_BUILD_MEMPROF=OFF is also needed to completely disable sanitizers. Unfortunately that just gives you a GCC ICE during the LLD build, which would need rust-lang/llvm-project@8b2c25d. So looks like upgrading the GCC version is the way to go after all.

[Kobzol]

Thanks, I'll try it. GCC 6.5 also can't compile it, trying 7.5 now.

[tschuett]

They want to address the cmake issue:
https://reviews.llvm.org/D120625
llvm/llvm-project#54099

[Mark-Simulacrum]

@bors try @rust-timer queue

[rust-timer]

Awaiting bors try build completion.

@rustbot label: +S-waiting-on-perf

[rust-timer]

Finished benchmarking commit (8dfb407): comparison URL.

Overall result: ✅ improvements - no action needed

@rustbot label: -perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean1	range	count2
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-1.5%	[-1.5%, -1.5%]	1
All ❌✅ (primary)	-	-	0

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean1	range	count2
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-3.1%	[-10.3%, -0.7%]	202
Improvements ✅ (secondary)	-4.7%	[-9.9%, -1.0%]	232
All ❌✅ (primary)	-3.1%	[-10.3%, -0.7%]	202

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean1	range	count2
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-3.5%	[-5.1%, -1.8%]	55
Improvements ✅ (secondary)	-4.3%	[-6.6%, -2.1%]	17
All ❌✅ (primary)	-3.5%	[-5.1%, -1.8%]	55

Footnotes

1. the arithmetic mean of the percent change ↩ ↩² ↩³

2. number of relevant changes ↩ ↩² ↩³

[nnethercote]

Great results here, including a 3.5% bootstrap time reduction. Also a 3.6% max-rss across the board, I guess that must be due to code size shrinking?

[Kobzol]

I'm not sure what's the effect, I checked the LLVM lib size against recent master and it hasn't decreated. I hope that we ship the optimized LLVM 😅 I'll add more statistics once I port pgo.sh to Python.