Build library/profiler_builtins from ci-llvm if appropriate

[Zalathar]

Running all of tests/coverage requires the LLVM profiler runtime, which requires setting build.profiler = true.

Historically, doing that has required checking out the entire src/llvm-project submodule. For compiler contributors who otherwise don't need that submodule (thanks to download-ci-vm), that's quite inconvenient.

However, thanks to #129116, the downloaded CI LLVM tarball now contains a copy of LLVM's compiler-rt directory, which includes all the files needed to build the profiler runtime. So with a little bit of extra logic in bootstrap, we can have library/profiler_builtins look for the compiler-rt files in ci-llvm instead of the src/llvm-project submodule.

[rustbot]

r? @Kobzol

rustbot has assigned @Kobzol.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[Zalathar]

As mentioned in the comments, I've added a new environment variable so that there's no risk of breaking whatever it is that compiler_builtins is currently doing.

[Kobzol]

Thanks! Left a few comments.

[Zalathar]

Added fallback if RUST_COMPILER_RT_FOR_PROFILER was not set (diff).

[Kobzol]

After this PR, if people use download-ci-llvm = true, compiler-rt sources needed to build library/profiler_builtins (enabled with build.profiler = true in config.toml) will not be used from the local LLVM checkout, but from the downloaded CI LLVM archive. If download-ci-llvm = "if-unchanged" is used, local sources will be correctly used if someone modifies the compiler-rt sources locally.

@onur-ozkan Do you think that this warrants an Info record in the change tracker, to let people know that if they download LLVM from CI unconditionally, the local compiler-rt sources will not be used anymore?

[onur-ozkan]

After this PR, if people use download-ci-llvm = true, compiler-rt sources needed to build library/profiler_builtins (enabled with build.profiler = true in config.toml) will not be used from the local LLVM checkout, but from the downloaded CI LLVM archive. If download-ci-llvm = "if-unchanged" is used, local sources will be correctly used if someone modifies the compiler-rt sources locally.

@onur-ozkan Do you think that this warrants an Info record in the change tracker, to let people know that if they download LLVM from CI unconditionally, the local compiler-rt sources will not be used anymore?

IMO it's worth to have this information in the change history (so devs can open this PR and see the details behind this work).

[Kobzol]

Ok, agreed. @Zalathar could you please mention it there, with the INFO level? Thank you!

[Kobzol]

Thank you! :)

@bors r+

[bors]

📌 Commit 4acc531 has been approved by Kobzol

It is now in the queue for this repository.

[jieyouxu]

Failed in 🧻 #129474 (comment):

2024-08-23T18:28:32.1585250Z thread 'main' panicked at src/core/build_steps/compile.rs:480:9:
2024-08-23T18:28:32.1586270Z compiler-rt directory not found: "/Users/runner/work/rust/rust/build/aarch64-apple-darwin/ci-llvm/compiler-rt"
2024-08-23T18:28:32.1587060Z note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[jieyouxu]

@bors r-

[Kobzol]

We either need to wait until the ci-llvm archives are rebuilt on CI (after the next LLVM change), or implement the suggestion I had in a comment, to fallback to the checked out path.

[Zalathar]

The actual problem seems to be that llvm_from_ci is unexpectedly true on CI builds, so we can't rely on it when deciding whether using downloaded compiler-rt should work or not.

So I think my overall strategy is going to be:

• Sync src/llvm-project if it's already active, and use compiler-rt from there if possible.
• Otherwise, if llvm_from_ci is true and we can find its copy of compiler-rt, use that.
• Otherwise, fall back to requiring the src/llvm-project submodule as before, and use compiler-rt from there.

[Zalathar]

I'm sad that I couldn't get this to be fully robust, but I don't think that's possible without sweeping changes to how bootstrap keeps track of which LLVM it's using.

So hopefully this should be good enough for realistic use cases.

[Kobzol]

The actual problem seems to be that llvm_from_ci is unexpectedly true on CI builds, so we can't rely on it when deciding whether using downloaded compiler-rt should work or not.

Yes, test builders use download-ci-llvm, but why is that a problem? On CI, it shouldn't matter if we use the sources from a previous CI archive, as long as the git hash of the LLVM subdirectory hasn't changed (in which case download-ci-llvm will not be applied).

[Kobzol]

@bors retry (spurious Windows error)

[bors]

⌛ Testing commit 94aadf0 with merge a9bc1a9...

[rust-log-analyzer]

The job x86_64-msvc-ext failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

test result: ok. 25 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s

     Running tests\ui.rs (build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\deps\ui-9700297a935a1cdc.exe)
## Running ui tests in tests/pass for x86_64-pc-windows-msvc
   Compiler: "MIRI_ENV_VAR_TEST"="0" "MIRI_TEMP"="C:\\a\\_temp\\msys64\\tmp\\miri-uitest-gkhHhS" "RUST_BACKTRACE"="1" C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1\bin\miri.exe "--error-format=json" "--sysroot=C:\\a\\rust\\rust\\build\\x86_64-pc-windows-msvc\\miri-sysroot" "-Dwarnings" "-Dunused" "-Ainternal_features" "-Zmiri-provenance-gc=1" "-Zui-testing" "--target" "x86_64-pc-windows-msvc" "--out-dir" OUT_DIR
tests/pass\align_repeat_into_well_aligned_array.rs ... ok
tests/pass\alloc-access-tracking.rs ... ignored (in-test comment)
tests/pass\align_repeat_into_packed_field.rs ... ok
tests/pass\align.rs ... ok
---

test result: ok. 350 passed; 9 ignored;

## Running ui tests in tests/pass-dep for x86_64-pc-windows-msvc
   Compiler: "MIRI_ENV_VAR_TEST"="0" "MIRI_TEMP"="C:\\a\\_temp\\msys64\\tmp\\miri-uitest-gkhHhS" "RUST_BACKTRACE"="1" C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1\bin\miri.exe "--error-format=json" "--sysroot=C:\\a\\rust\\rust\\build\\x86_64-pc-windows-msvc\\miri-sysroot" "-Dwarnings" "-Dunused" "-Ainternal_features" "-Zmiri-provenance-gc=1" "-Zui-testing" "--target" "x86_64-pc-windows-msvc" "--out-dir" OUT_DIR
tests/pass-dep\tempfile.rs ... ignored (in-test comment)
tests/pass-dep\concurrency\apple-os-unfair-lock.rs ... ignored (in-test comment)
tests/pass-dep\concurrency\env-cleanup-data-race.rs ... ignored (in-test comment)
tests/pass-dep\concurrency\libc_pthread_cond_timedwait.rs ... ignored (in-test comment)
---

test result: ok. 12 passed; 28 ignored;

## Running ui tests in tests/panic for x86_64-pc-windows-msvc
   Compiler: "MIRI_ENV_VAR_TEST"="0" "MIRI_TEMP"="C:\\a\\_temp\\msys64\\tmp\\miri-uitest-gkhHhS" "RUST_BACKTRACE"="1" C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1\bin\miri.exe "--error-format=json" "--sysroot=C:\\a\\rust\\rust\\build\\x86_64-pc-windows-msvc\\miri-sysroot" "-Dwarnings" "-Dunused" "-Ainternal_features" "-Zmiri-provenance-gc=1" "-Zui-testing" "--target" "x86_64-pc-windows-msvc" "--out-dir" OUT_DIR
Building dependencies ... ok
tests/panic\overflowing-rsh-1.rs ... ok
tests/panic\overflowing-lsh-neg.rs ... ok
tests/panic\overflowing-rsh-2.rs ... ok
---

test result: ok. 14 passed; 2 ignored;

## Running ui tests in tests/fail for x86_64-pc-windows-msvc
   Compiler: "MIRI_ENV_VAR_TEST"="0" "MIRI_TEMP"="C:\\a\\_temp\\msys64\\tmp\\miri-uitest-gkhHhS" "RUST_BACKTRACE"="1" C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1\bin\miri.exe "--error-format=json" "--sysroot=C:\\a\\rust\\rust\\build\\x86_64-pc-windows-msvc\\miri-sysroot" "-Dwarnings" "-Dunused" "-Ainternal_features" "-Zmiri-provenance-gc=1" "-Zui-testing" "--target" "x86_64-pc-windows-msvc" "--out-dir" OUT_DIR
tests/fail\deny_lint.rs ... ok
tests/fail\const-ub-checks.rs ... ok
tests/fail\dyn-call-trait-mismatch.rs ... ok
tests/fail\environ-gets-deallocated.rs ... ignored (in-test comment)
---

test result: ok. 451 passed; 4 ignored;

## Running ui tests in tests/fail-dep for x86_64-pc-windows-msvc
   Compiler: "MIRI_ENV_VAR_TEST"="0" "MIRI_TEMP"="C:\\a\\_temp\\msys64\\tmp\\miri-uitest-gkhHhS" "RUST_BACKTRACE"="1" C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1\bin\miri.exe "--error-format=json" "--sysroot=C:\\a\\rust\\rust\\build\\x86_64-pc-windows-msvc\\miri-sysroot" "-Dwarnings" "-Dunused" "-Ainternal_features" "-Zmiri-provenance-gc=1" "-Zui-testing" "--target" "x86_64-pc-windows-msvc" "--out-dir" OUT_DIR
tests/fail-dep\concurrency\apple_os_unfair_lock_assert_owner.rs ... ignored (in-test comment)
tests/fail-dep\concurrency\apple_os_unfair_lock_reentrant.rs ... ignored (in-test comment)
tests/fail-dep\concurrency\apple_os_unfair_lock_unowned.rs ... ignored (in-test comment)
tests/fail-dep\concurrency\libc_pthread_cond_double_destroy.rs ... ignored (in-test comment)
---
[RUSTC-TIMING] miri test:false 4.385
error: failed to remove file `C:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-tools\x86_64-pc-windows-msvc\release\miri.exe`

Caused by:
  Access is denied. (os error 5)
Command has failed. Rerun with -v to see more details.
  local time: Sun, Aug 25, 2024  2:48:55 AM
  network time: Sun, 25 Aug 2024 02:48:55 GMT
##[error]Process completed with exit code 1.
Post job cleanup.

[bors]

💔 Test failed - checks-actions

[tgross35]

@bors retry #127883

[bors]

⌛ Testing commit 94aadf0 with merge 1a94d83...

[bors]

☀️ Test successful - checks-actions
Approved by: Kobzol
Pushing 1a94d83 to master...

[rust-timer]

Finished benchmarking commit (1a94d83): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results (secondary 0.1%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.3%	[2.3%, 2.3%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-2.1%	[-2.1%, -2.1%]	1
All ❌✅ (primary)	-	-	0

Cycles

Results (secondary 2.6%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.6%	[2.2%, 3.0%]	5
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 751.721s -> 752.849s (0.15%)
Artifact size: 338.75 MiB -> 338.80 MiB (0.02%)