on macOS isysroot should be set to a stable execroot relative path

[brentleyjones]

Description of the problem / feature request:

isysroot should be set to a stable exec root relative path (such as a symlink to the real Xcode SDK) to prevent the path of the build machine's Xcode from being embedded in build artifacts.

Currently isysroot is set to __BAZEL_XCODE_SDKROOT__, which currently resolves to something like /Applications/Xcode-11.3.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk. This causes paths like /Applications/Xcode-11.3.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/dispatch being embedded into artifacts when they contain debug symbols.

Feature requests: what underlying problem are you trying to solve with this feature?

Making this change would make macOS debug or dSYM generating targets more reproducible.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

On macOS:

bazel build --compilation_mode=dbg -- :some-objc-target

The .o files compiled will have embedded paths, generated by DW_AT_LLVM_isysroot, pointing to machine specific paths for Xcode.

What operating system are you running Bazel on?

macOS

What's the output of bazel info release?

release 2.1.0

Have you found anything relevant by searching the web?

None

Any other information, logs, or outputs that you want to share?

http://blog.llvm.org/2019/11/deterministic-builds-with-clang-and-lld.html

You then need to modify your build files to use --sysroot (Linux), -isysroot (macOS), -imsvc (Windows) to use these hermetic SDKs for builds. They need to be somewhere below your source root to not regress build directory name invariance.

[DavidGoldman]

FWIW swiftc emits the SDK path here, it may be best contributing to Swift directly.

[tetromino]

Question: if you set the DEVELOPER_DIR env variable to point to a stable Xcode symlink on your system, would that hide the paths sufficiently for your requirements?

I don't think we can generally assume any particular stable symlink exists (different teams will do this differently, and indeed may prefer a path to the exact xcode version for reproducibility of debugging), but perhaps we can document some suggested best practices.

[brentleyjones]

I don't think we can generally assume any particular stable symlink exists (different teams will do this differently, and indeed may prefer a path to the exact xcode version for reproducibility of debugging), but perhaps we can document some suggested best practices.

I'm suggesting that as part of external/local_config_cc we create a symlink, and Bazel uses that when computing __BAZEL_XCODE_SDKROOT__.

[tetromino]

Such a symlink would have be created under /private/var/tmp/_bazel_${USER}/${HASH}/execroot, which is even worse for reproducibility - since we would need to use the link's absolute path, if I understand correctly.

[brentleyjones]

I believe __BAZEL_XCODE_SDKROOT__ can be a relative path. Per http://blog.llvm.org/2019/11/deterministic-builds-with-clang-and-lld.html:

You then need to modify your build files to use --sysroot (Linux), -isysroot (macOS), -imsvc (Windows) to use these hermetic SDKs for builds. They need to be somewhere below your source root to not regress build directory name invariance.

[tetromino]

Can you debug binaries with a relative sysroot directory?

[brentleyjones]

You might have to set target.sdk-path, but we already have to set that with cached items because of invalid paths currently. I can test this though and see if this removes the need for that.

[brentleyjones]

XcodeLocalEnvProvider.rewriteLocalEnv() causes DEVELOPER_DIR to be ignored, such that when I do bazel build --action_env=DEVELOPER_DIR //target I get:

Caused by: java.lang.IllegalArgumentException: Multiple entries with same key: DEVELOPER_DIR=/Applications/Xcode-11.3-different.app/Contents/Developer and DEVELOPER_DIR=Xcode-11.3-symlink.app
        at com.google.common.collect.ImmutableMap.conflictException(ImmutableMap.java:215)
        at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:209)
        at com.google.common.collect.RegularImmutableMap.checkNoConflictInKeyBucket(RegularImmutableMap.java:147)
        at com.google.common.collect.RegularImmutableMap.fromEntryArray(RegularImmutableMap.java:110)
        at com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:393)
        at com.google.devtools.build.lib.exec.local.XcodeLocalEnvProvider.rewriteLocalEnv(XcodeLocalEnvProvider.java:109)
        at com.google.devtools.build.lib.exec.local.LocalSpawnRunner$SubprocessHandler.start(LocalSpawnRunner.java:342)
        at com.google.devtools.build.lib.exec.local.LocalSpawnRunner$SubprocessHandler.run(LocalSpawnRunner.java:218)

I'll look into modifying xcode_locator.m to hack around this to test if a relative path works.

[brentleyjones]

DEVELOPER_DIR can't be relative because xcrun is called from multiple places. I don't have the experience to try to get xcrunwrapper.sh to work with a relative path as a test.

[tetromino]

@DavidGoldman - do you think that an execroot-relative isysroot is feasible?

[DavidGoldman]

Ah, I completely missed the objc/clang part of this. Hmm, we could try to use a symlink for __BAZEL_XCODE_SDKROOT__ - we'd need a symlink for the Xcode part of the path.

Another option would be to upstream changes to Clang to allow -debug-prefix-map to apply to these paths if it doesn't already, and then remap them in the debugger.

[keith]

FWIW I made swift remap isysroot with debug-prefix-map (this isn't released and currently rules_swift does not pass a remapping for this) swiftlang/swift#32580 does anyone know right now if that doesn't work for clang?

[DavidGoldman]

From a quick check it looks like Clang supports remapping the sysroot here but not the SDK path? EDIT: Ah looks like the SDK is the just the SDK name, e.g. MacOSX.sdk

You could test this out by adding the following line here:

processed_args.push_back("-fdebug-prefix-map=" + developer_dir + "=__BAZEL_XCODE_DEVELOPER_DIR__");

and then you'd need to remap __BAZEL_XCODE_DEVELOPER_DIR__ for the debugger

[DavidGoldman]

From Adrian's reply it seems LLDB doesn't rely upon that value for Swift? swiftlang/swift#32580 (review) Wonder if it's the same for Clang as well. We might be okay with just using debug-prefix-map to remap the sysroot values to a constant without remapping in the debugger.

[keith]

Yea for us we do still set that in our lldbinit, so we'd prefer the values be scrubbed, but yea it would be great if that would work for everyone too!

[DavidGoldman]

Posted on the swift forums: https://forums.swift.org/t/lldb-usage-of-dw-at-llvm-isysroot/40575

[keith]

Through some more testing with Swift I found many of these entries, that were also able to be remapped:

DW_AT_LLVM_include_path ("DEVELOPER_DIR/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/lib/swift/Swift.swiftmodule/x86_64.swiftinterface")

AFAICT remapping this in the binary has no impact on debugging.

[keith]

These features should fix this bazelbuild/bazel#12175 bazelbuild/rules_swift#511

Note with Swift there was a bug where debug-prefix-map didn't apply to sysroot, it has been fixed but not made it to Xcode yet.