Preserve paths under hermetic /tmp in the sandbox

[fmeum]

The bind mounting scheme used with the Linux sandbox' hermetic /tmp feature is modified to preserve all paths as they are outside the sandbox, which removes the need to rewrite paths when staging inputs into and, crucially, moving outputs out of the sandbox.

Source roots and output base paths under /tmp are now treated just like any user-specified bind mount under /tmp: They are mounted under the hermetic tmp directory with their path relativized against /tmp before the hermetic tmp directory is mounted as /tmp as the final step.

There is one caveat compared to user-specified mounts: Source roots, which may themselves not lie under /tmp, can be symlinks to directories under /tmp (e.g., when they arise from a local_repository). To handle this situation in the common case, all parent directories of package path entries (up to direct children of /tmp) are mounted into the sandbox. If users use local_repositorys with fixed target paths under /tmp, they will need to specify --sandbox_add_mount_pair.

Overlayfs has been considered as an alternative to this approach, but ultimately doesn't seem to work for this use case since its lowerpath, which would be /tmp, is not allowed to have child mounts from a different user namespace (see https://unix.stackexchange.com/questions/776030/mounting-overlayfs-in-a-user-namespace-with-child-mounts). However, this is exactly the situation created by a Bazel-in-Bazel test and can also arise if the user has existing mounts under /tmp when using Bazel (e.g. the JetBrains toolbox on Linux uses such mounts).

This replaces and mostly reverts the following commits, but keeps their tests:

• bf6ebe9
• fb6658c
• bc1d9d3
• 1829883
• 70691f2
• a556969
• 8e32f44 (had its test lost in an incorrect merge conflict resolution, this PR adds it back)

Fixes #20533
Work towards #20753
Fixes #21215
Fixes #22117
Fixes #22226
Fixes #22290

RELNOTES: Paths in the Linux sandbox are now again identical to those outside the sandbox, even with --incompatible_sandbox_hermetic_tmp.

[fmeum]

@bazel-io fork 7.2.0

[fmeum]

I introduced this check instead of checking for whether /tmp is shared with the host directly. We don't care so much about the latter, so I felt that encoding it in tests would make them more brittle as we explore alternative options.

[oquenchil]

Thanks a lot for the PR Fabian.

#22226 was broken because of hermetic tmp.

The error used to be:

external/protobuf~/src/google/protobuf/port.cc:12:10: fatal error: google/protobuf/port_def.inc: No such file or directory
   12 | #include "google/protobuf/port_def.inc"

Now the error is:

Could not make proto path relative: bazel-out/k8-fastbuild/bin/external/protobuf~/src/google/protobuf/_virtual_imports/descriptor_proto/google/protobuf/descriptor.proto: No such file or directory

Do you think this is fixable in this PR?

[fmeum]

I extended an existing integration test to cover what I think is the root cause of the issue you linked: Neither the logic on `master` nor the current PR track external roots that only arise as targets of resolved symlink artifacts. We probably need similar logic to that in fb6658c to handle this. That commit by itself doesn't suffice since it only covers source roots on the package path.

[fmeum]

Tests are passing 🎉

[oquenchil]

Amazing contribution Fabian. A lot of bug fixes with simpler code than before 🥳

LGTM

[tjgq]

This is amazing work, Fabian! Thank you.

[fmeum]

Just noticed that the PR description was out of date, I updated it.