Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

--incompatible_sandbox_hermetic_tmp breaks virtual includes if using output base under /tmp #22117

Closed
thii opened this issue Apr 25, 2024 · 1 comment
Assignees
Labels
P2 We'll consider working on this in future. (Assignee optional) team-Local-Exec Issues and PRs for the Execution (Local) team type: bug

Comments

@thii
Copy link
Member

thii commented Apr 25, 2024

Description of the bug:

As the title says, --incompatible_sandbox_hermetic_tmp (default since Bazel 7) breaks virtual includes if building with an output base starts with /tmp.

Which category does this issue belong to?

No response

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

  1. Create a directory like this:
repro-bazel-virtual-includes » tree
.
├── BUILD
├── include
│   ├── BUILD
│   └── h.h
├── main.c
└── WORKSPACE

Where:

BUILD

cc_library(
    name = "main",
    srcs = ["main.c"],
    deps = ["@include//:h"],
)

include/BUILD

cc_library(
    name = "h",
    hdrs = ["h.h"],
    include_prefix = "prefix/",
    visibility = ["//visibility:public"],
)

h.h is an empty file.

main.c

#include <prefix/h.h>

WORKSPACE

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
    name = "include",
    url = "file:///home/user/repro-bazel-virtual-includes/include.tar.gz", # replace with your local path
    strip_prefix = "include",
)
  1. Create include.tar.gz with (needs to be a tarball -- can't reproduce with a local_repository):
tar czf include.tar.gz include
  1. Run:
bazel --output_base=/tmp/outbase build //:main
INFO: Analyzed target //:main (68 packages loaded, 300 targets configured).
ERROR: /home/user/repro-bazel-virtual-includes/BUILD:1:11: Compiling main.c failed: (Exit 1): gcc failed: error executing CppCompile command (from target //:main) /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -MD -MF bazel-out/k8-fastbuild/bin/_objs/main/main.pic.d ... (remaining 20 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
main.c:1:10: fatal error: prefix/h.h: No such file or directory
    1 | #include <prefix/h.h>
      |          ^~~~~~~~~~~~
compilation terminated.
Target //:main failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 2.364s, Critical Path: 0.04s
INFO: 5 processes: 5 internal.
ERROR: Build did NOT complete successfully

The following command works:

bazel build //:main

Which operating system are you running Bazel on?

Linux

What is the output of bazel info release?

7.1.1 release

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

@thii
Copy link
Member Author

thii commented Apr 25, 2024

cc @fmeum

@iancha1992 iancha1992 added the team-Local-Exec Issues and PRs for the Execution (Local) team label Apr 25, 2024
@meisterT meisterT added P2 We'll consider working on this in future. (Assignee optional) and removed untriaged labels Apr 30, 2024
@fmeum fmeum assigned fmeum and unassigned oquenchil May 8, 2024
fmeum added a commit to fmeum/bazel that referenced this issue May 16, 2024
The bind mounting scheme used with the Linux sandbox' hermetic `/tmp` feature is modified to preserve all paths as they are outside the sandbox, which removes the need to rewrite paths when staging inputs into and, crucially, moving outputs out of the sandbox.

Source roots and output base paths under `/tmp` are now treated just like any user-specified bind mount under `/tmp`: They are mounted under the hermetic tmp directory with their path relativized against `/tmp` before the hermetic tmp directory is mounted as `/tmp` as the final step.

There is one caveat compared to user-specified mounts: Source roots, which may themselves not lie under `/tmp`, can be symlinks to directories under `/tmp` (e.g., when they arise from a `local_repository`). To handle this situation in the common case, all parent directories of package path entries (up to direct children of `/tmp`) are mounted into the sandbox. If users use `local_repository`s with fixed target paths under `/tmp`, they will need to specify `--sandbox_add_mount_pair`.

Overlayfs has been considered as an alternative to this approach, but ultimately doesn't seem to work for this use case since its `lowerpath`, which would be `/tmp`, is not allowed to have child mounts from a different user namespace (see https://unix.stackexchange.com/questions/776030/mounting-overlayfs-in-a-user-namespace-with-child-mounts). However, this is exactly the situation created by a Bazel-in-Bazel test and can also arise if the user has existing mounts under `/tmp` when using Bazel (e.g. the JetBrains toolbox on Linux uses such mounts).

This replaces and mostly reverts the following commits, but keeps their tests:
* bazelbuild@bf6ebe9
* bazelbuild@fb6658c
* bazelbuild@bc1d9d3
* bazelbuild@1829883
* bazelbuild@70691f2
* bazelbuild@a556969
* bazelbuild@8e32f44 (had its test lost in an incorrect merge conflict resolution, this PR adds it back)

Fixes bazelbuild#20533
Work towards bazelbuild#20753
Fixes bazelbuild#21215
Fixes bazelbuild#22117
Fixes bazelbuild#22226
Fixes bazelbuild#22290

RELNOTES: Paths in the Linux sandbox are now again identical to those outside the sandbox, even with `--incompatible_sandbox_hermetic_tmp`.

Closes bazelbuild#22001.

PiperOrigin-RevId: 634381503
Change-Id: I9f7f3948c705be120c55c9b0c51204e5bea45f61
github-merge-queue bot pushed a commit that referenced this issue May 16, 2024
The bind mounting scheme used with the Linux sandbox' hermetic `/tmp`
feature is modified to preserve all paths as they are outside the
sandbox, which removes the need to rewrite paths when staging inputs
into and, crucially, moving outputs out of the sandbox.

Source roots and output base paths under `/tmp` are now treated just
like any user-specified bind mount under `/tmp`: They are mounted under
the hermetic tmp directory with their path relativized against `/tmp`
before the hermetic tmp directory is mounted as `/tmp` as the final
step.

There is one caveat compared to user-specified mounts: Source roots,
which may themselves not lie under `/tmp`, can be symlinks to
directories under `/tmp` (e.g., when they arise from a
`local_repository`). To handle this situation in the common case, all
parent directories of package path entries (up to direct children of
`/tmp`) are mounted into the sandbox. If users use `local_repository`s
with fixed target paths under `/tmp`, they will need to specify
`--sandbox_add_mount_pair`.

Overlayfs has been considered as an alternative to this approach, but
ultimately doesn't seem to work for this use case since its `lowerpath`,
which would be `/tmp`, is not allowed to have child mounts from a
different user namespace (see
https://unix.stackexchange.com/questions/776030/mounting-overlayfs-in-a-user-namespace-with-child-mounts).
However, this is exactly the situation created by a Bazel-in-Bazel test
and can also arise if the user has existing mounts under `/tmp` when
using Bazel (e.g. the JetBrains toolbox on Linux uses such mounts).

This replaces and mostly reverts the following commits, but keeps their
tests:
*
bf6ebe9
*
fb6658c
*
bc1d9d3
*
1829883
*
70691f2
*
a556969
*
8e32f44
(had its test lost in an incorrect merge conflict resolution, this PR
adds it back)

Fixes #20533
Work towards #20753
Fixes #21215
Fixes #22117
Fixes #22226
Fixes #22290

RELNOTES: Paths in the Linux sandbox are now again identical to those
outside the sandbox, even with `--incompatible_sandbox_hermetic_tmp`.

Closes #22001.

PiperOrigin-RevId: 634381503
Change-Id: I9f7f3948c705be120c55c9b0c51204e5bea45f61

Fixes #22291
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P2 We'll consider working on this in future. (Assignee optional) team-Local-Exec Issues and PRs for the Execution (Local) team type: bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants