Parent directory of output file not made writable when building without the bytes

[william-smith-skydio]

Description of the bug:

This is the same as #13229, except it seems to have resurfaced with --remote_download_toplevel (or --remote_download_minimal) enabled. Since --remote_download_toplevel enabled by default in bazel 7.x+, it occurs in the default configuration of bazel 7.x+ if remote cache is enabled.

To summarize, if I have an action that creates a directory artifact, and then I replace it with a different action that creates a file in the directory, that action will fail because bazel fails to add write permissions to the directory. I can work around this with bazel clean, but that is prohibitively expensive for large repos.

The error looks like:

ERROR: BUILD.bazel:8:8: Action foobar/foobar2 failed: I/O exception during sandboxed execution: Could not move output artifacts from sandboxed execution

The original issue was supposed to be fixed in cff2ea5, but it seems like that logic isn't being used when --remote_download_toplevel is enabled.

Which category does this issue belong to?

Local Execution

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Create the workspace from this gist: https://gist.github.com/william-smith-skydio/654bfe393e9388bb11c2c2f32f60628c

Run:

bazel build //:foobar

This will succeed. Now change use_dir to False for the foobar rule and re-run:

bazel build //:foobar

This fails:

ERROR: BUILD.bazel:8:8: Action foobar/foobar2 failed: I/O exception during sandboxed execution: Could not move output artifacts from sandboxed execution
Target //:foobar failed to build

The output directory has these permissions (the directory is missing write permissions):

$ ls -lh bazel-out/k8-fastbuild/bin/ 
total 4.0K
dr-xr-xr-x 2 williamsmith williamsmith 4.0K Aug 28 15:24 foobar

Which operating system are you running Bazel on?

Ubuntu 18.04

What is the output of bazel info release?

7.3.1

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

No response

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

No response

Have you found anything relevant by searching the web?

Other potentially-related issues:

• "I/O exception during sandboxed execution" when using tree artifacts #13229
• Hermetic sandbox fails to move output artifacts if one of them is a read-only directory #20032
• Could not move output artifacts from sandboxed execution #22260
• Reason for Could not move output artifacts from sandboxed execution is not given #23283

Any other information, logs, or outputs that you want to share?

This issue is present in 7.3.1 and 8.0.0-pre.20240821.2 with remote cache enabled (since --remote_download_toplevel is default). In the above reproducer I use --disk_cache for simplicity, but it also happens with a proper remote cache. The issue is also present in 6.4.0 if I enable --remote_download_toplevel, but for whatever reason I couldn't reproduce with --disk_cache, only a proper remote cache.

[tjgq]

Debugging note: it's not a sandboxing issue, an error also occurs with --spawn_strategy=standalone (touch: bazel-out/darwin_arm64-fastbuild/bin/foobar/foobar2: Permission denied). I expect the issue to be that Bazel doesn't enforce the precondition that the parent directory of an action output must be writable, when that directory was an internal subdirectory of a tree artifact in a previous invocation. And it only happens in the BwoB path, because without --disk_cache it works just fine.

[Silic0nS0ldier]

This looks to be the same issue that spurred the creation of #23283.

Had this issue crop up internally today and repro'd with the logging improvements in #23381.

ERROR: /_monorepo_/tools/logging/BUILD.bazel:15:15: Transpiling & type-checking TypeScript project @@//tools/logging:lib [tsc -p tools/logging/tsconfig.json] failed: Could not copy outputs from sandbox: /_bzl_output_base_/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/tools/logging/dist/logger.js (Permission denied)
Target //tools/___:___ failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /_monorepo_/tools/logging/BUILD.bazel:8:12 Copying files to directory tools/logging/pkg failed: Could not copy outputs from sandbox: /_bzl_output_base_/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/tools/logging/dist/logger.js (Permission denied)

With some more digging;

_bzl_out="/_monorepo_/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin"
echo "Checking ${_bzl_out}"
ls -l "${_bzl_out}"
# All fine
echo "Checking ${_bzl_out}/tools"
ls -l "${_bzl_out}/tools"
# All fine
echo "Checking ${_bzl_out}/tools/logging"
ls -l "${_bzl_out}/tools/logging"
# dist/ missing writable permission, node_modules/ (a bazel output via Rules JS) also shown for comparison
# dr-xr-xr-x  2 _user_  _group_    64 15 Aug 16:00 dist
# drwxr-xr-x  5 _user_  _group_   160 10 Sep 11:29 node_modules
echo "Checking ${_bzl_out}/tools/logging/dist"
ls -l "${_bzl_out}/tools/logging/dist"
# Empty (as expected)
echo "Checking ${_bzl_out}/tools/logging/dist/logger.js"
ls -l "${_bzl_out}/tools/logging/dist/logger.js"
# Missing (as expected)

--remote_download_all may offer a workaround (haven't been able to confirm) however a bazel clean would be necessary for it to have any affect (so the directory can be recreated).

The fix (backported for 7.4.0 via #23577) likely will solve this issue 🤞.

[Silic0nS0ldier]

Had the issue manifest again and confirmed the backport worked.

[iancha1992]

A fix for this issue has been included in Bazel 7.4.0 RC1. Please test out the release candidate and report any issues as soon as possible.
If you're using Bazelisk, you can point to the latest RC by setting USE_BAZEL_VERSION=7.4.0rc1. Thanks!