build without the bytes: "dangling symbolic link" error for symlink output

[william-smith-skydio]

Description of the bug:

I have an action with an output that is a symlink to another output of the same action. When build without the bytes is enabled (e.g. --remote_download_toplevel or --remote_download_minimal), the symlink gets downloaded, but the target does not, resulting in bazel failing due to a "dangling symbolic link".

What category does this issue belong to?

Remote Execution

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

This is a simple action which produces two outputs, one of which is a symlink to the other:

genrule(
    name = "foobar",
    outs = [
        "a",
        "b",
    ],
    cmd = "touch $(location a) && ln -s a $(location b)",
)

This succeeds and populates the disk cache:

bazel build //:foobar --disk_cache /tmp/cache --remote_download_toplevel

Then this fails:

bazel clean
bazel build //:foobar --disk_cache /tmp/cache --remote_download_toplevel

Error:

ERROR: /<redacted>/BUILD.bazel:1:8: declared output 'b' is a dangling symbolic link
ERROR: /<redacted>/BUILD.bazel:1:8: Executing genrule //:foobar failed: not all outputs were created or valid

The output directory contains just a broken b symlink, no a in sight.

Which operating system are you running Bazel on?

Ubuntu 18.04

What is the output of bazel info release?

6.3.1

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

This reproduces in bazel 6.3.0 and 6.3.1, but not in 6.2.1 or 7.0.0-pre.20230710.5.

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

I have shown an example with a genrule and disk cache, but this also reproduces with a rule with declared outputs and a proper remote cache setup.

[coeuvre]

This has been fixed by Bazel@HEAD and due to massive changes to BwoB recently, it's unlikely we can cherry-pick the fix to 6.x.

cc @tjgq

[brentleyjones]

@coeuvre Would a localized fix be possible? The report says that it only broke in 6.3.x, but 6.2.1 worked, so this is minor release regression?

[tjgq]

@brentleyjones I will take a look to see if a simple fix is possible.

[joeljeske]

6.4 is slated to go out soon. Is there any update on the feasibility of fixing this 6.x regression?

[tjgq]

A fix has been submitted into the 6.4.0 release branch.

[iancha1992]

A fix for this issue has been included in Bazel 6.4.0 RC2. Please test out the release candidate and report any issues as soon as possible. Thanks!