Build event file creation fails with symlinks when using --remote_download_minimal

[exoson]

Description of the problem / feature request:

If generating build event file for a target which creates a file and a symlink to this file with --remote_download_minimal and the target exists in remote cache, BEP upload code will fail with ERROR: Unable to write all BEP events to file due to 'java.io.FileNotFoundException: /home/exoson/.cache/dazel/_dazel_exoson/5d75fbbb8d44bec03b8dc3734a1ad976/execroot/test_repo/bazel-out/k8-py3-fastbuild/bin/asdf_link (No such file or directory)'

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Have symlink.bzl

def _symlink_impl(ctx):
    out = ctx.actions.declare_file(ctx.label.name)
    ctx.actions.run_shell(
        command = "echo 'test' > $@",
        arguments = [out.path],
        outputs = [out],
    )
    link = ctx.actions.declare_symlink(ctx.label.name + "_link")
    ctx.actions.symlink(
        output = link,
        target_path = out.path,
    )
    return DefaultInfo(files = depset([link, out]))

symlink = rule(
    implementation = _symlink_impl,
)

and BUILD

load("//:symlink.bzl", "symlink")

symlink(
    name = "asdf",
)

Run

# First build the target and cache everything remotely
dazel build --experimental_allow_unresolved_symlinks --remote_cache=grpc://127.0.0.1:8980 //:asdf

# Clean so the files won't exist locally anymore
dazel clean

# Run the build with --remote_download_minimal while generating a build even file.
dazel build --experimental_allow_unresolved_symlinks --remote_cache=grpc://127.0.0.1:8980 --build_event_json_file=be.json  --remote_download_minimal //:asdf

What operating system are you running Bazel on?

Ubuntu 18.04

If bazel info release returns "development version" or "(@non-git)", tell us how you built Bazel.

Ran the installation script built with this

git checkout master
git pull
bazel build //scripts/packages:with-jdk/install.sh

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

https://github.com/bazelbuild/bazel.git
40e7636
40e7636

Have you found anything relevant by searching the web?

#6862 (comment)
#11119

Any other information, logs, or outputs that you want to share?

Seems like when using --remote_download_minimal symlinks are created while the file linked to isn't and thus BEP upload code will fail when trying to compute the digest for the symlinked file.

bazel/src/main/java/com/google/devtools/build/lib/remote/ByteStreamBuildEventArtifactUploader.java

Line 129 in 40e7636

Digest digest = digestUtil.compute(file);

[coeuvre]

The example fails in the second build because:

1. There are two actions required to run for target //:asdf:

   1. actions.run_shell which output asdf and can be executed remotely;
   2. actions.symlink which output asdf_link.

2. When remote cache enabled with minimal mode, it

   Generally don't download remote action outputs. The only outputs that are being downloaded are: stdout, stderr and .d and .jdeps files for C++ and Java compilation actions.

   asdf is not download.

3. A TargetCompleteEvent is posted which will upload referenced local files, in this case asdf and asdf_link, by reading the metadata of those files, using RemoteActionFileSystem.

4. The RemoteActionFileSystem

   mostly delegates to the local file system except for the case where an action input is a remotely stored action output.

5. For asdf, it read the metadata from remote cache, but for asdf_link it delegates to local file system which fellows the symlink. The file asdf is not downloaded, so FileNotFoundException.

Symlinks in action outputs are not yet supported with --remote_download_minimal. I am not sure the right solution to this.

[ulfjack]

I think this might be a duplicate of #11532, for which I have a pending PR.

[exoson]

I think this might be a duplicate of #11532, for which I have a pending PR.

I think this would be different as this problem exists even when the symlinks are not toplevel.

[ulfjack]

@exoson, you're probably right. I am thinking that RemoteActionFS should not delegate to the underlying file system in this case because that can't resolve symlinks. Or maybe we shouldn't be creating dangling symlinks on the underlying file system, but instead create them in RemoteActionFS?

[tjgq]

This became a warning instead of an error at some point between 4.x and 5.x (likely in e855a26).