"I/O exception during sandboxed execution" when using tree artifacts

[scele]

Description of the problem:

When generating an output artifact at a path that was previously a tree artifact, the incremental build fails with "I/O exception during sandboxed execution: Could not move output artifacts from sandboxed execution".

It seems that bazel makes the tree artifact directory read-only. A subsequent incremental build (where the tree artifact no longer exists, but a normal artifact would need to be created in that directory), can then fail with a permission error.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Repro script:

#!/bin/bash
set -ex

rm -rf bazel_test
mkdir bazel_test
cd bazel_test
echo > WORKSPACE
cat > defs.bzl <<EOF
def _rule1_impl(ctx):
    d = ctx.actions.declare_directory(ctx.attr.name)
    ctx.actions.run_shell(
        command = "echo foo > %s/bar.txt" % d.path,
        outputs = [d],
    )
    return [DefaultInfo(files = depset([d]))]

def _rule2_impl(ctx):
    d = ctx.actions.declare_file(ctx.attr.name + "/foo.txt")
    ctx.actions.run_shell(
        command = "echo foo > %s" % d.path,
        outputs = [d],
    )
    return [DefaultInfo(files = depset([d]))]

rule1 = rule(
    implementation = _rule1_impl,
)

rule2 = rule(
    implementation = _rule2_impl,
)
EOF
for x in "4.0.0" "last_green"; do
    export USE_BAZEL_VERSION=$x
    bazel version
    bazel clean
    cat > BUILD <<EOF
load("//:defs.bzl", "rule1")

rule1(
    name = "target",
)
EOF
    bazel build :target
    cat > BUILD <<EOF
load("//:defs.bzl", "rule2")

rule2(
    name = "target",
)
EOF
    bazel build :target || true
    cp $(bazel info server_log) ./java.$x.log
done

What operating system are you running Bazel on?

Ubuntu 18.04.3 LTS

What's the output of bazel info release?

Build label: 4.0.0
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Jan 21 07:33:24 2021 (1611214404)
Build timestamp: 1611214404
Build timestamp as int: 1611214404

Also reproduces with bazelisk USE_BAZEL_VERSION=last_green (commit 055c93d11ab20cc4479539b24bbdfa5cab78a342).

Have you found anything relevant by searching the web?

No.

Any other information, logs, or outputs that you want to share?

java.log shows a permission error:

Caused by: java.io.IOException: Could not move output artifacts from sandboxed execution
>       at com.google.devtools.build.lib.sandbox.AbstractSandboxSpawnRunner.runSpawn(AbstractSandboxSpawnRunner.java:132)
>       at com.google.devtools.build.lib.sandbox.AbstractSandboxSpawnRunner.exec(AbstractSandboxSpawnRunner.java:88)
>       ... 22 more
Caused by: com.google.devtools.build.lib.vfs.FileAccessException: <redacted>/execroot/__main__/bazel-out/k8-fastbuild/bin/target/foo.txt (Permission denied)
>       at com.google.devtools.build.lib.unix.NativePosixFiles.openWrite(Native Method)
>       at com.google.devtools.build.lib.unix.UnixFileSystem.createFileOutputStream(UnixFileSystem.java:483)
>       at com.google.devtools.build.lib.vfs.AbstractFileSystem.getOutputStream(AbstractFileSystem.java:128)
>       at com.google.devtools.build.lib.vfs.Path.getOutputStream(Path.java:533)
>       at com.google.devtools.build.lib.vfs.Path.getOutputStream(Path.java:521)
>       at com.google.devtools.build.lib.vfs.FileSystemUtils.moveFile(FileSystemUtils.java:458)
>       at com.google.devtools.build.lib.sandbox.SandboxHelpers.moveOutputs(SandboxHelpers.java:132)
>       at com.google.devtools.build.lib.sandbox.AbstractContainerizingSandboxedSpawn.copyOutputs(AbstractContainerizingSandboxedSpawn.java:199)
>       at com.google.devtools.build.lib.sandbox.AbstractSandboxSpawnRunner.runSpawn(AbstractSandboxSpawnRunner.java:130)
>       ... 23 more