Windows, CI: getting "Failed to delete output files after incomplete download"

[laszlocsomor]

Description of the problem / feature request:

On Bazel CI, I'm repeatedly getting the following error:

ERROR: D:/b/bk-worker-windows-java8-w528/bazel/google-bazel-presubmit/third_party/protobuf/3.6.1/BUILD:123:1: Couldn't build file third_party/protobuf/3.6.1/_objs/protobuf_lite/stringpiece.obj: C++ compilation of rule '//third_party/protobuf/3.6.1:protobuf_lite' failed: Failed to delete output files after incomplete download. Cannot continue with local execution.: D:/b/f622aej4/execroot/io_bazel/bazel-out/host/bin/third_party/protobuf/3.6.1/_objs/protobuf_lite/stringpiece.obj (Permission denied)
--
  | Target //src:bazel failed to build

(https://buildkite.com/bazel/google-bazel-presubmit/builds/12883#61fdc327-7d64-4979-a3b4-9d35f2c729a2)

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I don't know. I saw the error on CI for my change https://bazel-review.googlesource.com/c/bazel/+/84151.

Retrying the job doesn't seem to help.

Looks like this is where the error comes from:

bazel/src/main/java/com/google/devtools/build/lib/remote/AbstractRemoteActionCache.java

Lines 280 to 284 in b3f28d3

	// We don't propagate the downloadException, as this is a recoverable error and the cause
	// of the build failure is really that we couldn't delete output files.
	throw new EnvironmentalExecException(
	"Failed to delete output files after incomplete "
	+ "download. Cannot continue with local execution.",

And I suppose the reason is that outErr has open streams for stdout and stderr, and those streams should be closed before attempting to delete the files here:

bazel/src/main/java/com/google/devtools/build/lib/remote/AbstractRemoteActionCache.java

Lines 273 to 274 in b3f28d3

	outErr.getOutputPath().delete();
	outErr.getErrorPath().delete();

What operating system are you running Bazel on?

Replace this line with your answer.

What's the output of bazel info release?

0.20.0

[laszlocsomor]

/cc @ola-rozenfeld @buchgr

[laurentlb]

This is blocking a Bazel release: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/695#e741bbc1-151b-479b-8af0-72bf7388150d

I'm not even able to test downstream projects with the Bazel candidate, so this can hide unrelated bugs.

[laurentlb]

If it can be fixed or worked around on Monday, we still have a chance of release Bazel before the holiday.

[laszlocsomor]

If it can be fixed or worked around on Monday, we still have a chance of release Bazel before the holiday.

Is that what we want? Will we be around to push a patch release if the main release turns out to be bad?

[laszlocsomor]

I have a repro: #6945 (comment)

[laszlocsomor]

I'll start bisecting. Apparently this is a regression in 0.20.0, because I cannot repro the bug with 0.19.0

[laszlocsomor]

The good news is, I found the culprit: d2920e3

The bad news is, it was a rollback of 1a95502, which caused problems.

So we cannot be with or without this feature, i.e. of opening files with deletion-sharing on Windows.

The principled fix would be for the AbstractRemoteActionCache to close files before it tries to delete them.

[laszlocsomor]

FYI @philwo

[meteorcloudy]

Wow, thanks for debugging!

[philwo]

Note to self: Mark Google bug b/121159713 as fixed, when we fixed this to let my colleagues know when this is working again.

Discussions and progress tracking should happen here.