Bazel should accept ALREADY_EXISTS for byte stream uploads

[ulfjack]

Description of the problem / feature request:

com.google.devtools.build.lib.remote.ByteStreamUploader only treats uploads as successful when they return an OK status. Naively, this seems fine - Bazel first checks if the file exists remotely and then uploads it. However, this is a check-then-act race, where either another thread in the same Bazel process or another Bazel process may finish an upload of the same file between the check and the upload (or even while the upload is ongoing). In that case, the CAS service might return ALREADY_EXISTS for this file, which Bazel currently treats as a non-retryable error.

While the documentation does not clearly indicate what the CAS service should do if a file already exists, the ALREADY_EXISTS return code seems appropriate.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Build a CAS service that returns ALREADY_EXISTS and run a build where multiple actions have the same input. Bazel currently does not deduplicate uploads.

ERROR: /home/ulfjack/Google/bazel2/foo/BUILD:15:9: Executing genrule //foo:bar-0 failed (Exit 34): com.google.devtools.build.lib.remote.BulkTransferException
	at com.google.devtools.build.lib.remote.RemoteCache.waitForBulkTransfer(RemoteCache.java:225)
...
	Suppressed: java.io.IOException: io.grpc.StatusRuntimeException: ALREADY_EXISTS: Digest ca6e692227cd35f862d56f0ef4cb0a3ba06cc86674cff63d8b34a5a35d4999a2/159 already exists
		at com.google.devtools.build.lib.remote.ByteStreamUploader.lambda$uploadBlobAsync$1(ByteStreamUploader.java:247)
		at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.doFallback(AbstractCatchingFuture.java:175)
		at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.doFallback(AbstractCatchingFuture.java:162)
		at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:107)
...
		at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
		at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
		at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
		... 3 more
	Caused by: io.grpc.StatusRuntimeException: ALREADY_EXISTS: Digest ca6e692227cd35f862d56f0ef4cb0a3ba06cc86674cff63d8b34a5a35d4999a2/159 already exists
		at io.grpc.Status.asRuntimeException(Status.java:523)
		... 23 more

What operating system are you running Bazel on?

Linux.