Cache uploads shouldn't block dependent local actions

[brentleyjones]

Description of the problem / feature request:

Currently cache uploads take place as the last part of a local action spawn. This ties up the local_cpu_resources, and also prevents dependent local action spawns from starting (even though they have the inputs they need now).

Instead, the upload of local actions outputs to the cache should be independent spawns, that don't count towards local_cpu_resources, and don't block dependent local actions. Remote actions that need the output as input (found via FindMissingBlobs) should still wait for the upload to finish, including the already started upload with the other inputs in RemoteExecutionCache.ensureInputsPresent().

Feature requests: what underlying problem are you trying to solve with this feature?

By doing this, a non-remote build can complete a lot faster, as it no longer has to wait for uploads to finish to move onto the next action. These uploads will no longer take up local_cpu_resources, allowing for more parallel local actions (as long as --jobs is set higher than --local_cpu_resources). The end of the build can report that it's waiting for uploads to finish, similar to what it does today for BES and Profile uploading.

What operating system are you running Bazel on?

macOS 11.4

What's the output of bazel info release?

4.1.0 and HEAD

Have you found anything relevant by searching the web?

• Bazel remote cache is not a clear win #7664
• https://github.com/buildbuddy-io/buildbuddy-cli/blob/171585fc6a5af723f418c935990e97909cd62f8f/cache_proxy/cache_proxy.go#L37-L44
• https://github.com/Asana/bazels3cache#main-features

[coeuvre]

Thanks for bringing this up! I will look into it!

BTW, I had similar idea recently but for downloads i.e. downloading the outputs of a remote action shouldn't block dependent remote actions. #13630 is working towards this feature.

[coeuvre]

Fixed by #13655.