Bazel fails build if Remote Cache returns AC but not CAS

[artem-zinnatullin]

Description of the problem / feature request:

We have Remote Build Cache for Bazel that can't guarantee that between request for AC and CAS for a given key CAS won't be evicted from the cache.

It seems that this can result in Bazel build failing with following error:

ERROR: /a/b/c/BUILD:3:21: output 'a/b/c/lib_base_symbols/symbols.zip' was not created
ERROR: /a/b/c/BUILD:3:21: not all outputs were created or valid

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I think the issue happens when Remote Cache returns AC (metadata) but by the time Bazel requests CAS, it's evicted from the remote cache so it returns 404 and Bazel fails because it doesn't have output for an action.

What operating system are you running Bazel on?

macOS, Linux.

What's the output of bazel info release?

release 3.6.0

Have you found anything relevant by searching the web?

• not all outputs were created or valid (Remote caching, all platforms) #8125 but I think it's a bit different case

---

My assumption might be wrong, but if it was right, I think Bazel should not expect CAS to be guaranteed if AC was returned and gracefully run the action locally to generate the output.

[artem-zinnatullin]

Ah, I just noticed #8250, which is basically same issue

Please feel free to close this one as duplicate, if you think it is

[ulfjack]

What flags are you using for that build?

[keith]

@ulfjack which flags specifically are you wondering about?

[cocreature]

In case it’s a useful data point: We regularly (unfortunately) encounter the same error message on our Windows CI nodes during Haskell builds (and only there). I tried to verify that this is caused y this issue however, it seems like like deleting CAS outputs (or overwriting them with empty files) results in a BuilkTransferException for us and it gets rebuild locally gracefully and not the error mentioned above. We are not using remote builds without the bytes so in case this is really the same as #8250, it would explain why we see different behavior.

[github-actions]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 2+ years. It will be closed in the next 14 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

[coeuvre]

This bug has been fixed in recent releases.