Occasionally missing output files from remote cache with BwtB

[joeljeske]

Description of the bug:

After bumping from Bazel 6.4.0 -> 7.1.1, I see that after Bazel completes a build with remote_download_topelevel, sometimes there are one or more missing files that should have been downloaded from RBE/RC. It is inconsistent which files are missing, and do not have a pattern in mnemonic or file extension. If I retry the same build on that machine immediately after, the file is still missing.

Which category does this issue belong to?

Remote Execution

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I do not have a consistent repro yet.

Which operating system are you running Bazel on?

Ubuntu

What is the output of bazel info release?

release 7.1.1

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

Yes. I did not see these errors in 6.4.0

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

[brentleyjones]

Maybe #20737 is related? Does --noexperimental_merged_skyframe_analysis_execution fix it for you?

[joeljeske]

Ah interesting. Let me give it a shot.

[joeljeske]

Yes, @brentleyjones you are right. Flipping that flag off has eliminated this class of errors. Thank you!

Perhaps #20737 should be re-titled to not appear BEP-specific, but rather any output files could be missing?

[brentleyjones]

cc: @coeuvre

[tjgq]

Pretty sure this is a dupe of #20737.