Bazel 7.1.1 re-runs analysis and re-fetches external repositories during each invocations

[ivan-golub]

Description of the bug:

After upgrade from bazel 7.0.2 to bazel 7.1.1 seeing an issue where bazel analysis cache is lost between invocations and "Analyzing: target"/"checking cached actions"/fetching external repositories run on each invocation vs being retained in-memory. Server pid doesnt check and there are no logs like WARNING: something something has changed, discarding analysis cache (this can be expensive, see https://bazel.build/advanced/performance/iteration-speed)

Reverting 18ba449 and related bd4bddf solves this issue

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

No response

Which operating system are you running Bazel on?

macOS

What is the output of bazel info release?

7.1.1

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

18ba449

Any other information, logs, or outputs that you want to share?

Logs from running same build command several times in a row

Expected (7.0.2 behavior)

INFO: Invocation ID: c6749968-4795-4107-a952-b03bb0899d56
Computing main repo mapping: 
Loading: 
Loading: 0 packages loaded
Analyzing: target //Apps/Foo:Foo (0 packages loaded, 0 targets configured)
Analyzing: target //Apps/Foo:Foo (0 packages loaded, 0 targets configured)
[0 / 1] [Prepa] BazelWorkspaceStatusAction stable-status.txt
INFO: Analyzed target //Apps/Foo:Foo (0 packages loaded, 0 targets configured).
[28,051 / 37,230] checking cached actions
INFO: Found 1 target...
Target //Apps/Foo:Foo up-to-date:
  bazel-bin/Apps/Foo/Foo.app
INFO: Elapsed time: 8.511s (0.14m), Critical Path: 0.56s (0.01m)
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action

Actual (7.1.1):

INFO: Invocation ID: baf1cdaa-c43e-4a4e-9301-1df575e25f64
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Loading: 
Loading: 2 packages loaded
Analyzing: target //Apps/Foo:Foo (3 packages loaded)
Analyzing: target //Apps/Foo:Foo (3 packages loaded, 0 targets configured)
Analyzing: target //Apps/Foo:Foo (3 packages loaded, 0 targets configured)
[0 / 1] [Prepa] BazelWorkspaceStatusAction stable-status.txt
Analyzing: target //Apps/Foo:Foo (4815 packages loaded, 55237 targets configured)
[1 / 1] checking cached actions
Analyzing: target //Apps/Foo:Foo (9950 packages loaded, 134275 targets configured)
[1 / 1] checking cached actions
Analyzing: target //Apps/Foo:Foo (10023 packages loaded, 155924 targets configured)
[1 / 1] checking cached actions
Analyzing: target //Apps/Foo:Foo (10244 packages loaded, 163078 targets configured)
[1 / 1] checking cached actions
Analyzing: target //Apps/Foo:Foo (10244 packages loaded, 163078 targets configured)
[1 / 1] checking cached actions
Analyzing: target //Apps/Foo:Foo (10244 packages loaded, 163078 targets configured)
[1 / 1] checking cached actions
Analyzing: target //Apps/Foo:Foo (10247 packages loaded, 163483 targets configured)
[1 / 1] checking cached actions
Analyzing: target //Apps/Foo:Foo (10247 packages loaded, 163483 targets configured)
[1 / 1] checking cached actions
Analyzing: target //Apps/Foo:Foo (10248 packages loaded, 180673 targets configured)
[1 / 1] checking cached actions
Analyzing: target //Apps/Foo:Foo (10248 packages loaded, 180673 targets configured)
[1 / 1] checking cached actions
Analyzing: target //Apps/Foo:Foo (10248 packages loaded, 180673 targets configured)
[1 / 1] checking cached actions
Analyzing: target //Apps/Foo:Foo (10248 packages loaded, 180673 targets configured)
[1 / 1] checking cached actions
Analyzing: target //Apps/Foo:Foo (10248 packages loaded, 180673 targets configured)
[1 / 1] checking cached actions
Analyzing: target //Apps/Foo:Foo (10248 packages loaded, 180673 targets configured)
[1 / 1] checking cached actions
Analyzing: target //Apps/Foo:Foo (10248 packages loaded, 180673 targets configured)
[1 / 1] checking cached actions
Analyzing: target //Apps/Foo:Foo (10248 packages loaded, 180673 targets configured)
[1 / 1] checking cached actions
Analyzing: target //Apps/Foo:Foo (10248 packages loaded, 180673 targets configured)
[1 / 1] checking cached actions
Analyzing: target //Apps/Foo:Foo (10248 packages loaded, 180673 targets configured)
[1 / 1] checking cached actions
Analyzing: target //Apps/Foo:Foo (10248 packages loaded, 180673 targets configured)
[1 / 1] checking cached actions
Analyzing: target //Apps/Foo:Foo (10251 packages loaded, 207048 targets configured)
[1 / 1] checking cached actions
INFO: Analyzed target //Apps/Foo:Foo (10251 packages loaded, 236964 targets configured).
[64,876 / 92,099] [Scann] Compiling Features/Bar/Sources/Bar.m
[72,967 / 92,099] [Scann] Compiling Libraries/Baz/Internal/Baz.m
[77,410 / 92,099] [Scann] Compiling Features/Qux/Sources/Qux.m
[81,359 / 92,099] checking cached actions
[83,679 / 92,099] checking cached actions
[85,643 / 92,099] checking cached actions
INFO: Found 1 target...
Target //Apps/Foo:Foo up-to-date:
  bazel-bin/Apps/Foo/Foo.app
INFO: Elapsed time: 69.108s (1.15m), Critical Path: 1.58s (0.03m)
INFO: 11 processes: 10 remote cache hit, 1 internal.
INFO: Build completed successfully, 11 total actions

[iancha1992]

@ivan-golub Could you please provide complete steps to reproduce this issue?

[ivan-golub]

Trying to get a repro project but its not trivial, I can repro within project with a lot of external dependencies but not in isolated new project. Added additional logs to issue description

[dieortin]

I'm hitting the same problem with dependencies built with rules_foreign_cc. They are discarded and rebuilt every single time, even if I do not modify anything. In fact, the sources I'm building with rules_foreign_cc come from a downloaded .tar.gz archive which I have no easy way to modify.

This happens on versions 7.1.0 and 7.1.1, but not on 7.0.2 or earlier. I'm having to revert to an older version for the moment, as this is killing our build times.

[fmeum]

@dieortin Could you share the rules_foreign_cc target reproducing this behavior? We do still lack a reproducer for this issue.

[kcon-stackav]

I was seeing this issue in 7.1.1 too, and for me it seems to be #22118 where my .netrc file's contents were changing on every CI job run due to me using an ephemeral token there, and that caused certain external repos to be unnecessarily refetched at the start of every CI job. Upgrading to 7.1.2 (which has a fix for that issue via #22186) resolves the issue for me.

[dieortin]

My issue turned out to be related to artifacts produced by rules_foreign_cc not being deterministic, as some build IDs end up in its build log file.
I’m not entirely sure about why the action was not cached, as the inputs were still the same and Bazel should have no way of knowing the action is not deterministic. Anyways, I solved it by not depending on the entire folder generated by rules_foreign_cc, but only on one binary.