Bazel hangs on INFO: Invocation ID while fetching dependencies

[fmeum]

Description of the bug:

I am regularly (but not reproducibly, not even very frequently) experiencing Bazel hanging for a long time (> 1 min) after printing the initial invocation ID, but before printing the current date. According to the recorded profile, it is fetching a toolchain repo, but doesn't print any progress information.

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I haven't been able to produce a reliable reproducer yet. I am observing this in a medium-size monorepo that fetches large hermetic toolchains.

Which operating system are you running Bazel on?

Linux

What is the output of bazel info release?

6.0.0rc4

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

Happy to provide additional information, I just don't know what kind would be helpful.

[meteorcloudy]

@fmeum Have you noticed similar issue with Bazel 5.x releases?

[fmeum]

@meteorcloudy I am fairly certain that I haven't seen it with 5.0.0, but since we switched to rolling releases after that, I unfortunately can't tell whether it is present in 5.X or not. I also can't easily run a full bisect since the monorepo uses relatively recent Bazel features.

I will investigate this further. It isn't too bad since it's essentially only a UI issue.

[meteorcloudy]

OK, then I'll not consider it as a release blocker for 6.0 for now, if we figure out a fix we can cherry pick it later into 6.x.

[fmeum]

I can now reliably reproduce this by modifying the sha256 of a large toolchain fetched via repository_ctx.download_and_extract. Could this be related to the changes @Wyverald made to that function? I will try to come up with a minimal reproducer.

[fmeum]

Turned out to be a dupe of #14215, I didn't notice that a Starlark flag was added.

[fmeum]

@meteorcloudy Nevermind, this is a regression and #14215 provides the reproducer for it (https://github.com/fmeum/bazel-hangs-on-startup). If you run the example linked there, you get:

❯ USE_BAZEL_VERSION=5.3.2 bazel build //...
Starting local Bazel server and connecting to it...
INFO: Invocation ID: eb68a9c3-d82e-48eb-8f98-b12c1162c277
Loading: 0 packages loaded
    Fetching @hangs_on_fetch; Will sleep now...
❯ USE_BAZEL_VERSION=6.0.0rc4 bazel build //...
Starting local Bazel server and connecting to it...
INFO: Invocation ID: d7163491-5864-4234-aa87-390f7ffc0655
<hangs indefinitely, interrupt with Ctrl-C>
Bazel caught interrupt signal; shutting down.

ERROR: command interrupted while computing main repo mapping

So this appears to be different variant of #14215 with the same root cause: The loading phase is never properly started since the fetch is triggered by some preparatory phase (option parsing, main repo mapping computation). Just that with Bazel 6, this can trigger even without Starlark flags.

Sorry for the back and forth, all the flags in our repo made this somewhat confusing.

[fmeum]

The first commit this reproduces with is 943e8cf.

[meteorcloudy]

/cc @Wyverald

[meteorcloudy]

While this is indeed a regression in 6.0, but as @fmeum analysed, this is only a UI issue and a proper fix probably requires more time, so let's not block the 6.0.0 release in the last minute. We'll fix this in later 6.x.x releases.

[meteorcloudy]

@bazel-io fork 6.1.0

[Wyverald]

I think this might be a dupe of #16338. Something around the event reporting during the initial main repo mapping computation must be broken.

[rpwoodbu]

The eagerly-loaded issue referenced above is a significant clue, I believe. My observation is that this bug happens only if a .bzl file from the external repository is load()-ed. If all you are doing is referencing targets (e.g., as deps), status output is reported properly.

I discovered this while cloning a large git repo with git_repository(). The issue showed in one Bazel workspace, but not another, although they both were using git_repository() to fetch the same git repo, and were both running 6.0.0. The difference was that one needed to load() some things from it, and the other didn't.

I can confirm that this bug does not happen in 5.2.0, but does in 6.0.0, and still does in the latest 7.0.0 prerelease.