Bazel crashes with InterruptedException when certain benign errors happen #14787

wisechengyi · 2022-02-10T19:47:58Z

Slack discussion: https://bazelbuild.slack.com/archives/CA31HN1T3/p1644524174413049

Description of the problem:

bazel error with piped

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

% bazel query tools/... | head
Loading: 0 packages loaded
//tools:bzl_srcs
//tools:embedded_tools_srcs
//tools:srcs
//tools:upload_bazel_mirror
//tools/allowlists:srcs
//tools/allowlists/config_feature_flag:srcs
//tools/allowlists/function_transition_allowlist:srcs
//tools/android:aar_embedded_jars_extractor
//tools/android:aar_embedded_jars_extractor_test
//tools/android:aar_embedded_proguard_extractor

Cannot write to standard output; exiting...

Loading: 0 packages loaded
Loading: 0 packages loaded

What operating system are you running Bazel on?

macos, but also observed similar issue on linux

$ ./bazel query tools/... | head
INFO: Invocation ID: 8fed028a-18a6-4a6c-a0b2-5a0dcfed396b
Loading: 0 packages loaded
...

Cannot write to standard output; exiting...

Loading: 0 packages loaded
FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.RuntimeException: java.lang.InterruptedException: DefaultPromise@502c56e6(incomplete)
	at io.netty.channel.pool.FixedChannelPool.close(FixedChannelPool.java:454)
	at com.google.devtools.build.lib.remote.http.HttpCacheClient.close(HttpCacheClient.java:736)
	at com.google.devtools.build.lib.remote.disk.DiskAndRemoteCacheClient.close(DiskAndRemoteCacheClient.java:71)
	at com.google.devtools.build.lib.remote.RemoteCache.deallocate(RemoteCache.java:454)
	at io.netty.util.AbstractReferenceCounted.handleRelease(AbstractReferenceCounted.java:86)
	at io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76)
	at com.google.devtools.build.lib.remote.RemoteActionContextProvider.afterCommand(RemoteActionContextProvider.java:217)
	at com.google.devtools.build.lib.remote.RemoteModule.afterCommand(RemoteModule.java:862)
	at com.google.devtools.build.lib.runtime.BlazeRuntime.afterCommand(BlazeRuntime.java:626)
	at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:609)
	at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:231)
	at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:550)
	at com.google.devtools.build.lib.server.GrpcServerImpl.lambda$run$1(GrpcServerImpl.java:614)
	at io.grpc.Context$1.run(Context.java:579)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.InterruptedException: DefaultPromise@502c56e6(incomplete)
	at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:244)
	at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:35)
	at io.netty.channel.pool.FixedChannelPool.close(FixedChannelPool.java:451)
	... 16 more

What's the output of `bazel info release`?

release 5.0.0

If `bazel info release` returns "development version" or "(@non-git)", tell us how you built Bazel.

Replace this line with your answer.

What's the output of `git remote get-url origin ; git rev-parse master ; git rev-parse HEAD` ?

% git remote get-url origin ; git rev-parse master ; git rev-parse HEAD
https://github.com/bazelbuild/bazel
1deb3f7aa22ec322786360085f9eb723e624d7c7
1deb3f7aa22ec322786360085f9eb723e624d7c7

Have you found anything relevant by searching the web?

None yet

Other info

writing to a file works.

% bazel query tools/... > xxx && wc -l xxx
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
     282 xxx
% echo $?
0

The text was updated successfully, but these errors were encountered:

brentleyjones · 2022-02-10T20:45:44Z

@bazel-io flag

brentleyjones · 2022-02-10T20:46:24Z

Does this work in 4.2.2?

wisechengyi · 2022-02-10T21:01:06Z

was on 4.2.1, and it seems to have this issue as well.

$ ./bazel version
Build label: 4.2.1
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Mon Aug 30 15:17:47 2021 (1630336667)
Build timestamp: 1630336667
Build timestamp as int: 1630336667
$ ./bazel query tools/... | head
INFO: Invocation ID: 5793bfb8-1943-4b50-8ac3-d200bd81902c
Loading: 0 packages loaded
//tools/transitive-deps-updater/src/test/scala/com/twitter/transitiveDepsUpdater:tests
//tools/transitive-deps-updater/src/test/resources:resources
//tools/transitive-deps-updater/src/test/resources:resources-scala
//tools/transitive-deps-updater/src/test/resources:resources-py
//tools/transitive-deps-updater/src/main/scala/com/twitter/transitiveDepsUpdater:transitiveDepsUpdater
//tools/transitive-deps-updater/src/main/scala/com/twitter/transitiveDepsUpdater:transitiveDepsUpdater.semanticdb
//tools/toolchains:testing_toolchains_with_all_deps_impl
//tools/toolchains:testing_toolchain
//tools/toolchains:specs2_junit_toolchain_impl
//tools/toolchains:specs2_junit_toolchain

Cannot write to standard output; exiting...

Loading: 0 packages loaded
INFO: Build Event Protocol files produced successfully.
Loading: 0 packages loaded

however exit code seems to be 0.

ulfjack · 2022-02-10T22:56:46Z

Bazel writes progress and error messages to stderr while writing 'normal' query output to stdout. That looks weird when you redirect stdout (and keep stderr), but it is working as expected!

Try running it like so:
bazel query --noshow_progress tools/... | head

Or like so:
bazel query tools/... 2>/dev/null | head

Or like so (if you want to see the progress messages):
bazel query tools/... 2>&1 | head

This is 'standard' Unix behavior, although it is a bit unusual for a tool to generate that much stderr if it runs successfully.

wisechengyi · 2022-02-10T23:23:42Z

Ok to clarify, the issue isn't

Cannot write to standard output; exiting...

but the exception which then causes bazel to crash:

FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.RuntimeException: java.lang.InterruptedException: DefaultPromise@6fac5e26(incomplete)
	at io.netty.channel.pool.FixedChannelPool.close(FixedChannelPool.java:454)
	at com.google.devtools.build.lib.remote.http.HttpCacheClient.close(HttpCacheClient.java:736)
	at com.google.devtools.build.lib.remote.disk.DiskAndRemoteCacheClient.close(DiskAndRemoteCacheClient.java:71)
	at com.google.devtools.build.lib.remote.RemoteCache.deallocate(RemoteCache.java:454)
	at io.netty.util.AbstractReferenceCounted.handleRelease(AbstractReferenceCounted.java:86)
	at io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76)
	at com.google.devtools.build.lib.remote.RemoteActionContextProvider.afterCommand(RemoteAction
...

and then subsequent bazel invocation will have to start a new server.

So it seems to be remain as a serious issue.

And the issue did start with release 5.0.0

Wyverald · 2022-02-11T13:10:49Z

Hmm. This might be a new thing in 5.0.0 after all. @coeuvre does this look like an issue with remote execution?

eed3si9n · 2022-02-11T16:56:23Z

I regularly run bazel query ... | head so initially I thought pipe was doing something wrong, but now I see that this also happens when I have a typo in a custom bzl file. (OP @wisechengyi and I work in the same team)

ERROR: /xxx/build_rules/targets/something_library.bzl:63:19: name 'merged' is not defined (did you mean 'merged0'?)

FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.RuntimeException: java.lang.InterruptedException: DefaultPromise@2eb04551(incomplete)
	at io.netty.channel.pool.FixedChannelPool.close(FixedChannelPool.java:454)
	at com.google.devtools.build.lib.remote.http.HttpCacheClient.close(HttpCacheClient.java:736)
	at com.google.devtools.build.lib.remote.disk.DiskAndRemoteCacheClient.close(DiskAndRemoteCacheClient.java:71)
	at com.google.devtools.build.lib.remote.RemoteCache.deallocate(RemoteCache.java:454)
	at io.netty.util.AbstractReferenceCounted.handleRelease(AbstractReferenceCounted.java:86)
	at io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76)
	at com.google.devtools.build.lib.remote.RemoteActionContextProvider.afterCommand(RemoteActionContextProvider.java:217)
	at com.google.devtools.build.lib.remote.RemoteModule.afterCommand(RemoteModule.java:862)
	at com.google.devtools.build.lib.runtime.BlazeRuntime.afterCommand(BlazeRuntime.java:626)
	at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:619)
	at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:231)
	at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:550)
	at com.google.devtools.build.lib.server.GrpcServerImpl.lambda$run$1(GrpcServerImpl.java:614)
	at io.grpc.Context$1.run(Context.java:579)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.InterruptedException: DefaultPromise@2eb04551(incomplete)
	at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:244)
	at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:35)
	at io.netty.channel.pool.FixedChannelPool.close(FixedChannelPool.java:451)
	... 16 more

Maybe this is more to do with some deferred finalizer running in a thread-backed future, and it needs to ignore some exceptions?

wisechengyi · 2022-02-12T07:34:03Z

To add, unsurprisingly we are also seeing the error when interrupting bazel with ctrl-c.

% bazel build strato/...
Starting local Bazel server and connecting to it...
^C
Bazel caught interrupt signal; shutting down.

INFO: Invocation ID: 48473608-372c-4ac2-b1f1-fc41cf67fe5d
ERROR: command interrupted while syncing package loading
INFO: Invocation ID: 48473608-372c-4ac2-b1f1-fc41cf67fe5d
FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.RuntimeException: java.lang.InterruptedException: DefaultPromise@788c8811(incomplete)
	at io.netty.channel.pool.FixedChannelPool.close(FixedChannelPool.java:454)
	at com.google.devtools.build.lib.remote.http.HttpCacheClient.close(HttpCacheClient.java:736)
	at com.google.devtools.build.lib.remote.disk.DiskAndRemoteCacheClient.close(DiskAndRemoteCacheClient.java:71)
	at com.google.devtools.build.lib.remote.RemoteCache.deallocate(RemoteCache.java:454)
	at io.netty.util.AbstractReferenceCounted.handleRelease(AbstractReferenceCounted.java:86)
	at io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76)
	at com.google.devtools.build.lib.remote.RemoteActionContextProvider.afterCommand(RemoteActionContextProvider.java:217)
	at com.google.devtools.build.lib.remote.RemoteModule.afterCommand(RemoteModule.java:862)
	at com.google.devtools.build.lib.runtime.BlazeRuntime.afterCommand(BlazeRuntime.java:626)
	at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:619)
	at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:231)
	at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:550)
	at com.google.devtools.build.lib.server.GrpcServerImpl.lambda$run$1(GrpcServerImpl.java:614)
	at io.grpc.Context$1.run(Context.java:579)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.InterruptedException: DefaultPromise@788c8811(incomplete)
	at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:244)
	at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:35)
	at io.netty.channel.pool.FixedChannelPool.close(FixedChannelPool.java:451)
	... 16 more

Wyverald · 2022-03-02T14:58:07Z

@coeuvre could you take a look at this? If this is indeed a 5.0.0 regression, we could look at fixing it in 5.1.0.

coeuvre · 2022-03-02T15:37:57Z

Sorry for the late reply, somehow I missed this from my inbox. This is indeed a 5.0.0 regression and I will fix it in 5.1.0.

Wyverald · 2022-03-02T15:39:43Z

@bazel-io fork 5.1

Also cancels tasks submitted during `afterCommand` if interrupted. Fixes bazelbuild#14787. Closes bazelbuild#14992. PiperOrigin-RevId: 433205726 (cherry picked from commit a73aa12)

…14999) Also cancels tasks submitted during `afterCommand` if interrupted. Fixes #14787. Closes #14992. PiperOrigin-RevId: 433205726 (cherry picked from commit a73aa12) Co-authored-by: Chi Wang <[email protected]>

coryan · 2024-07-12T14:33:49Z

I am seeing this problem with Bazel 7.2.1:

bazel version
Bazelisk version: v1.18.0
Build label: 7.2.1
Build target: @@//src/main/java/com/google/devtools/build/lib/bazel:BazelServer
Build time: Tue Jun 25 15:53:05 2024 (1719330785)
Build timestamp: 1719330785
Build timestamp as int: 1719330785

Then you get:

$ bazel query --noshow_progress "..." | head -2
//:accessapproval
//:accessapproval_mocks

Cannot write to standard output; exiting...
$ echo $?
0

However, if your script has set -o pipefail:

$ set -o pipefail

$ bazel query --noshow_progress "..." | head -2
//:accessapproval
//:accessapproval_mocks

Cannot write to standard output; exiting...

$ echo $?
255

bazel-io added the potential release blocker Flagged by community members using "@bazel-io flag". Should be added to a release blocker milestone label Feb 10, 2022

Wyverald added P2 We'll consider working on this in future. (Assignee optional) team-Remote-Exec Issues and PRs for the Execution (Remote) team type: bug under investigation labels Feb 11, 2022

Wyverald changed the title ~~pipe breaking with bazel output release 5.0.0~~ Bazel crashes with InterruptedException when certain benign errors happen Feb 11, 2022

Wyverald assigned coeuvre Feb 16, 2022

bazel-io mentioned this issue Mar 2, 2022

[5.1] Bazel crashes with InterruptedException when certain benign errors happen #14932

Closed

coeuvre mentioned this issue Mar 7, 2022

Remote: Fix crashes with InterruptedException when using http cache. #14992

Closed

bazel-io closed this as completed in a73aa12 Mar 8, 2022

brentleyjones mentioned this issue Mar 8, 2022

[5.1] Remote: Fix crashes with InterruptedException when using http cache. #14999

Merged

coryan mentioned this issue Jul 12, 2024

ci: workaround bazel startup error googleapis/google-cloud-cpp#14469

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bazel crashes with InterruptedException when certain benign errors happen #14787

Bazel crashes with InterruptedException when certain benign errors happen #14787

wisechengyi commented Feb 10, 2022 •

edited

Loading

brentleyjones commented Feb 10, 2022

brentleyjones commented Feb 10, 2022

wisechengyi commented Feb 10, 2022 •

edited

Loading

ulfjack commented Feb 10, 2022

wisechengyi commented Feb 10, 2022 •

edited

Loading

Wyverald commented Feb 11, 2022

eed3si9n commented Feb 11, 2022

wisechengyi commented Feb 12, 2022

Wyverald commented Mar 2, 2022

coeuvre commented Mar 2, 2022

Wyverald commented Mar 2, 2022

coryan commented Jul 12, 2024

Bazel crashes with InterruptedException when certain benign errors happen #14787

Bazel crashes with InterruptedException when certain benign errors happen #14787

Comments

wisechengyi commented Feb 10, 2022 • edited Loading

Description of the problem:

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

What operating system are you running Bazel on?

What's the output of bazel info release?

If bazel info release returns "development version" or "(@non-git)", tell us how you built Bazel.

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

Have you found anything relevant by searching the web?

Other info

brentleyjones commented Feb 10, 2022

brentleyjones commented Feb 10, 2022

wisechengyi commented Feb 10, 2022 • edited Loading

ulfjack commented Feb 10, 2022

wisechengyi commented Feb 10, 2022 • edited Loading

Wyverald commented Feb 11, 2022

eed3si9n commented Feb 11, 2022

wisechengyi commented Feb 12, 2022

Wyverald commented Mar 2, 2022

coeuvre commented Mar 2, 2022

Wyverald commented Mar 2, 2022

coryan commented Jul 12, 2024

wisechengyi commented Feb 10, 2022 •

edited

Loading

What's the output of `bazel info release`?

If `bazel info release` returns "development version" or "(@non-git)", tell us how you built Bazel.

What's the output of `git remote get-url origin ; git rev-parse master ; git rev-parse HEAD` ?

wisechengyi commented Feb 10, 2022 •

edited

Loading

wisechengyi commented Feb 10, 2022 •

edited

Loading