Crash after build in downloadBlob #18337
Labels
P2
We'll consider working on this in future. (Assignee optional)
team-Remote-Exec
Issues and PRs for the Execution (Remote) team
type: bug
Description of the bug:
While running a build against a remote build farm, we hit the crash below.
We configure
--remote_timeout=10m
, which probably explains the 10-minute delays I've annotated in the log. We are running with--remote_download_toplevel
.The crash seems to originate from
downloadBlob
. It's entirely possible that the connection to the build farm died / was aborted, but note that because the gaps are almost-exactly 10-minute long, this could also be Bazel itself aborting the connection due to a timeout. In any case, Bazel should definitely not crash.What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
Probably: make Bazel download a very large artifact with
--remote_download_toplevel
(say ~4 GB) and then make the remote build farm node (cache?) crash to reset its open connections.Code inspection based on the stack trace pasted above should lead to the place where the exception is not being handled though.
Which operating system are you running Bazel on?
Linux
What is the output of
bazel info release
?bazel-6.1.1
If
bazel info release
returnsdevelopment version
or(@non-git)
, tell us how you built Bazel.No response
What's the output of
git remote get-url origin; git rev-parse master; git rev-parse HEAD
?No response
Have you found anything relevant by searching the web?
No response
Any other information, logs, or outputs that you want to share?
No response
The text was updated successfully, but these errors were encountered: