--experimental_remote_cache_async hangs forever when using a HTTP remote cache

[vogelsgesang]

Description of the bug:

When combining --experimental_remote_cache_async with a HTTP remote cache, Bazel sometimes hangs after the build is finished with the output

Waiting for remote cache: 1 upload; 1458s

The output is still updating (so it's not completely hanging), but there is no observable progress.
While the example below reproduces the issue for a bazel test invocation, the same issue also occurs for bazel build.

The issue does not appear if:

• using grpc instead of HTTP for the remote cache
• calling bazel clean --expunge right before the test invocation.

Which category does this issue belong to?

Remote Execution

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

The bug is not deterministic, so it might need a couple of tries.

The following script usually reproduces the hang after 2-5 iterations of the for loop. Execute it in a newly created, empty directory

wget https://github.com/buchgr/bazel-remote/releases/download/v2.4.1/bazel-remote-2.4.1-linux-x86_64
chmod +x bazel-remote-2.4.1-linux-x86_64
./bazel-remote-2.4.1-linux-x86_64 --dir cache-dir --max_size=1 --http_address=0.0.0.0:8080 &

touch WORKSPACE
cat > BUILD.bazel << EOF
sh_test(
   name = "test",
   srcs = ["test.sh"],
)
EOF

bazelisk clean --expunge

for i in `seq 1 100`; do
   echo "try $i"
   # If you enable the following line, the issue disappears
   # bazelisk clean --expunge
   # Recreate `test.sh` with a random comment in it, so we don't get cache hits
   timestamp=`date +"%s"`
   echo "# ${timestamp}" > test.sh
   echo "exit 0" >> test.sh
   chmod +x test.sh
   # This will hang after a couple of tries
   bazelisk test :test --remote_cache=http://localhost:8080 --experimental_remote_cache_async
done

Which operating system are you running Bazel on?

Linux

What is the output of bazel info release?

release 6.3.2

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

not inside any git repository

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

afaict, not a regression

Have you found anything relevant by searching the web?

• Remote: async upload #13655

[coeuvre]

Might be related to #18296.

[jmelahman]

Just wanna mention this also occurs even when setting an explicit timeout like --remote_timeout=600

[coeuvre]

Should be fixed by a804fb1. The fix will be available in 7.2.