"java.lang.OutOfMemoryError: Java heap space" during repository fetch

[jfirebaugh]

Description of the bug:

We observe the following exception during the repository fetch phase:

    Fetching ...linux_x86_64__x86_64-unknown-linux-gnu__stable_tools; starting
    Fetching ..._tools; Extracting rustc-1.76.0-x86_64-unknown-linux-gnu.tar.x\
z
    Fetching repository @@nodejs_linux_amd64; starting
    Fetching repository @@npm_typescript; starting
    Fetching ...0ae0d/external/npm_typescript; Extracting typescript-5.4.2.tgz
    Fetching ...ux_amd64/bin/nodejs; Extracting node-v20.14.0-linux-x64.tar.xz
FATAL: bazel ran out of memory and crashed. Printing stack trace:
net.starlark.java.eval.Starlark$UncheckedEvalError: OutOfMemoryError thrown during Starlark evaluation (LOADING)
	at <starlark>.download_and_extract(<builtin>:0)
	at <starlark>._download_node(/home/circleci/.cache/bazel/_bazel_circleci/a6e8cb0ce75ceb1ea1faa0cca0b0ae0d/external/rules_nodejs/nodejs/repositories.bzl:102)
	at <starlark>._nodejs_repositories_impl(/home/circleci/.cache/bazel/_bazel_circleci/a6e8cb0ce75ceb1ea1faa0cca0b0ae0d/external/rules_nodejs/nodejs/repositories.bzl:308)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at org.tukaani.xz.ArrayCache.getByteArray(Unknown Source)
	at org.tukaani.xz.lz.LZDecoder.<init>(Unknown Source)
	at org.tukaani.xz.LZMA2InputStream.<init>(Unknown Source)
	at org.tukaani.xz.LZMA2Decoder.getInputStream(Unknown Source)
	at org.tukaani.xz.BlockInputStream.<init>(Unknown Source)
	at org.tukaani.xz.SingleXZInputStream.read(Unknown Source)
	at org.tukaani.xz.XZInputStream.read(Unknown Source)
	at org.apache.commons.io.IOUtils$$Lambda/0x000000010080a220.apply(Unknown Source)
	at org.apache.commons.io.IOUtils.read(IOUtils.java:1958)
	at org.apache.commons.io.IOUtils.read(IOUtils.java:1934)
	at org.apache.commons.compress.utils.IOUtils.readFully(IOUtils.java:203)
	at org.apache.commons.compress.utils.IOUtils.readFully(IOUtils.java:183)
	at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.readRecord(TarArchiveInputStream.java:689)
	at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getRecord(TarArchiveInputStream.java:497)
	at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:415)
	at com.google.devtools.build.lib.bazel.repository.CompressedTarFunction.decompress(CompressedTarFunction.java:83)
	at com.google.devtools.build.lib.bazel.repository.DecompressorValue.decompress(DecompressorValue.java:131)
	at com.google.devtools.build.lib.bazel.repository.starlark.StarlarkBaseExternalContext.downloadAndExtract(StarlarkBaseExternalContext.java:955)
	at java.base/java.lang.invoke.LambdaForm$DMH/0x00000001007f3400.invokeVirtual(LambdaForm$DMH)
	at java.base/java.lang.invoke.LambdaForm$MH/0x00000001007f5800.invoke(LambdaForm$MH)
	at java.base/java.lang.invoke.LambdaForm$MH/0x0000000100530400.invoke(LambdaForm$MH)
	at java.base/java.lang.invoke.Invokers$Holder.invokeExact_MT(Unknown Source)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(Unknown Source)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at net.starlark.java.eval.MethodDescriptor.call(MethodDescriptor.java:178)
	at net.starlark.java.eval.BuiltinFunction.fastcall(BuiltinFunction.java:78)
	at net.starlark.java.eval.Starlark.fastcall(Starlark.java:806)
	at net.starlark.java.eval.Eval.evalCall(Eval.java:682)
	at net.starlark.java.eval.Eval.eval(Eval.java:497)
	at net.starlark.java.eval.Eval.exec(Eval.java:271)
	at net.starlark.java.eval.Eval.execStatements(Eval.java:82)

This happens only intermittently. It appears to be a sudden large spike in memory use, because Bazel server memory use up to this point is reasonable and our memory metrics (on 10s collection interval) do not show the process reaching anywhere close to its limit.

Bazel 7.2.0, using --experimental_worker_for_repo_fetching=off due to #22680.

Which category does this issue belong to?

Core

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Sorry, don't have a minimal example.

Which operating system are you running Bazel on?

linux

What is the output of bazel info release?

release 7.2.0

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

n/a

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

[fmeum]

Please test with 7.2.1, which has a number of fixes in this area: 7.2.0...7.2.1

[jfirebaugh]

Still happens occasionally in 7.2.1.

net.starlark.java.eval.Starlark$UncheckedEvalError: OutOfMemoryError thrown during Starlark evaluation (LOADING)
	at <starlark>.download_and_extract(<builtin>:0)
	at <starlark>.load_arbitrary_tool(/home/circleci/.cache/bazel/_bazel_circleci/a6e8cb0ce75ceb1ea1faa0cca0b0ae0d/external/rules_rust/rust/private/repository_utils.bzl:733)
	at <starlark>.load_rust_compiler(/home/circleci/.cache/bazel/_bazel_circleci/a6e8cb0ce75ceb1ea1faa0cca0b0ae0d/external/rules_rust/rust/private/repository_utils.bzl:399)
	at <starlark>._rust_toolchain_tools_repository_impl(/home/circleci/.cache/bazel/_bazel_circleci/a6e8cb0ce75ceb1ea1faa0cca0b0ae0d/external/rules_rust/rust/repositories.bzl:300)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at org.tukaani.xz.ArrayCache.getByteArray(Unknown Source)
	at org.tukaani.xz.lz.LZDecoder.<init>(Unknown Source)
	at org.tukaani.xz.LZMA2InputStream.<init>(Unknown Source)
	at org.tukaani.xz.LZMA2Decoder.getInputStream(Unknown Source)
	at org.tukaani.xz.BlockInputStream.<init>(Unknown Source)
	at org.tukaani.xz.SingleXZInputStream.read(Unknown Source)
	at org.tukaani.xz.XZInputStream.read(Unknown Source)
	at org.apache.commons.io.IOUtils$$Lambda/0x0000000100808410.apply(Unknown Source)
	at org.apache.commons.io.IOUtils.read(IOUtils.java:1958)
	at org.apache.commons.io.IOUtils.read(IOUtils.java:1934)
	at org.apache.commons.compress.utils.IOUtils.readFully(IOUtils.java:203)
	at org.apache.commons.compress.utils.IOUtils.readFully(IOUtils.java:183)
	at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.readRecord(TarArchiveInputStream.java:689)
	at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getRecord(TarArchiveInputStream.java:497)
	at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:415)
	at com.google.devtools.build.lib.bazel.repository.CompressedTarFunction.decompress(CompressedTarFunction.java:83)
	at com.google.devtools.build.lib.bazel.repository.DecompressorValue.decompress(DecompressorValue.java:131)
	at com.google.devtools.build.lib.bazel.repository.starlark.StarlarkBaseExternalContext.downloadAndExtract(StarlarkBaseExternalContext.java:995)
	at java.base/java.lang.invoke.LambdaForm$DMH/0x00000001007f9800.invokeVirtual(LambdaForm$DMH)
	at java.base/java.lang.invoke.LambdaForm$MH/0x0000000100801000.invoke(LambdaForm$MH)
	at java.base/java.lang.invoke.LambdaForm$MH/0x0000000100530400.invoke(LambdaForm$MH)
	at java.base/java.lang.invoke.Invokers$Holder.invokeExact_MT(Unknown Source)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(Unknown Source)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at net.starlark.java.eval.MethodDescriptor.call(MethodDescriptor.java:178)
	at net.starlark.java.eval.BuiltinFunction.fastcall(BuiltinFunction.java:78)
	at net.starlark.java.eval.Starlark.fastcall(Starlark.java:806)
	at net.starlark.java.eval.Eval.evalCall(Eval.java:682)
	at net.starlark.java.eval.Eval.eval(Eval.java:497)
	at net.starlark.java.eval.Eval.execAssignment(Eval.java:109)
	at net.starlark.java.eval.Eval.exec(Eval.java:268)