Increased memory consumption when creating native images

[ilopmar]

Describe the issue

During the last run of our Micronaut-GraalVM integration suite last night, some of the jobs have failed with out of memory exceptions during the native-image compile phase. The error is Fatal error:org.graalvm.compiler.debug.GraalError: java.lang.OutOfMemoryError: GC overhead limit exceeded

Some of our runners have limited resources so we can detect things like this. Others are more powerful so they worked properly.
This happens for both JDK 8 and 11 although we have more JDK 8 apps running on limited runners because JDK11 as been more eager regarding memory needs for a long time.

Steps to reproduce the issue

Not sure if there is anything you can do about it. If you want a reproducer please let me know and I can provide one.

Describe GraalVM and your environment:

• GraalVM version: d214dc0
• JDK major version: 8 & 11
• OS: Linux
• Architecture: AMD64

More details

Full log extracted from https://gitlab.com/micronaut-projects/micronaut-graal-tests/-/jobs/1287862797

[basic-app:162]    classlist:  16,254.55 ms,  1.39 GB
[basic-app:162]        (cap):   1,451.49 ms,  1.91 GB
[basic-app:162]        setup:   7,565.91 ms,  1.91 GB
[basic-app:162]     (clinit):   1,379.39 ms,  2.73 GB
[basic-app:162]   (typeflow):  93,803.49 ms,  2.73 GB
[basic-app:162]    (objects):  93,778.20 ms,  2.73 GB
[basic-app:162]   (features):   6,873.52 ms,  2.73 GB
[basic-app:162]     analysis: 198,589.16 ms,  2.73 GB
[basic-app:162]     universe:  13,575.14 ms,  2.74 GB
[basic-app:162]      (parse):  17,287.87 ms,  2.59 GB
[basic-app:162]     (inline): 278,288.82 ms,  2.57 GB
[basic-app:162]    (compile): 1,975,104.11 ms,  2.57 GB
[basic-app:162]      compile: 2,295,371.71 ms,  2.57 GB
3 fatal errors detected:
Fatal error:org.graalvm.compiler.debug.GraalError: java.lang.OutOfMemoryError: GC overhead limit exceeded
	at method: Object io.micronaut.context.AbstractBeanDefinition.getValueForField(BeanResolutionContext, BeanContext, int)  [Direct call from Object $UserExecutorConfigurationDefinition.injectBean(BeanResolutionContext, BeanContext, Object)]
	at com.oracle.svm.hosted.code.CompileQueue.defaultCompileFunction(CompileQueue.java:1192)
	at com.oracle.svm.hosted.code.CompileQueue.doCompile(CompileQueue.java:1124)
	at com.oracle.svm.hosted.code.CompileQueue$CompileTask.run(CompileQueue.java:282)
	at com.oracle.graal.pointsto.util.CompletionExecutor.lambda$execute$0(CompletionExecutor.java:173)
	at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
	at org.graalvm.compiler.phases.schedule.SchedulePhase$Instance$MicroBlock.add(SchedulePhase.java:699)
	at org.graalvm.compiler.phases.schedule.SchedulePhase$Instance.processStack(SchedulePhase.java:976)
	at org.graalvm.compiler.phases.schedule.SchedulePhase$Instance.processNodes(SchedulePhase.java:923)
	at org.graalvm.compiler.phases.schedule.SchedulePhase$Instance.scheduleEarliestIterative(SchedulePhase.java:832)
	at org.graalvm.compiler.phases.schedule.SchedulePhase$Instance.run(SchedulePhase.java:199)
	at org.graalvm.compiler.phases.schedule.SchedulePhase.run(SchedulePhase.java:158)
	at org.graalvm.compiler.phases.Phase.run(Phase.java:49)
	at org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:212)
	at org.graalvm.compiler.phases.Phase.apply(Phase.java:42)
	at org.graalvm.compiler.phases.Phase.apply(Phase.java:38)
	at org.graalvm.compiler.phases.common.FixReadsPhase.run(FixReadsPhase.java:608)
	at org.graalvm.compiler.phases.common.FixReadsPhase.run(FixReadsPhase.java:96)
	at org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:212)
	at org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:147)
	at org.graalvm.compiler.phases.PhaseSuite.run(PhaseSuite.java:221)
	at org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:212)
	at org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:147)
	at org.graalvm.compiler.core.GraalCompiler.emitFrontEnd(GraalCompiler.java:242)
	at org.graalvm.compiler.core.GraalCompiler.compile(GraalCompiler.java:145)
	at org.graalvm.compiler.core.GraalCompiler.compileGraph(GraalCompiler.java:130)
	at com.oracle.svm.hosted.code.CompileQueue.defaultCompileFunction(CompileQueue.java:1162)
	at com.oracle.svm.hosted.code.CompileQueue$$Lambda$690/1004710091.compile(Unknown Source)
	at com.oracle.svm.hosted.code.CompileQueue.doCompile(CompileQueue.java:1124)
	at com.oracle.svm.hosted.code.CompileQueue$CompileTask.run(CompileQueue.java:282)
	at com.oracle.graal.pointsto.util.CompletionExecutor.lambda$execute$0(CompletionExecutor.java:173)
	at com.oracle.graal.pointsto.util.CompletionExecutor$$Lambda$278/1257612127.run(Unknown Source)
	... 5 more
Fatal error:org.graalvm.compiler.debug.GraalError: java.lang.OutOfMemoryError: GC overhead limit exceeded
	at method: void io.micronaut.core.beans.SimpleBeanInfo.fixGetSet(HashMap)  [Direct call from List SimpleBeanInfo.introspectProperties(Method[])]
	at com.oracle.svm.hosted.code.CompileQueue.defaultCompileFunction(CompileQueue.java:1192)
	at com.oracle.svm.hosted.code.CompileQueue.doCompile(CompileQueue.java:1124)
	at com.oracle.svm.hosted.code.CompileQueue$CompileTask.run(CompileQueue.java:282)
	at com.oracle.graal.pointsto.util.CompletionExecutor.lambda$execute$0(CompletionExecutor.java:173)
	at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
	at jdk.vm.ci.meta.Signature.toParameterTypes(Signature.java:113)
	at com.oracle.svm.core.graal.snippets.NonSnippetLowerings$InvokeLowering.lower(NonSnippetLowerings.java:245)
	at com.oracle.svm.core.graal.snippets.NonSnippetLowerings$InvokeLowering.lower(NonSnippetLowerings.java:228)
	at com.oracle.svm.core.graal.amd64.SubstrateAMD64LoweringProvider.lower(SubstrateAMD64LoweringProvider.java:72)
	at org.graalvm.compiler.nodes.spi.Lowerable.lower(Lowerable.java:40)
	at org.graalvm.compiler.phases.common.LoweringPhase$Round.process(LoweringPhase.java:453)
	at org.graalvm.compiler.phases.common.LoweringPhase$Round.access$300(LoweringPhase.java:329)
	at org.graalvm.compiler.phases.common.LoweringPhase$Round$ProcessFrame.preprocess(LoweringPhase.java:393)
	at org.graalvm.compiler.phases.common.LoweringPhase.processBlock(LoweringPhase.java:552)
	at org.graalvm.compiler.phases.common.LoweringPhase$Round.run(LoweringPhase.java:378)
	at org.graalvm.compiler.phases.Phase.run(Phase.java:49)
	at org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:212)
	at org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:147)
	at org.graalvm.compiler.phases.PhaseSuite.run(PhaseSuite.java:221)
	at org.graalvm.compiler.phases.common.IncrementalCanonicalizerPhase.run(IncrementalCanonicalizerPhase.java:56)
	at org.graalvm.compiler.phases.common.IncrementalCanonicalizerPhase.run(IncrementalCanonicalizerPhase.java:38)
	at org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:212)
	at org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:147)
	at org.graalvm.compiler.phases.common.LoweringPhase.lower(LoweringPhase.java:247)
	at org.graalvm.compiler.phases.common.LoweringPhase.run(LoweringPhase.java:237)
	at org.graalvm.compiler.phases.common.LoweringPhase.run(LoweringPhase.java:92)
	at org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:212)
	at org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:147)
	at org.graalvm.compiler.phases.PhaseSuite.run(PhaseSuite.java:221)
	at org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:212)
	at org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:147)
	at org.graalvm.compiler.core.GraalCompiler.emitFrontEnd(GraalCompiler.java:232)
	at org.graalvm.compiler.core.GraalCompiler.compile(GraalCompiler.java:145)
	at org.graalvm.compiler.core.GraalCompiler.compileGraph(GraalCompiler.java:130)
	at com.oracle.svm.hosted.code.CompileQueue.defaultCompileFunction(CompileQueue.java:1162)
	at com.oracle.svm.hosted.code.CompileQueue$$Lambda$690/1004710091.compile(Unknown Source)
	at com.oracle.svm.hosted.code.CompileQueue.doCompile(CompileQueue.java:1124)
Fatal error:org.graalvm.compiler.debug.GraalError: java.lang.OutOfMemoryError: GC overhead limit exceeded
	at method: Object io.micronaut.context.AbstractBeanDefinition.resolveBeanWithGenericsFromConstructorArgument(BeanResolutionContext, Argument, AbstractBeanDefinition$BeanResolver)  [Direct call from Collection AbstractBeanDefinition.getBeansOfTypeForConstructorArgument(BeanResolutionContext, BeanContext, ConstructorInjectionPoint, Argument)]
	at com.oracle.svm.hosted.code.CompileQueue.defaultCompileFunction(CompileQueue.java:1192)
	at com.oracle.svm.hosted.code.CompileQueue.doCompile(CompileQueue.java:1124)
	at com.oracle.svm.hosted.code.CompileQueue$CompileTask.run(CompileQueue.java:282)
	at com.oracle.graal.pointsto.util.CompletionExecutor.lambda$execute$0(CompletionExecutor.java:173)
	at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
Error: Image build request failed with exit status 1
> Task :nativeImage FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':nativeImage'.
> Process 'command '/builds/micronaut-projects/micronaut-graal-tests/graal_dist/jre/lib/svm/bin/native-image'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 46m 21s

Full pipeline https://gitlab.com/micronaut-projects/micronaut-graal-tests/-/pipelines/308223577

[iamrecursion]

Anecdotally, we have also observed higher memory usage during compilation on 21.1.0.

[oubidar-Abderrahim]

Tracked internally on GR-31714

[zakkak]

Hello @oubidar-Abderrahim ,

are there any updates on this or any hints on what might be causing it?

[oubidar-Abderrahim]

The memory footprint of the image generator has been optimized since then. This should be fixed