Incorrect keepAliveMode used for Gradle worker daemon

[remy-tiitre]

Right now spotbug tasks are executed in a separate java fork and that fork is started with keepAliveMode=DAEMON option.

Started Gradle worker daemon (0.999 secs) with fork options DaemonForkOptions{executable=/usr/lib/jvm/java-11-openjdk-11.0.6.10-1.el7_7.x86_64/bin/java, minHeapSize=null, maxHeapSize=null, jvmArgs=[], keepAliveMode=**DAEMON**}.

I'm not 100% certain but I think this should be SESSION. What happens right now is that my project has multiple modules and spotbugs worker processes are left behind. When your CI runs inside containers and they have more restrictive memory limits than developer machines the containers will just get OOMkilled. Problem gets worse when you have more modules to use the spotbugs on. We have even more than main and test sourceSets that makes the issue even worse.

Even if its not that keepAliveMode then there is something that does not allow the process to end and they are left behind hogging memory.

[KengoTODA]

Our fork spec is specified as follows:

spotbugs-gradle-plugin/src/main/groovy/com/github/spotbugs/snom/internal/SpotBugsRunnerForWorker.java

Lines 56 to 64 in 3be9d63

	spec.forkOptions(
	option -> {
	option.jvmArgs(buildJvmArguments(task));
	String maxHeapSize = task.getMaxHeapSize().getOrNull();
	if (maxHeapSize != null) {
	option.setMaxHeapSize(maxHeapSize);
	}
	});
	};

It seems that we use the default keepAliveMode defined in the following class in the Gradle core:

https://github.com/gradle/gradle/blob/d7577a31e232925cb1ba4ae5e423a9cf436579d4/subprojects/workers/src/main/java/org/gradle/workers/internal/DefaultWorkerExecutor.java#L278-L279

ProcessWorkerSpec and JavaForkOptions provide no API to set keepAliveMode so I think current config is valid.

[remy-tiitre]

Like I said, I was not certain about the keepAliveMode. What is the real issue is that when you have multiple submodules and more than just main and test sourceSets then Spotbugs creates multiple forks and does not release the memory. For that reason we are running ./gradlew spotbugs --no-daemon as a separate task right now and after that we run ./gradlew build. Otherwise the project takes more memory than we can allow it to take. I think this problem is even worse when you don't create a clean container for every task. If you have pre created CI agent workers then you have to restart those agents periodically to get back the memory from zombi Spotbugs forks.

[KengoTODA]

Got it. Document says nothing aboit how to terminate worker process, I will investigate. Thanks!

https://docs.gradle.org/current/userguide/worker_api.html

[KengoTODA]

Successfully reproduced high memory usage by Gradle workers:

$ ps x -o rss,vsz,command | grep gradle-worker
 41184  7068740 /Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home/bin/java @/private/var/folders/f3/y4_b7t5j7q14_sfzfpdd9lb40000gn/T/gradle-worker-classpath12249792866259719308txt -Xmx1g -Dfile.encoding=UTF-8 -Duser.country=JP -Duser.language=en -Duser.variant worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Worker Daemon 6'
 74324  7137900 /Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home/bin/java @/private/var/folders/f3/y4_b7t5j7q14_sfzfpdd9lb40000gn/T/gradle-worker-classpath6615786728956135602txt -Xmx1g -Dfile.encoding=UTF-8 -Duser.country=JP -Duser.language=en -Duser.variant worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Worker Daemon 7'
 28544  7102404 /Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home/bin/java @/private/var/folders/f3/y4_b7t5j7q14_sfzfpdd9lb40000gn/T/gradle-worker-classpath2082060286803396054txt -Xmx1g -Dfile.encoding=UTF-8 -Duser.country=JP -Duser.language=en -Duser.variant worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Worker Daemon 8'
 48732  7096352 /Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home/bin/java @/private/var/folders/f3/y4_b7t5j7q14_sfzfpdd9lb40000gn/T/gradle-worker-classpath6500340904190778txt -Xmx1g -Dfile.encoding=UTF-8 -Duser.country=JP -Duser.language=en -Duser.variant worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Worker Daemon 9'
 30760  7131240 /Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home/bin/java @/private/var/folders/f3/y4_b7t5j7q14_sfzfpdd9lb40000gn/T/gradle-worker-classpath18295092730015090668txt -Xmx1g -Dfile.encoding=UTF-8 -Duser.country=JP -Duser.language=en -Duser.variant worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Worker Daemon 10'
 45656  7068260 /Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home/bin/java @/private/var/folders/f3/y4_b7t5j7q14_sfzfpdd9lb40000gn/T/gradle-worker-classpath8284973813404729154txt -Xmx1g -Dfile.encoding=UTF-8 -Duser.country=JP -Duser.language=en -Duser.variant worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Worker Daemon 11'
 24764  7086904 /Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home/bin/java @/private/var/folders/f3/y4_b7t5j7q14_sfzfpdd9lb40000gn/T/gradle-worker-classpath14573339430964253526txt -Xmx1g -Dfile.encoding=UTF-8 -Duser.country=JP -Duser.language=en -Duser.variant worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Worker Daemon 12'
 31968  7106928 /Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home/bin/java @/private/var/folders/f3/y4_b7t5j7q14_sfzfpdd9lb40000gn/T/gradle-worker-classpath9391467343059149267txt -Xmx1g -Dfile.encoding=UTF-8 -Duser.country=JP -Duser.language=en -Duser.variant worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Worker Daemon 16'
 20960  7082928 /Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home/bin/java @/private/var/folders/f3/y4_b7t5j7q14_sfzfpdd9lb40000gn/T/gradle-worker-classpath6695202723842985678txt -Xmx1g -Dfile.encoding=UTF-8 -Duser.country=JP -Duser.language=en -Duser.variant worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Worker Daemon 17'
 56860  7114600 /Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home/bin/java @/private/var/folders/f3/y4_b7t5j7q14_sfzfpdd9lb40000gn/T/gradle-worker-classpath12999000338956034425txt -Xmx1g -Dfile.encoding=UTF-8 -Duser.country=JP -Duser.language=en -Duser.variant worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Worker Daemon 18'
 41880  7103808 /Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home/bin/java @/private/var/folders/f3/y4_b7t5j7q14_sfzfpdd9lb40000gn/T/gradle-worker-classpath2012632288450145464txt -Xmx1g -Dfile.encoding=UTF-8 -Duser.country=JP -Duser.language=en -Duser.variant worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Worker Daemon 19'
   220  4258892 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox gradle-worker

[KengoTODA]

Again confirmed that worker surely improves execution performance. I will ask Gradle form to find better solution than disabling worker.

With worker

Without worker (-Pcom.github.spotbugs.snom.worker=false)

[KengoTODA]

posted a related question to the community forum.

[github-actions]

🎉 This issue has been resolved in version 4.7.0 🎉

The release is available on:

• 4.7.0
• GitHub release

Your semantic-release bot 📦🚀

[KengoTODA]

@remy-tiitre I released #429 as version 4.7.0. Please run your build with -Pcom.github.spotbugs.snom.javaexec-in-worker=true option, and verify how it changed memory usage. In theory, it should solve the reported issue by running not daemon process but one-shot process.

This option will be needless in future release, after I confirm that the new strategy works as expected.

[KengoTODA]

memo: in my local, confirmed that performance isn't so changed

hyperfine './gradlew clean build' './gradlew clean build -Pcom.github.spotbugs.snom.javaexec-in-worker=true'           

Benchmark #1: ./gradlew clean build
  Time (mean ± σ):     181.512 s ± 28.436 s    [User: 1.802 s, System: 0.228 s]
  Range (min … max):   163.234 s … 258.145 s    10 runs
 
Benchmark #2: ./gradlew clean build -Pcom.github.spotbugs.snom.javaexec-in-worker=true
  Time (mean ± σ):     174.119 s ±  3.646 s    [User: 1.680 s, System: 0.199 s]
  Range (min … max):   170.960 s … 181.187 s    10 runs
 
Summary
  './gradlew clean build -Pcom.github.spotbugs.snom.javaexec-in-worker=true' ran
    1.04 ± 0.16 times faster than './gradlew clean build'

[KengoTODA]

Hi @remy-tiitre could you check my previous comment?

[KengoTODA]

@remy-tiitre ping

[KengoTODA]

This fix is applied by default in the beta channel. See #571 for detail.