Use Gradle Worker API

[vmishenev]

WIP

Motivation

1. Currently, Dokka reloads a big classpath for every task. Also, it has reflection magic to support different versions of plugins.
2. Dokka works parallel badly. For example, the run time on on a project with 100 tasks is 8 minutes.

Proposal

Gradle Worker API can help Dokka to avoid it. But we have 2 options for using it:

1. Use cached classpath and noIsolation mode (to keep a classpath in a static variable that is shared between tasks). This approach is used in Kapt.
2. Use processIsolation mode. Dokka task will be executed in worker daemon processes. The running processes with the same classpath can be reused for other tasks. In this case, the classpath is loaded once per process.

We need to choose only one option.
We already have prototypes. @aSemy created one for the second option and there is a prototype for the cached classpath here.

Pros&cons

• From a performance point of view, processIsolation requires time to run worker processes (default is number of CPU processors) and load classpath in each process. However, if daemons are already running, the time of Dokka executing is the same as for cached classpath.
  My observations:
  Coroutines (precompilied): ~35s (cached classpath) vs ~66s (processIsolation with 8 workers)
  Small project with 100 tasks : ~20s (cached classpath) vs ~60s (processIsolation with 8 workers)
  My hypothesis: the difference in execution time between the two approaches is a constant (for my computer with 8 workers the difference is ~40 sec) to run processes and load classpathes. But the constant depends on the number of workers (e.g. coroutines --max-workers=2 takes 44 sec).
  Also, we can adjust the number of workers (depending on a number of tasks) to get more performance.

• Stability. noIsolation needs synchronization for a static state. For example, we have a data race with static properties on the IDE part. Also, Kapt experienced OOM issues. Since we use external libraries (and the compiler) and Dokka is not designed for a multithreading environment, some such kinds of problems can appear.

• Library compatibility. The Gradle documentation says: "External libraries may rely on certain system properties to be set which may conflict between work items. Or a library might not be compatible with the version of JDK that Gradle is running with and may need to be run with a different version. ". But I am not sure that is relevant for our external libraries.

There are other points for choosing an approach. Everybody can share their opinion about it here.

To sum up, I personally vote for the first option. In the case of Dokka, we will increase little bit of time building but stability is more important than performance.

[aSemy]

There's another isolation mode too, classLoaderIsolation(). It's a halfway point between the other two, and I think it's worth considering. I strongly suspect that it would be the most performant of the 3 options.

One benefit to processIsolation is that it prevents problems with coroutines, and so hopefully it would make finalizeCoroutines option obsolete.

dokka/core/src/main/kotlin/configuration.kt

Lines 139 to 156 in 14c05d7

	/**
	* Whether coroutines dispatchers should be shutdown after
	* generating documentation via [DokkaGenerator.generate].
	*
	* It effectively stops all background threads associated with
	* coroutines in order to make classes unloadable by the JVM,
	* and rejects all new tasks with [RejectedExecutionException]
	*
	* This is primarily useful for multi-module builds where coroutines
	* can be shut down after each module's partial task to avoid
	* possible memory leaks.
	*
	* However, this can lead to problems in specific lifecycles where
	* coroutines are shared and will be reused after documentation generation,
	* and closing it down will leave the build in an inoperable state.
	* One such example is unit tests, for which finalization should be disabled.
	*/
	val finalizeCoroutines: Boolean

I think that whatever option is chosen, I think #2740 will be broadly the same.

[TWiStErRob]

@aSemy What is the reason for Dokka tasks executing one by one, one after the other in a Gradle parallel build? When I do a publish I can see compilation and all other tasks complete very quickly, then all workers (28+) are busy with different Dokka tasks from submodules. The console logs are suggesting that one needs to complete before the next one can start.

[aSemy]

@aSemy What is the reason for Dokka tasks executing one by one, one after the other in a Gradle parallel build? When I do a publish I can see compilation and all other tasks complete very quickly, then all workers (28+) are busy with different Dokka tasks from submodules. The console logs are suggesting that one needs to complete before the next one can start.

DGP (Dokka Gradle Plugin) is not compatible with many Gradle features like project isolation, build cache, configuration cache - see #2700 - so tasks run sequentially, even across different subprojects. And DGP does not use the Worker API at the moment (hence this issue and my PR #2740).

If you want to generate Dokka docs faster then look at Dokkatoo. Dokkatoo is a re-implemented DGP that supports all the speedy Gradle features. Dokkatoo is not a drop-in replacement for DGP, but it's pretty similar, and you run add both DGP and Dokkatoo in the same project to verify that the output of both is identical.

(Funny that you've pinged me (just a contributor) rather than the actual maintainers @IgnatBeresnev and @vmishenev 😄)

[TWiStErRob]

I pinged you exactly because of all those issues, PRs and repo you linked. I know DGP is not compatible with fancy new features, but org.gradle.parallel is a pretty pretty old feature. I'm wondering what is locking inside Dokka that doesn't allow basic parallelism.

[aSemy]

I pinged you exactly because of all those issues, PRs and repo you linked. I know DGP is not compatible with fancy new features, but org.gradle.parallel is a pretty pretty old feature. I'm wondering what is locking inside Dokka that doesn't allow basic parallelism.

Ah okay. Hmm, I'm not sure I can give a definitive answer because I'm not exactly sure how --parallel works and what the requirements are, and precisely what DGP is doing that would prevent it, but these items in particular are related:

• DGP multi-module tasks depend on DGP tasks from other subprojects. This means that subprojects are not isolated, and Dokka tasks from one subproject might have dependencies on tasks in other subprojects.

  • dokka/runners/gradle-plugin/src/main/kotlin/org/jetbrains/dokka/gradle/DokkaPlugin.kt

    Line 100 in 900fbcc

    addSubprojectChildTasks(name)

  • dokka/runners/gradle-plugin/src/main/kotlin/org/jetbrains/dokka/gradle/DokkaPlugin.kt

    Line 83 in 900fbcc

    addSubprojectChildTasks("${name}Partial")

  Dokkatoo fixes this by using Configurations to share files.

• DGP fetches the classpath directly from the Kotlin compilation task.

  dokka/runners/gradle-plugin/src/main/kotlin/org/jetbrains/dokka/gradle/kotlin/kotlinClasspathUtils.kt

  Line 63 in 900fbcc

  kotlinCompile.libraries // introduced in 1.7.0

  While this won't necessarily cause problems, it does cause task-linking that shouldn't exist. When coupled with cross-project tasks, this will probably interfere with Gradle workings and add unnecessary coupling between tasks.

  Dokkatoo fixes this by fetching the dependencies via the resolvable Configurations.

[TWiStErRob]

Thanks for the pointers!

[martinbonnin]

+1 for classloaderIsolation or at least making the isolation configurable. In addition to performance, classloaderIsolation makes it way easier to debug the builds.

[adam-enko]

One potential benefit of process isolation is that it would allow for class data sharing. With a multimodule project, Dokka Generator has to run multiple times with the same classpath. The Dokka classpath can be quite large (the analyzer component is ~80MB). Using CDS would mean that the classes could be 'cached' between generations, improving startup time and reducing memory usage.

[whyoleg]

Implemented in Dokka 2.0.0 in Dokka Gradle plugin v2

By default Dokka will be executed with processIsolation and xmx=2GB because of gradle/gradle#18313
Check the Troubleshooting section of migration guide if the default doesn't fit your project.

[martinbonnin]

FWIW, my view on this has changed. I now try avoid both classloader and process isolation altogether because of the memory leaks and/or multiple processes/heap to manage.

Instead, I'm now managing a classloader cache in a buildservice. This is a bit more manual work but at least it gives more control over the classloader lifecycle. A build service can cache a classloader for a given build but could even decide more advanced strategies like "keep alive for 30min" or "as long as the Gradle daemon is alive", ...

The only advantage I now see in using the worker API is to give parallel task execution to users without configuration cache enabled.

[adam-enko]

Instead, I'm now managing a classloader cache in a buildservice. This is a bit more manual work but at least it gives more control over the classloader lifecycle. A build service can cache a classloader for a given build but could even decide more advanced strategies like "keep alive for 30min" or "as long as the Gradle daemon is alive", ...

This sounds like a good approach, and something Dokka could use, but be careful you don't stub your toe on gradle/gradle#17559 :)