Minimal /lib/boot directory in built applications

[Sanne]

Description

Today when packaging an application in the "fast-jar" mode (our default and preferred format), we organize the jar files in this particular structure:

.
├── app
│   └── compiled-usercode.jar
├── lib
│   ├── boot
│   │   ├── io.quarkus.quarkus-bootstrap-runner-999-SNAPSHOT.jar
│   │   ├── io.quarkus.quarkus-development-mode-spi-999-SNAPSHOT.jar
│   │   └── [ ... other jars on the bootstrap classpath]
│   └── main
│       ├── extension1.jar
│       ├── extension1-runtime-dependency.jar
│       ├── extension2.jar
│       ├── extension2-runtime-dependency...
│       └── [ .. all runtime extensions and dependencies]
├── quarkus
│   ├── generated-bytecode.jar
│   ├── quarkus-application.dat
│   └── transformed-bytecode.jar
├── quarkus-app-dependencies.txt
└── quarkus-run.jar

Technically we have multiple classloaders, but for production mode in this format only two are relevant:

• the JVM boostrap classloader
• the Quarkus runtime classloader

JVM boostrap classloader

This one if fully controlled by the JVM and can not be overriden, as a safety precaution; it's great and stable, but is general purpose and can't take advantage of the details we have about the built application.
We use it to load the so called "Parent First Dependencies":

1. our custom Quarkus Runtime Classloader
2. any code that's a dependency of the Quarkus Runtime Classloader, or needs to be loaded before (e.g. the Logging system)
3. there's some exceptions made for some particular libraries which break if they are not loaded by this root classloader; these are workarounds and ideally non-existing in practice.

Quarkus Runtime Classloader

This one is used to load the majority of the application (hopefully), as it can apply several optimisations to help both with fast startup and lower memory consumption in the long term, and also lower memory consumption during the bootstrap process.

What to improve

Ideally we'd want all classes to be loaded by the optimised Quarkus Runtime Classloader; now of course this isn't possible as at least the code of this classloader needs to be loaded by the JVM classloader, but we can strife to minimise the amount of code that is sourced this way.

Number of jars

Beyond looking at the mere amount of classes that are being loaded by each classloader, it's also relevant to minimize the amount of jar files: each jar is a separate compressed zip stream stored in a file, and for each such zip stream a relatively heavyweight buffer needs to be allocated to help with efficient decompression.
Such zip streams and file handles are not closed by the JVM classloader until the app is shutdown, as the general purpose nature of the standard classloader is assuming that classloading might happen again at any point in time, while we believe this is unlikely after steady state - especially when dealing with small services whose code is all defined at bootstrap time.

Not so long ago, we had only a couple jars on the bootstrap classpath.
Today we have:

├── lib
│   ├── boot
│   │   ├── io.github.crac.org-crac-0.1.3.jar
│   │   ├── io.quarkus.quarkus-bootstrap-runner-999-SNAPSHOT.jar
│   │   ├── io.quarkus.quarkus-development-mode-spi-999-SNAPSHOT.jar
│   │   ├── io.quarkus.quarkus-vertx-latebound-mdc-provider-999-SNAPSHOT.jar
│   │   ├── io.smallrye.common.smallrye-common-constraint-2.1.2.jar
│   │   ├── io.smallrye.common.smallrye-common-cpu-2.1.2.jar
│   │   ├── io.smallrye.common.smallrye-common-expression-2.1.2.jar
│   │   ├── io.smallrye.common.smallrye-common-function-2.1.2.jar
│   │   ├── io.smallrye.common.smallrye-common-io-2.1.2.jar
│   │   ├── io.smallrye.common.smallrye-common-net-2.1.2.jar
│   │   ├── io.smallrye.common.smallrye-common-os-2.1.2.jar
│   │   ├── io.smallrye.common.smallrye-common-ref-2.1.2.jar
│   │   ├── org.jboss.logging.jboss-logging-3.5.3.Final.jar
│   │   └── org.jboss.logmanager.jboss-logmanager-3.0.2.Final.jar

Implementation ideas

Suggestions

Reduce the number of jars on /lib/boot

The first and most obvious one is to shade all dependencies in /lib/boot into a single jar file.
This will reduce the number of zip streams that need to stay open for the while lifespan of the application; beyond reducing the number of file handles, this could improve bootstrap performance, and will have an impact on the number of wasted buffers for those zip streams.

Ignore exceptional dependencies

Some extensions, and some user provided configuration options, have the ability to request for additional jars to be moved to the lib/boot directory.
I believe we should ignore this case; it will be suboptimal but hopefully rare.

Ship an optimized bootstrap

So I'm suggesting to not shade all dependencies in /lib/boot during each build, but shade these artifacts into a single "quarkus-bootstrap" artifact which we actually release to Maven Central.
This would help:

• quality testing as we'll all be using the same, no strange conflicts during shading.
• build times as this step already happened
• download times: less artifacts to fetch
• allow for future optimisations, e.g. removing dead code from some of such dependencies when we only need them partially, or repackaging this particular artifact in a more efficient format than the traditional JAR.