[native-image] Override META-INF native image properties

[fhanik]

Feature request

Please include the following information:

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. E.g. I'm always frustrated when [...]

Describe the solution you'd like.
Some libraries include meta files for the native-image generator inside the JAR file at the META-INF/native-image/groupId/artifactId location.

The files often have to include reflection and resource data that encompasses all use cases.
When using a small portion of the library, it may be desirable to override this configuration with a provided one, a smaller one, to optimize the size of the binary.

Describe who do you think will benefit the most.
GraalVM users that compile the applications to native images will benefit since they will have a greater control over what is included in the image.

Describe alternatives you've considered.
Creating a repackaging step during a build cycle to repackage 3rd party jars is an option. This becomes hard to maintain.

[olpaw]

That would essentially boil down to support passing of a blacklist of META-INF/native-image/**/* locations. Should not be too hard to add. This would be a good candidate for a PR provided by the community.

[eginez]

Hey @fhanik, I am not sure this falls under the responsibility of the native-image build process. The optimization of the size of an image might require a pre-processing step which could be responsible of the above, in fact it seems like we the spring team is already doing something like that. We, the native-image team are trying, to keep the build-process focused on its core competency, namely building native executables.

[galderz]

I've found myself needing this today. Btw, not the first time asked, @Sanne already asked for this in #1884 last year.

[vjovanov]

I would be reluctant to introduce this feature as I believe, it would make the progress of the community slower. If there is a problem in the library, the right approach is to submit a PR to that library.

Also, as @eginez pointed out, this can be achieved via other means (e.g., preprocessing). I would like to understand why submitting a PR to a library is not possible? Also, what makes it hard to remove invalid properties from a JAR?

[maxandersen]

I would like to understand why submitting a PR to a library is not possible? Also, what makes it hard to remove invalid properties from a JAR?

1. libraries does not necessarily release with the speed you need
2. removing invalid properties means forking/modifiying which one can do (we know we will have to it already in quarkus as many libs don't apply these things properly)

note we will properly already need to do the #2 processing but still find it interesting we can explicitly turn off graalvm/native-image processing of bundled content to just reduce chance of surprises.

[galderz]

From the POV of Quarkus, an integrator, we try to make an environment as efficient as possible for our users. We do native image configuration that is more aggressive than what libraries are ready to offer to the general public. We are one of their use cases, and so it'd be unfair for us to impose such configuration on all the libraries' users. At the same time, it's unfair for integrators to be forced to apply configurations that are suited for a wide range of uses.

Preprocessing is certainly an option but it has many risks. E.g. what if we take someone's jar and due to filtering we forgot to add something new that gets added at a later stage? The other problem of preprocessing is that this is something that would likely be needed by other integrators out there, leading to a duplication of efforts.

[Sanne]

@vjovanov we've been sending PRs to upstream libraries often, but now since I don't have this switch I regret doing so: by the time such metadata is accepted, merged, a release is happened, and then other libraries of a more complex platform are ready to upgrade to such latest version, it's highly likely that some more tweaking is necessary.

For example, if we start working / experimenting with a new optimisation, a better way, or a new GraalVM version.

We're now at a point that the libraries which don't embed ay such metadata are easier to use on GraalVM; we highly need such a switch, or alternatively I'll start stripping such metadata out for the libraries I care about - which could imply removing such metadata, should we not have a better solution.

I would agree that basic compatibility rules do belong within each library, but without more flexibility this is severely problematc.

[Sanne]

P.S. I wrote about what we did in Hibernate ORM here:

• https://in.relation.to/2020/02/14/hibernate-orm-5-4-12/

In particular I should stress that I did move this metadata insto a separate Maven artifact, so to allow people to exclude it.

I also contribute to several other libraries (100+ !) but I won't be adding rules in META-INF in any library if we don't have a way to disable / override them; I had some POCs and it was too complex for all to agree on a static set of directives which would strike a good balance between "making things easier" and causing no problems in future maintenance.

[vjovanov]

It would be very useful to have a list of cases where things went wrong in PRs made to the libraries. That way we could define what are the best practices related to native image configuration. Those practices would guide library authors on how to do configuration properly.

As for the flag to disable part of the config, I realize that it can be a useful tool. When designing this tool we need to keep in mind that there will also be annotations in the future that we might also need to disable (if wrong):

https://docs.google.com/document/d/1J7c2HcTCvYn2d17PV8LbsIfD7wlSGrFVJTaZNphC-Rg/edit?usp=sharing

This tool should also be fine-grained so people can disable only parts of the config.

[Sanne]

thanks @vjovanov , and yes I agree about the annotations described in the document.

I had mentioned the same in a comment on that document last week, which then someone resolved stating it was not a goal in the "first phase". While I generally agree with the need to structure code sprint in smaller phases, I would hope that a general override switch isn't too complex to implement, so maybe there's hope for that to be reconsidered?

It would be very useful to have a list of cases where things went wrong in PRs made to the libraries.

Good point; sadly I don't remember the precise details, I'll try to make notes in the future.

I remember we had several issues with Netty, as it had this metadata contributed early on - by a great colleague :) I know we had all best intentions and it was well thought, but there's many aspects at play that evolve quickly, that's why I'm stressing the need to de-couple these. In fact, that's why Quarkus has its extension tree structured like it is today.

Please see also my issue description in #1884

Thanks

[Sanne]

@vjovanov , @thomaswue the Oracle JDBC Driver released version 21.1.0.0 including native-image support by including /META-INF/native-image/.

While it's very welcome that there's an attempt, the fact that they are setting global flags which are not compatible with other frameworks is making them very hard to use in practice; I won't be able to use them as-is in Quarkus as some flags directly conflict with core requirements.

This is very much the issue I prediced back in #1884 (comment) , we will need to be able to implement my original plan.

Could this please be considered urgent now?

My issue #1884 was closed in favour of this one; fine but please consider my original suggestions, as suggested during the GraalVM workshop: ideally I wouldn't want to fully ignore what's in these resources, as I agree that it's nice for library developers to be able to enhance these as needed, but Quarkus needs to be aware of what's being set, and possibly trigger warnigs and exceptionally override some which are too dangerous (or plain mistakes).

One simple step would be to have a "kill switch" for native-image to simply ignore them all; but on top of that it would be nice if the GraalVM SDK could expose the resource parser as an API we can invoke during our build before native-image is invoked; that would allow us to keep honouring all flags which aren't particulartly problematic, and to support more libraries which don't necessarily have a Quarkus extension taking care of such details.

Specifics of the JDBC problems are listed here:

• Add jdbc support for Oracle quarkusio/quarkus#1658 (comment)
  It's work in progress though, I only started looking last night - I expect I'll need to refine this, but please consider I'm blocked because I can't prevent native-image to load these, nor can I modify the JDBC driver because of its licensing.

Many thanks!

[galderz]

FYI, I've created a PR that provides a simple solution by adding a global flag to skip all configuration files inside jar files, see #3149.

[sergey-morenets]

Hi @galderz

Thank you for your help. So is there any option flag to override default native image configuration folder?
By default it's META-INF/native-image inside jar file. But Spring Boot, for example, puts this folder under /BOOT-INF/classes/ folder. So native-image utility is unable to detect them.

Thank you.

[zakkak]

@sergey-morenets it looks like this is not possible at the moment.
The configuration folder is hard-coded here

graal/substratevm/src/com.oracle.svm.driver/src/com/oracle/svm/driver/NativeImage.java

Line 265 in e997cec

public static final String nativeImageMetaInf = "META-INF/native-image";

[galderz]

@sergey-morenets Do you know what option Spring are using to customize the parent folder for the configuration? I couldn't see how to do it in the docs, but native-image has a lot of options available. The reason I ask is because if there's a way to customize that, it seems to me that NativeImage class should take that into account and not hard code it.

[sergey-morenets]

@galderz I guess they repackage far JAR by their Spring Boot plugin so that META-INF/native-image would be top-level folder

[vjovanov]

This has been fixed.