App Engine Java 11 generator to create thin JARs

[saturnism]

Overview of the feature request

from @ludoch #10331 (comment)

What I really would like to do as well is the take benefit of a exploded fatjar deployment for App Engine.
The Jib plugin does it automatically to help optimizing Docker layers.
GAE Java deployment also does it (since 2009), ie it will only deploy updated files from a previous deployment... When there is a single fatjar, all those upload optimizations are gone, even it the app changed 1 character!

Basically, if we could unjar the fatjar in the target/appengine-staging directory (and remove the orginal fatjar), the entrypoint would have to be like:

entrypoint: java -agentpath:/opt/cdbg/cdbg_java_agent.so=--log_dir=/var/log -noverify -XX:+AlwaysPreTouch -XX:TieredStopAtLevel=1 -Djava.security.egd=file:/dev/./urandom -cp BOOT-INF/resources/:BOOT-INF/classes/:BOOT-INF/lib/* com.mycompany.myapp.YOUAPPCLASSNAME

If this can be done, one extra goodies would be to configure App Engine to use a CDN to access the static resources instead of the JVM itself (from a fatjar which is also not optimal at all)...

To configure static resources in the app.yaml with an exploded jHipster fatjar, you would have to add this section in app.yaml:

handlers:

url: (/.)
static_files: BOOT-INF/classes/static\1
upload: NOT_USED
require_matching_file: True
login: optional
secure: optional
url: /.
script: unsused
url: /./
script: unused
login: optional
secure: optional
url: .
script: unused
login: optional
secure: optional
With these 2 changes, we are back to the orginal Java GAE design from 2009 with appengine-werb.xml configuration: exploded war for optimal deployement and automatic static file servinf from a CDN. You would see the difference when loading an app where all the static html and images and js scripts are served immediately...

Motivation for or Use Case

To allow JHipster on App Engine Java 11 to take advantage of static file serving.

Related issues or PR

#10331

• Checking this box is mandatory (this is just to show you read everything)

[SudharakaP]

@saturnism : Thanks for creating the issue. Feel free to assign it to me as well. I'll start this once we finalize the java 11 gae enhancements.

[SudharakaP]

@ludoch @saturnism : I was trying this out by constructing the thin jar using spring-boot-thin-launcher which I believe is the standard approach of doing this (let me know if you have an alternative suggestion). Although the thin jar construction seems to work in my local after deploying it to Google App Engine I get;

Exception in thread "main" java.lang.IllegalStateException: Cannot create directory for library at /var/www/.m2/repository/org/springframework/boot/experimental/spring-boot-thin-launcher/1.0.22.RELEASE/spring-boot-thin-launcher-1.0.22.RELEASE-exec.jar at org.springframework.boot.loader.wrapper.ThinJarWrapper.download(ThinJarWrapper.java:170) at org.springframework.boot.loader.wrapper.ThinJarWrapper.launch(ThinJarWrapper.java:130) at org.springframework.boot.loader.wrapper.ThinJarWrapper.main(ThinJarWrapper.java:107)

I think this is due to the thin-launcher tries to grab some dependencies at runtime and app engine file-system is read-only. I am thinking of a way to overcome this.

Let me know if you guys have any examples or deployments of using thin jars in app engine with Java 11 or is there other library that is being used to create the thin jars for app engine? I couldn't find much documentation on this except for few lines mentioned at https://cloud.google.com/appengine/docs/standard/java11/runtime#application_startup.

On the plus side it does seems to significantly improve the deployment time (size of jar seems significantly reduced). 😄

[SudharakaP]

I've created a PR. This adds support for "unjaring" the jar file within the staging directory as ludoch suggested. However I found no information of this process in google documentation; so it was a lot of trial and error. The only piece of documentation I found is; https://cloud.google.com/appengine/docs/standard/java11/runtime#application_startup

There it says and I quote;

With a custom entrypoint, you can construct and package your application as a thin JAR file which only contains your application code and direct dependencies. When deploying your application, the App Engine plugin will only upload the files that changed, rather than the entire uber JAR package.

However I couldn't get this approach to work; the problem is that creating a standard thin jar with the spring-bot-thin-launcher for example doesn't work since it tries to download the dependencies at startup and appengine doesn't allow writing to the disk. I wonder whether the documentation here is wrong (maybe AppEngine Java 11 still in Beta?).

Anyways, therefore this approach just extracts the content of the fatjar within the staging folder, and it seems to work; I've tested it on an actual deployment.

However when running appengine:deploy (or in the case of gradle appengineDeploy) it does the staging again. I mean as far as I know there's no way to do the deployment only and therefore I didn't find a easy method of removing the jar file from the staging folder. If you guys have any suggestions about this let me know. Otherwise I think we have to just keep the jar file within the staging folder?

Also I've uncovered some other issues with the GAE sub-generator (particularly on the Gradle side); I'll create some more issues in the future. 😄

cc: @saturnism @ludoch

[chanseokoh]

I was trying this out by constructing the thin jar using spring-boot-thin-launcher which I believe is the standard approach of doing this

I don't think this is any way close to being standard.

[SudharakaP]

@chanseokoh : Thanks for your input; could you please elaborate what it is when it comes to deploying spring boot applications in GAE? As per the following documents it says it can support thin jars but doesn't say exactly which method to use (maybe I missed it, correct me if I am wrong);

https://cloud.google.com/appengine/docs/standard/java11/runtime#application_startup

EDIT: I see you've commented in the GoogleCloudPlatform/app-maven-plugin#399; any further correspondence will be there. Thanks a bunch again 😄

[SudharakaP]

As per my discussion in GoogleCloudPlatform/app-maven-plugin#399 we need to change this approach. I will refactor the current pull request soon. 😄

[ludoch]

You could read https://github.com/ludoch/samples/blob/f4fcfb6ea1c7bc5bc37c36307e1c91a38f2445ff/java11/springboot/pom.xml
It does 2 things:
1/ explode the fatjar in an area
2/ delete the jar itself using a hack (I would love a better official way) by forcing a useless param in the plugin pom section so that the fatjar is not used in the final staging area... See line 57...

[SudharakaP]

@ludoch : My discussion in the other thread; specifically the following comment;

GoogleCloudPlatform/app-maven-plugin#399 (comment)

will enable to not use the fat jar approach at all I believe (and would not need the hack). Let me know if you agree. 😄

[SudharakaP]

@ludoch : Sorry, I misunderstood you there; yes hack is still needed to remove the fatjar. 👍 And thanks much 🥇

[SudharakaP]

I've made some changes to the pull request with ludoch's workaround for removing the original jar and I think we are good to go with this. 😄

[SudharakaP]

@deepu105 : Thanks for assigning a bug bounty for this; although not strictly necessary as I enjoyed doing it. I am getting a lot of coffees and teas for myself these days thanks to the extra pocket money. 🍵 🍵