In OpenShift mode layers extracted via the SpringBootGenerator are not used as Docker layers

[friedeas]

Describe the bug

Should be tackled after #2393

Requires at least an integration test (e2e) for OpenShift build in Docker mode.

The enhancement (Fix #1674) introduced in version 1.14.0 doesn't have the expected effect if the OpenShift plugins are used with the build strategy Docker. This applies for both plugins Maven and Gradle. If the Spring Boot Generator is used the Spring Boot layered jar is detected, but the following image generation only contains one COPY STEP for the complete Spring Boot Application. So the extracted layers are not used.

Eclipse JKube version

1.14.0

Component

OpenShift Gradle Plugin

Apache Maven version

3.8.4

Gradle version

other (please specify in additional context)

Steps to reproduce

1. Create a project using the plugin id 'org.eclipse.jkube.openshift' version '1.14.0'
2. Configure the OpenShift plugin
3. Set the build strategy to Docker buildStrategy = 'docker'
4. Use the Spring Boot generator includes = ['spring-boot']
5. Execute Gradle clean build
6. Execute Gradle ocBuild ocResource ocApply (requires a configured OpenShift runtime)
7. Build and deploy will be executed an only one COPY STEP will be shown in the logs for the Spring Boot application like STEP 5/9: COPY /deployments /deployments/

Expected behavior

Depending on the Spring Boot layered jar content several COPY STEPs for the included layers, like application, dependencies, snapshot-dependencies and spring-boot-loader, should be executed during the image creation.

Runtime

OpenShift

Kubernetes API Server version

other (please specify in additional context)

Environment

Windows

Eclipse JKube Logs

> Task :ocBuild
oc: Running generator spring-boot
oc: spring-boot: Using Docker image quay.io/jkube/jkube-java:0.0.19 as base / builder
oc: Spring Boot layered jar detected
oc: Using OpenShift build with strategy Docker
oc: [XXXXXXX-XXXX:latest] "spring-boot": Created docker source tar C:\Projects\XX\XX XXXXXXX XXXXXXXXXXX XXXXX\XXXXXXXX-XXXX\build\docker\XXXXXXX-XXXX\latest\tmp\docker-build.tar
oc: Creating BuildServiceConfig XXXXXXX-XXXX-s2i for Docker build
oc: Creating ImageStream XXXXXXX-XXXX
oc: Starting Build XXXXXXX-XXXX-XXX
oc: Waiting for build XXXXXXX-XXXX-XXX-1 to complete...
oc: Adding cluster TLS certificate authority to trust store
oc: time="2023-09-13T10:28:21Z" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"
oc: I0913 10:28:21.043407       1 defaults.go:102] Defaulting to storage driver "overlay" with options [mountopt=metacopy=on].
oc: Caching blobs under "/var/cache/blobs".
oc:
oc: Pulling image quay.io/jkube/jkube-java:0.0.19 ...
oc: Trying to pull quay.io/jkube/jkube-java:0.0.19...
oc: Getting image source signatures
oc: Copying blob sha256:e9452697801f8500f5485fe233be1da041c288129050a5d60796fb456d46d2d6
oc: Copying blob sha256:36c12cb044acc178c78bbb4240cad63e4d7f790d1e4ef62da05a7d00969efceb
oc: Copying config sha256:d03f69fe626dbee93262db9571e27c1fa663ff8fbad0b2129268e7098f1d18a5
oc: Writing manifest to image destination
oc: Storing signatures
oc: Adding transient rw bind mount for /run/secrets/rhsm
oc: STEP 1/9: FROM quay.io/jkube/jkube-java:0.0.19
oc: STEP 2/9: ENV JAVA_MAIN_CLASS=org.springframework.boot.loader.JarLauncher JAVA_OPTIONS=-Dspring.output.ansi.enabled=always JAVA_APP_DIR=/deployments
oc: --> b5b59488384
oc: STEP 3/9: LABEL org.label-schema.description=XXXXXXX-XXXX org.label-schema.version=0.0.1-SNAPSHOT org.label-schema.schema-version=1.0 org.label-schema.build-date=2023-09-13T12:23:42.334832600 org.label-schema.name=XXXXXXX-XXXX org.label-schema.vcs-ref=0f1e630bc5bd26aoc: --> f81d76225ac
oc: STEP 4/9: EXPOSE 8080 8778 9779
oc: --> 4f7f31d01bd
oc: STEP 5/9: COPY /deployments /deployments/
oc: time="2023-09-13T10:28:36Z" level=warning msg="Adding metacopy option, configured globally"
oc: --> db4a0897c10
oc: STEP 6/9: WORKDIR /deployments
oc: --> ff23d6aa7c4
oc: STEP 7/9: ENTRYPOINT ["java","org.springframework.boot.loader.JarLauncher"]
oc: --> 2a0203c5037
oc: STEP 8/9: ENV "OPENSHIFT_BUILD_NAME"="XXXXXXX-XXXX-s2i-1" "OPENSHIFT_BUILD_NAMESPACE"="XX-XXX-XX-XXXXXXX-XXXX"
oc: --> 85a39d1fe33
oc: STEP 9/9: LABEL "io.openshift.build.name"="XXXXXXX-XXXX-s2i-1" "io.openshift.build.namespace"="XX-XXX-XX-XXXXXXX-XXXX"
oc: COMMIT temp.builder.openshift.io/XX-XXX-XX-XXXXXXX-XXXX/XXXXXXX-XXXX-s2i-1:6c3cd196
oc: --> 7ff4b2a0c49
oc: Successfully tagged temp.builder.openshift.io/XX-XXX-XX-XXXXXXX-XXXX/XXXXXXX-XXXX-s2i-1:6c3cd196
oc: 7ff4b2a0c49234fc62e258efdd7b385299b45b702827b48c25143fe56f457108

Sample Reproducer Project

No response

Additional context

Gradle version 8.3
Kubernetes API Server version v1.23.17+16bcd69

[rohanKanojia]

I've created #2413 that should not flatten Spring Boot image layers when in docker build strategy. However, while testing it I'm not able to see any kind of caching by docker host inside OpenShift.

We're correctly passing noCache: false inside the BuildConfig's build strategy. There is no difference in Dockerfile generated in consecutive builds either. Not sure if this is something specific to OpenShift.

[rohanKanojia]

Okay, this seems to change in behavior from OpenShift 3 to OpenShift 4 as specified in this OpenShift blogpost

With OpenShift 4 we undertook a fundamental redesign of how images are built on the platform. Instead of relying on a daemon on the host to manage containers, image creation, and image pushing, we are leveraging Buildah running inside our build pods. This aligns with the general OpenShift 4 theme of making everything “just another pod.”
The image cache is no longer shared. When using the host’s docker daemon, images and layers that had been pulled to the host previously could be reused by a subsequent build on that same host. Since builds no longer have access to the host, each build must pull any layers it requires each time the build is run. This helps improve the security posture of builds by providing an additional level of isolation, however it will also increase the time required to execute a build. We will be looking to reintroduce a shared image cache in the future.

When building the project against OpenShift 3.11.0 (minishift 1.34.3), I'm able to see docker cache getting used :

jkube-spring-boot3-openshift-docker-image-push : $ gradle ocBuild
To honour the JVM settings for this build a single-use Daemon process will be forked. See https://docs.gradle.org/7.6.1/userguide/gradle_daemon.html#sec:disabling_the_daemon.
Daemon will be stopped at the end of the build 

> Task :ocBuild
oc: Running generator spring-boot
oc: spring-boot: Using Docker image quay.io/jkube/jkube-java:0.0.19 as base / builder
oc: Spring Boot layered jar detected
oc: Using OpenShift build with strategy Docker
oc: [docker.io/testuser/jkube-spring-boot3-openshift-docker-image-push:0.0.1] "spring-boot": Created docker source tar /home/rokumar/work/repos/jkube-testing/jkube-spring-boot3-openshift-docker-image-push/build/docker/docker.io/testuser/jkube-spring-boot3-openshift-docker-image-push/0.0.1/tmp/docker-build.tar
oc: Creating BuildServiceConfig jkube-spring-boot3-openshift-docker-image-push-s2i for Docker build
oc: Starting Build jkube-spring-boot3-openshift-docker-image-push-s2i
oc: Waiting for build jkube-spring-boot3-openshift-docker-image-push-s2i-1 to complete...
oc: Step 1/11 : FROM quay.io/jkube/jkube-java:0.0.19
oc:  ---> d03f69fe626d
oc: Step 2/11 : ENV JAVA_MAIN_CLASS org.springframework.boot.loader.JarLauncher JAVA_APP_DIR /deployments
oc:  ---> Using cache
oc:  ---> 534191c61048
oc: Step 3/11 : LABEL org.label-schema.description "" org.label-schema.version 0.0.1 org.label-schema.schema-version 1.0 org.label-schema.build-date 2023-10-06 org.label-schema.name jkube-spring-boot3-openshift-docker-image-push org.label-schema.vcs-ref 706fa97b06efd6ac6c048b0ade461b764eb91a66 org.label-schema.vcs-url [email protected]:rohankanojia-forks/jkube-testing.git
oc:  ---> Running in 9d9037b9cad2
oc:  ---> dee416cb53e4
oc: Removing intermediate container 9d9037b9cad2
oc: Step 4/11 : EXPOSE 8080 8778 9779
oc:  ---> Running in 4a6c9335db0c
oc:  ---> 4d01dcebed03CUTING [13s]
oc: Removing intermediate container 4a6c9335db0c
oc: Step 5/11 : COPY /dependencies/deployments /deployments/
oc:  ---> 1e48316c126aCUTING [13s]
oc: Removing intermediate container 4106c273dae8
oc: Step 6/11 : COPY /spring-boot-loader/deployments /deployments/
oc:  ---> b019b1dc6a80CUTING [13s]
oc: Removing intermediate container ff1b7eb6a4a0
oc: Step 7/11 : COPY /application/deployments /deployments/
oc:  ---> 30a2e03788bbCUTING [14s]
oc: Removing intermediate container 1c3e5fbdf135
oc: Step 8/11 : WORKDIR /deployments
oc:  ---> 3179904561f2CUTING [14s]
oc: Removing intermediate container 04b1f34e3700
oc: Step 9/11 : ENTRYPOINT java org.springframework.boot.loader.JarLauncher
oc:  ---> Running in d200ef9cae87
oc:  ---> 163e7be19769CUTING [14s]
oc: Removing intermediate container d200ef9cae87
oc: Step 10/11 : ENV "OPENSHIFT_BUILD_NAME" "jkube-spring-boot3-openshift-docker-image-push-s2i-1" "OPENSHIFT_BUILD_NAMESPACE" "myproject"
oc:  ---> Running in 0354276924d1]
oc:  ---> 7a42ef023f3eCUTING [14s]
oc: Removing intermediate container 0354276924d1
oc: Step 11/11 : LABEL "io.openshift.build.name" "jkube-spring-boot3-openshift-docker-image-push-s2i-1" "io.openshift.build.namespace" "myproject"
oc:  ---> Running in a2215e45ec9f]
oc:  ---> b39d8814f1b1
oc: Removing intermediate container a2215e45ec9f
oc: Successfully built b39d8814f1b1
oc: Pushing image docker.io/testuser/jkube-spring-boot3-openshift-docker-image-push:0.0.1 ...
oc: Pushed 2/5 layers, 43% complete
oc: Pushed 3/5 layers, 97% complete
oc: Pushed 4/5 layers, 100% complete
oc: Pushed 5/5 layers, 100% complete
oc: Push successful
oc: Build jkube-spring-boot3-openshift-docker-image-push-s2i-1 in status Complete

BUILD SUCCESSFUL in 36s
1 actionable task: 1 executed
jkube-spring-boot3-openshift-docker-image-push : $ gradle ocBuild
To honour the JVM settings for this build a single-use Daemon process will be forked. See https://docs.gradle.org/7.6.1/userguide/gradle_daemon.html#sec:disabling_the_daemon.
Daemon will be stopped at the end of the build 

> Task :ocBuild
oc: Running generator spring-boot
oc: spring-boot: Using Docker image quay.io/jkube/jkube-java:0.0.19 as base / builder
oc: Spring Boot layered jar detected
oc: Using OpenShift build with strategy Docker
oc: [docker.io/testuser/jkube-spring-boot3-openshift-docker-image-push:0.0.1] "spring-boot": Created docker source tar /home/rokumar/work/repos/jkube-testing/jkube-spring-boot3-openshift-docker-image-push/build/docker/docker.io/testuser/jkube-spring-boot3-openshift-docker-image-push/0.0.1/tmp/docker-build.tar
oc: Updating BuildServiceConfig jkube-spring-boot3-openshift-docker-image-push-s2i for Docker strategy:ocBuild
oc: Starting Build jkube-spring-boot3-openshift-docker-image-push-s2i
oc: Waiting for build jkube-spring-boot3-openshift-docker-image-push-s2i-2 to complete...
oc: Step 1/11 : FROM quay.io/jkube/jkube-java:0.0.19
oc:  ---> d03f69fe626d
oc: Step 2/11 : ENV JAVA_MAIN_CLASS org.springframework.boot.loader.JarLauncher JAVA_APP_DIR /deployments
oc:  ---> Using cache
oc:  ---> 534191c61048
oc: Step 3/11 : LABEL org.label-schema.description "" org.label-schema.version 0.0.1 org.label-schema.schema-version 1.0 org.label-schema.build-date 2023-10-06 org.label-schema.name jkube-spring-boot3-openshift-docker-image-push org.label-schema.vcs-ref 706fa97b06efd6ac6c048b0ade461b764eb91a66 org.label-schema.vcs-url [email protected]:rohankanojia-forks/jkube-testing.git
oc:  ---> Using cache
oc:  ---> dee416cb53e4
oc: Step 4/11 : EXPOSE 8080 8778 9779
oc:  ---> Using cache
oc:  ---> 4d01dcebed03
oc: Step 5/11 : COPY /dependencies/deployments /deployments/
oc:  ---> Using cache
oc:  ---> 1e48316c126a
oc: Step 6/11 : COPY /spring-boot-loader/deployments /deployments/
oc:  ---> Using cache
oc:  ---> b019b1dc6a80
oc: Step 7/11 : COPY /application/deployments /deployments/
oc:  ---> Using cache
oc:  ---> 30a2e03788bb
oc: Step 8/11 : WORKDIR /deployments
oc:  ---> Using cache
oc:  ---> 3179904561f2
oc: Step 9/11 : ENTRYPOINT java org.springframework.boot.loader.JarLauncher
oc:  ---> Using cache
oc:  ---> 163e7be19769
oc: Step 10/11 : ENV "OPENSHIFT_BUILD_NAME" "jkube-spring-boot3-openshift-docker-image-push-s2i-2" "OPENSHIFT_BUILD_NAMESPACE" "myproject"
oc:  ---> Running in 84381f68f2d7
oc:  ---> 4e11ce6a7b7a
oc: Removing intermediate container 84381f68f2d7
oc: Step 11/11 : LABEL "io.openshift.build.name" "jkube-spring-boot3-openshift-docker-image-push-s2i-2" "io.openshift.build.namespace" "myproject"
oc:  ---> Running in 464510916c4b
oc:  ---> 342e40203bff
oc: Removing intermediate container 464510916c4b
oc: Successfully built 342e40203bff
oc: Pushing image docker.io/testuser/jkube-spring-boot3-openshift-docker-image-push:0.0.1 ...
oc: Push successful
oc: Build jkube-spring-boot3-openshift-docker-image-push-s2i-2 in status Complete

BUILD SUCCESSFUL in 26s
1 actionable task: 1 executed
jkube-spring-boot3-openshift-docker-image-push : $ 

I think we can provide individual layers in container image as extracted by spring boot, but it won't provide caching benefits on OpenShift 4 due to no shared docker cache.