Deploying to OpenShift via openshift extension not working in 2.7.1.Final

[edeandrea]

Describe the bug

In Quarkus 2.7.1.Final using the openshift extension, doing ./mvnw clean package -DskipTests -Dquarkus.kubernetes.deployment-target=openshift -Dquarkus.container-image.builder=openshift -Dquarkus.openshift.resources.limits.memory=768Mi -Dquarkus.openshift.resources.requests.memory=256Mi -Dquarkus.kubernetes.deploy=true -Dquarkus.openshift.env.vars.JAVA_DEBUG=true -Dquarkus.container-image.build=true does not work.

If I back down to Quarkus 2.7.0.Final it works fine.

Expected behavior

I would expect the app to be deployed to OpenShift

Actual behavior

It does not work. This is the log:

[INFO] --- quarkus-maven-plugin:2.7.1.Final:build (default) @ rest-villains ---
[WARNING] [io.quarkus.kubernetes.deployment.KubernetesDeployer] An openshift deployment was requested, but the container image group:quarkus-super-heroes is not aligned with the currently selected project:eric-deandrea-dev.it is strongly advised to align them, or else the image might not be reachable.
[INFO] [org.hibernate.Version] HHH000412: Hibernate ORM core version 5.6.5.Final
[INFO] [io.quarkus.kubernetes.deployment.KubernetesDeploy] Kubernetes API Server at 'https://api.sandbox-m2.ll9k.p1.openshiftapps.com:6443/' successfully contacted.
[INFO] Checking for existing resources in: /Users/edeandre/workspaces/quarkus/quarkus-super-heroes/rest-villains/src/main/kubernetes.
[INFO] Adding existing Secret with name: villains-db-config.
[INFO] Adding existing Service with name: villains-db.
[INFO] Adding existing ConfigMap with name: rest-villains-config.
[INFO] Adding existing ConfigMap with name: villains-db-init.
[INFO] Adding existing Deployment with name: villains-db.
[INFO] Adding existing Secret with name: villains-db-config.
[INFO] Adding existing Service with name: villains-db.
[INFO] Adding existing ConfigMap with name: rest-villains-config.
[INFO] Adding existing ConfigMap with name: villains-db-init.
[INFO] Adding existing Deployment with name: villains-db.
[INFO] [io.quarkus.container.image.openshift.deployment.OpenshiftProcessor] Starting (in-cluster) container image build for jar using: BINARY on server: https://api.sandbox-m2.ll9k.p1.openshiftapps.com:6443/ in namespace:eric-deandrea-dev.
[INFO] [io.quarkus.container.image.openshift.deployment.OpenshiftProcessor] Applied: Secret villains-db-config
[INFO] [io.quarkus.container.image.openshift.deployment.OpenshiftProcessor] Applied: ImageStream rest-villains
[INFO] [io.quarkus.container.image.openshift.deployment.OpenshiftProcessor] Applied: BuildConfig rest-villains
[ERROR] Failed to upload archive file for the build: rest-villains
[ERROR] Please check cluster events via `oc get events` to see what could have possibly gone wrong
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  02:27 min
[INFO] Finished at: 2022-02-17T08:27:06-05:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal io.quarkus.platform:quarkus-maven-plugin:2.7.1.Final:build (default) on project rest-villains: Failed to build quarkus application: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
[ERROR]         [error]: Build step io.quarkus.container.image.openshift.deployment.OpenshiftProcessor#openshiftBuildFromJar threw an exception: io.fabric8.kubernetes.client.KubernetesClientException: Operation: [list]  for kind: [Event]  with name: [null]  in namespace: [eric-deandrea-dev]  failed.
[ERROR]         at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:130)
[ERROR]         at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:140)
[ERROR]         at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:415)
[ERROR]         at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:404)
[ERROR]         at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:83)
[ERROR]         at io.fabric8.openshift.client.dsl.internal.build.BuildConfigOperationsImpl.getRecentEvents(BuildConfigOperationsImpl.java:296)
[ERROR]         at io.fabric8.openshift.client.dsl.internal.build.BuildConfigOperationsImpl.submitToApiServer(BuildConfigOperationsImpl.java:290)
[ERROR]         at io.fabric8.openshift.client.dsl.internal.build.BuildConfigOperationsImpl.fromFile(BuildConfigOperationsImpl.java:171)
[ERROR]         at io.fabric8.openshift.client.dsl.internal.build.BuildConfigOperationsImpl.fromFile(BuildConfigOperationsImpl.java:66)
[ERROR]         at io.quarkus.container.image.openshift.deployment.OpenshiftProcessor.openshiftBuild(OpenshiftProcessor.java:431)
[ERROR]         at io.quarkus.container.image.openshift.deployment.OpenshiftProcessor.lambda$openshiftBuild$10(OpenshiftProcessor.java:412)
[ERROR]         at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
[ERROR]         at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
[ERROR]         at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
[ERROR]         at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654)
[ERROR]         at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
[ERROR]         at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
[ERROR]         at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
[ERROR]         at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
[ERROR]         at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
[ERROR]         at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
[ERROR]         at io.quarkus.container.image.openshift.deployment.OpenshiftProcessor.openshiftBuild(OpenshiftProcessor.java:412)
[ERROR]         at io.quarkus.container.image.openshift.deployment.OpenshiftProcessor.createContainerImage(OpenshiftProcessor.java:356)
[ERROR]         at io.quarkus.container.image.openshift.deployment.OpenshiftProcessor.openshiftBuildFromJar(OpenshiftProcessor.java:258)
[ERROR]         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR]         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[ERROR]         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR]         at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[ERROR]         at io.quarkus.deployment.ExtensionLoader$2.execute(ExtensionLoader.java:882)
[ERROR]         at io.quarkus.builder.BuildContext.run(BuildContext.java:277)
[ERROR]         at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
[ERROR]         at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)
[ERROR]         at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1478)
[ERROR]         at java.base/java.lang.Thread.run(Thread.java:834)
[ERROR]         at org.jboss.threads.JBossThread.run(JBossThread.java:501)
[ERROR] Caused by: java.net.SocketTimeoutException: timeout
[ERROR]         at okio.Okio$4.newTimeoutException(Okio.java:232)
[ERROR]         at okio.AsyncTimeout.exit(AsyncTimeout.java:286)
[ERROR]         at okio.AsyncTimeout$2.read(AsyncTimeout.java:241)
[ERROR]         at okio.RealBufferedSource.indexOf(RealBufferedSource.java:358)
[ERROR]         at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:230)
[ERROR]         at okhttp3.internal.http1.Http1ExchangeCodec.readHeaderLine(Http1ExchangeCodec.java:242)
[ERROR]         at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.java:213)
[ERROR]         at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.java:115)
[ERROR]         at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:94)
[ERROR]         at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
[ERROR]         at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:43)
[ERROR]         at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
[ERROR]         at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
[ERROR]         at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
[ERROR]         at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
[ERROR]         at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
[ERROR]         at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
[ERROR]         at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
[ERROR]         at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
[ERROR]         at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
[ERROR]         at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
[ERROR]         at io.fabric8.kubernetes.client.okhttp.OkHttpClientBuilderImpl$InteceptorAdapter.intercept(OkHttpClientBuilderImpl.java:62)
[ERROR]         at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
[ERROR]         at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
[ERROR]         at io.fabric8.kubernetes.client.okhttp.OkHttpClientBuilderImpl$InteceptorAdapter.intercept(OkHttpClientBuilderImpl.java:62)
[ERROR]         at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
[ERROR]         at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
[ERROR]         at io.fabric8.kubernetes.client.okhttp.OkHttpClientBuilderImpl$InteceptorAdapter.intercept(OkHttpClientBuilderImpl.java:62)
[ERROR]         at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
[ERROR]         at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
[ERROR]         at io.fabric8.kubernetes.client.okhttp.OkHttpClientBuilderImpl$InteceptorAdapter.intercept(OkHttpClientBuilderImpl.java:62)
[ERROR]         at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
[ERROR]         at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
[ERROR]         at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229)
[ERROR]         at okhttp3.RealCall.execute(RealCall.java:81)
[ERROR]         at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl.send(OkHttpClientImpl.java:138)
[ERROR]         at io.fabric8.kubernetes.client.dsl.base.OperationSupport.retryWithExponentialBackoff(OperationSupport.java:574)
[ERROR]         at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:553)
[ERROR]         at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:518)
[ERROR]         at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:502)
[ERROR]         at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:133)
[ERROR]         ... 33 more
[ERROR] Caused by: java.net.SocketTimeoutException: Read timed out
[ERROR]         at java.base/java.net.SocketInputStream.socketRead0(Native Method)
[ERROR]         at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
[ERROR]         at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168)
[ERROR]         at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
[ERROR]         at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:448)
[ERROR]         at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:68)
[ERROR]         at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1104)
[ERROR]         at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:823)
[ERROR]         at okio.Okio$2.read(Okio.java:140)
[ERROR]         at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
[ERROR]         ... 71 more
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

How to Reproduce?

I'm running on the Red Hat Developer Sandbox

1. Clone https://github.com/quarkusio/quarkus-super-heroes
2. cd rest-villains
3. Run ./mvnw clean package -DskipTests -Dquarkus.kubernetes.deployment-target=openshift -Dquarkus.container-image.builder=openshift -Dquarkus.openshift.resources.limits.memory=768Mi -Dquarkus.openshift.resources.requests.memory=256Mi -Dquarkus.kubernetes.deploy=true -Dquarkus.openshift.env.vars.JAVA_DEBUG=true -Dquarkus.container-image.build=true

This will fail. If you bump the Quarkus version in rest-villains/pom.xml down to 2.7.0.Final and re-try, it works fine.

Output of uname -a or ver

Darwin edeandre-mac 21.3.0 Darwin Kernel Version 21.3.0: Wed Jan 5 21:37:58 PST 2022; root:xnu-8019.80.24~20/RELEASE_X86_64 x86_64

Output of java -version

openjdk 11.0.2 2019-01-15

GraalVM version (if different from Java)

No response

Quarkus version or git rev

2.7.1.Final

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
Maven home: /Users/edeandre/.m2/wrapper/dists/apache-maven-3.8.4-bin/52ccbt68d252mdldqsfsn03jlf/apache-maven-3.8.4
Java version: 11.0.2, vendor: Oracle Corporation, runtime: /Users/edeandre/.sdkman/candidates/java/11.0.2-open
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.16", arch: "x86_64", family: "mac"

Additional information

No response

[quarkus-bot]

/cc @geoand, @iocanel

[geoand]

Are you sure this isn't just some transient failure?

[edeandrea]

It happens every time I try it. I was banging my head against my desk most of yesterday afternoon trying to figure out what the problem might be. Same this morning.

As soon as I switched to Quarkus 2.7.0.Final it worked 1st time.

[geoand]

Interesting, thanks.

@iocanel @Sgitario sounds like a severe regression worth looking into...

[Sgitario]

I think that the problem is that, in 2.7.1, the ImageStream resource that should be populated from the Build job output (s2i from binary), contains a wrong spec: dockerImageRepository: quay.io/quarkus-super-heroes/rest-villains. I guess this is the problematic code: https://github.com/quarkusio/quarkus/pull/23239/files#diff-2f7f8b1e6f4a44f6f156a44d9a60ec9f4a233e7874e8059fb0c627771a505fc4R242

@iocanel to confirm!

[iocanel]

I can reproduce the issue.
However, I am not able to see spot anything obvious.
@Sgitario I don't see where: dockerImageRepository: quay.io/quarkus-super-heroes/rest-villains comes from.
Will keep looking.

[iocanel]

What I do see, is that we are missing the ImageStream of the builder image.

[edeandrea]

Will this make it into a 2.7.x backport?

[geoand]

See the label in the link PR