Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

io.quarkus.kubernetes.deployment.OpenshiftDeployer throws java.io.IOException: Failed to resolve <host> but evidently did reach the host #31476

Closed
yrodiere opened this issue Feb 28, 2023 · 3 comments · Fixed by #31503
Labels
area/kubernetes kind/bug Something isn't working
Milestone

Comments

@yrodiere
Copy link
Member

Describe the bug

Since recently, with Quarkus 3.0.Alpha4, using quarkus.kubernetes.deploy=true leads to an error:

Error:  Failed to execute goal io.quarkus:quarkus-maven-plugin:3.0.0.Alpha4:build (default) on project quarkus-github-lottery: Failed to build quarkus application: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
Error: [ERROR] 	[error]: Build step io.quarkus.kubernetes.deployment.OpenshiftDeployer#checkEnvironment threw an exception: java.lang.RuntimeException: Failed to configure OpenShift. Make sure you have the Quarkus OpenShift extension.
Error:  	at io.quarkus.kubernetes.deployment.OpenshiftDeployer.lambda$checkEnvironment$0(OpenshiftDeployer.java:35)
Error:  	at java.base/java.util.Optional.ifPresent(Optional.java:178)
Error:  	at io.quarkus.kubernetes.deployment.OpenshiftDeployer.checkEnvironment(OpenshiftDeployer.java:22)
Error:  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Error:  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
Error:  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Error:  	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
Error:  	at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:909)
Error:  	at io.quarkus.builder.BuildContext.run(BuildContext.java:282)
Error:  	at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
Error:  	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)
Error:  	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1478)
Error:  	at java.base/java.lang.Thread.run(Thread.java:833)
Error:  	at org.jboss.threads.JBossThread.run(JBossThread.java:501)
Error:  Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
Error:  	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:129)
Error:  	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:122)
Error:  	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.restCall(OperationSupport.java:791)
Error:  	at io.fabric8.kubernetes.client.impl.BaseClient.getApiGroups(BaseClient.java:241)
Error:  	at io.fabric8.kubernetes.client.impl.BaseClient.hasApiGroup(BaseClient.java:164)
Error:  	at io.fabric8.openshift.client.impl.OpenShiftClientImpl.isSupported(OpenShiftClientImpl.java:742)
Error:  	at io.quarkus.kubernetes.deployment.OpenshiftDeployer.lambda$checkEnvironment$0(OpenshiftDeployer.java:28)
Error:  	... 13 more
Error:  Caused by: java.io.IOException: Failed to resolve 'api.ospo-osci.z3b1.p1.openshiftapps.com' and search domain query for configured domains failed as well: [lxlrj1mao4menpo2mo3udeoyab.bx.internal.cloudapp.net]
Error:  	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:535)
Error:  	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:558)
Error:  	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.restCall(OperationSupport.java:784)
Error:  	... 17 more
Error:  Caused by: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'api.ospo-osci.z3b1.p1.openshiftapps.com' and search domain query for configured domains failed as well: [lxlrj1mao4menpo2mo3udeoyab.bx.internal.cloudapp.net]
Error:  	at io.netty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:1069)
Error:  	at io.netty.resolver.dns.DnsResolveContext.tryToFinishResolve(DnsResolveContext.java:1022)
Error:  	at io.netty.resolver.dns.DnsResolveContext.query(DnsResolveContext.java:418)
Error:  	at io.netty.resolver.dns.DnsResolveContext.access$600(DnsResolveContext.java:66)
Error:  	at io.netty.resolver.dns.DnsResolveContext$2.operationComplete(DnsResolveContext.java:489)
Error:  	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
Error:  	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583)
Error:  	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559)
Error:  	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
Error:  	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
Error:  	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629)
Error:  	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118)
Error:  	at io.netty.resolver.dns.DnsQueryContext.tryFailure(DnsQueryContext.java:256)
Error:  	at io.netty.resolver.dns.DnsQueryContext.onQueryWriteCompletion(DnsQueryContext.java:193)
Error:  	at io.netty.resolver.dns.DnsQueryContext.access$200(DnsQueryContext.java:43)
Error:  	at io.netty.resolver.dns.InflightNameResolver.resolve(InflightNameResolver.java:66)
Error:  	at io.netty.resolver.dns.InflightNameResolver.resolve(InflightNameResolver.java:51)
Error:  	at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:55)
Error:  	at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:31)
Error:  	at io.netty.resolver.AbstractAddressResolver.resolve(AbstractAddressResolver.java:106)
Error:  	at io.netty.bootstrap.Bootstrap.doResolveAndConnect0(Bootstrap.java:206)
Error:  	at io.netty.bootstrap.Bootstrap.doResolveAndConnect(Bootstrap.java:162)
Error:  	at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:139)
Error:  	at io.vertx.core.net.impl.ChannelProvider.handleConnect(ChannelProvider.java:155)
Error:  	at io.vertx.core.net.impl.ChannelProvider.connect(ChannelProvider.java:106)
Error:  	at io.vertx.core.net.impl.ChannelProvider.connect(ChannelProvider.java:92)
Error:  	at io.vertx.core.net.impl.NetClientImpl.connectInternal2(NetClientImpl.java:273)
Error:  	at io.vertx.core.net.impl.NetClientImpl.lambda$connectInternal$1(NetClientImpl.java:240)
Error:  	at io.vertx.core.impl.future.FutureImpl$3.onSuccess(FutureImpl.java:141)
Error:  	at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:60)
Error:  	at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211)
Error:  	at io.vertx.core.impl.future.PromiseImpl.tryComplete(PromiseImpl.java:23)
Error:  	at io.vertx.core.impl.future.PromiseImpl.onSuccess(PromiseImpl.java:49)
Error:  	at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:60)
Error:  	at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211)
Error:  	at io.vertx.core.impl.future.FixedMapping.onSuccess(FixedMapping.java:31)
Error:  	at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:60)
Error:  	at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211)
Error:  	at io.vertx.core.impl.future.PromiseImpl.tryComplete(PromiseImpl.java:23)
Error:  	at io.vertx.core.impl.future.PromiseImpl.onSuccess(PromiseImpl.java:49)
Error:  	at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:60)
Error:  	at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211)
Error:  	at io.vertx.core.impl.future.Composition$1.onSuccess(Composition.java:62)
Error:  	at io.vertx.core.impl.future.FutureBase.lambda$emitSuccess$0(FutureBase.java:54)
Error:  	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
Error:  	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
Error:  	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
Error:  	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
Error:  	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
Error:  	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
Error:  	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Error:  	at java.base/java.lang.Thread.run(Thread.java:833): Unknown host Failed to resolve 'api.ospo-osci.z3b1.p1.openshiftapps.com' and search domain query for configured domains failed as well: [lxlrj1mao4menpo2mo3udeoyab.bx.internal.cloudapp.net]
Error:  -> [Help 1]
Error:  
Error:  To see the full stack trace of the errors, re-run Maven with the -e switch.
Error:  Re-run Maven using the -X switch to enable full debug logging.
Error:  
Error:  For more information about the errors and possible solutions, please read the following articles:
Error:  [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
Error: Process completed with exit code 1.

See https://github.com/quarkusio/quarkus-github-lottery/actions/runs/4290767409/jobs/7479679333#step:6:30555

OpenShift seems to be reachable though, since I can see the deployment causes changes in OpenShift (a build is triggered and the pod restarts). It's stuck on an older version of my application, though.

It's probably a coincidence, but this failure started started to appear right after I upgraded to Maven Compiler Plugin 3.11.0: quarkusio/quarkus-github-lottery@2f75ba2

Expected behavior

No response

Actual behavior

No response

How to Reproduce?

I don't have a reproducer but here is the project: https://github.com/quarkusio/quarkus-github-lottery/

Command:

mvn clean package -Dquarkus.kubernetes.deploy=true -Dquarkus.native.container-build=true -Dnative -Drevision=${{ github.sha }}

See https://github.com/quarkusio/quarkus-github-lottery/blob/4a0f8d6655025fffa099396854de89d2fc3c1d73/.github/workflows/deploy.yml#L36-L43

Configuration:

quarkus.application.name=quarkus-github-lottery
quarkus.application.version=${buildNumber:999-SNAPSHOT}

quarkus.qute.suffixes=md
quarkus.qute.content-types."md"=text/markdown

%test.quarkus.scheduler.enabled=false
%test.quarkus.log.category."io.quarkus.github.lottery".level=DEBUG

%dev.quarkus.scheduler.enabled=false
%dev.quarkus.log.category."io.quarkus.github.lottery".level=DEBUG

%prod.quarkus.openshift.labels."app"=quarkus-github-lottery
# renew the SSL certificate automatically
%prod.quarkus.openshift.annotations."kubernetes.io/tls-acme"=true
%prod.quarkus.openshift.env.configmaps=quarkus-github-lottery-config
%prod.quarkus.openshift.env.secrets=quarkus-github-lottery-secrets

See https://github.com/quarkusio/quarkus-github-lottery/blob/4a0f8d6655025fffa099396854de89d2fc3c1d73/src/main/resources/application.properties

Output of uname -a or ver

No response

Output of java -version

No response

GraalVM version (if different from Java)

No response

Quarkus version or git rev

3.0.0.Alpha4

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response

@yrodiere yrodiere added the kind/bug Something isn't working label Feb 28, 2023
@quarkus-bot
Copy link

quarkus-bot bot commented Feb 28, 2023

/cc @Sgitario (kubernetes), @geoand (kubernetes,openshift), @iocanel (kubernetes,openshift)

@gsmet
Copy link
Member

gsmet commented Feb 28, 2023

@Sgitario could you have a look at this one when you have some time? It's blocking the deployment of our support apps to OpenShift and we would like to push as many apps with Quarkus 3 to gather feedback. Thanks!

@Sgitario
Copy link
Contributor

Sgitario commented Mar 1, 2023

This is very related to #30951 which was meant to be fixed by #30952.

I've been able to reproduce it on native as stated in the issue description, but also randomly on JVM. I think the problem is related to restoring the disable vertx DNS resolver system property right after creating the vert.x instance, so I'm changing a bit the workflow to do this after the build is finished (when the K8s resources will have been pushed to Kubernetes/OpenShift already).

Sgitario added a commit to Sgitario/quarkus that referenced this issue Mar 2, 2023
There were several places where a new KubernetesClient was being created:
- Some places use `Clients.fromConfig`.
- Other places use `KubernetesClientUtils.createConfig`.
- And other places use `KubernetesClientBuildItem.getClient`

And the problem is that every time we invoke any of these methods, Fabric8 KubernetesClient tries to locate one HttpClient.Factory and it seems that the logic to get one HttpClient.Factory sometimes gets the `VertxHttpClientFactory` implementation over the expected one `QuarkusHttpClientFactory`

With this solution, it avoids to find a HttpClient.Factory using the ServiceLoader logic in Fabric8 Kubernetes Client, but it provides the expected `QuarkusHttpClientFactory` implementation always.

Fix quarkusio#31476
Sgitario added a commit to Sgitario/quarkus that referenced this issue Mar 2, 2023
There were several places where a new KubernetesClient was being created:
- Some places use `Clients.fromConfig`.
- Other places use `KubernetesClientUtils.createConfig`.
- And other places use `KubernetesClientBuildItem.getClient`

And the problem is that every time we invoke any of these methods, Fabric8 KubernetesClient tries to locate one HttpClient.Factory and it seems that the logic to get one HttpClient.Factory sometimes gets the `VertxHttpClientFactory` implementation over the expected one `QuarkusHttpClientFactory`

With this solution, it avoids to find a HttpClient.Factory using the ServiceLoader logic in Fabric8 Kubernetes Client, but it provides the expected `QuarkusHttpClientFactory` implementation always.

Fix quarkusio#31476
Sgitario added a commit to Sgitario/quarkus that referenced this issue Mar 2, 2023
There were several places where a new KubernetesClient was being created:
- Some places use `Clients.fromConfig`.
- Other places use `KubernetesClientUtils.createConfig`.
- And other places use `KubernetesClientBuildItem.getClient`

And the problem is that every time we invoke any of these methods, Fabric8 KubernetesClient tries to locate one HttpClient.Factory and it seems that the logic to get one HttpClient.Factory sometimes gets the `VertxHttpClientFactory` implementation over the expected one `QuarkusHttpClientFactory`

With this solution, it avoids to find a HttpClient.Factory using the ServiceLoader logic in Fabric8 Kubernetes Client, but it provides the expected `QuarkusHttpClientFactory` implementation always.

Fix quarkusio#31476
Sgitario added a commit to Sgitario/quarkus that referenced this issue Mar 2, 2023
There were several places where a new KubernetesClient was being created:
- Some places use `Clients.fromConfig`.
- Other places use `KubernetesClientUtils.createConfig`.
- And other places use `KubernetesClientBuildItem.getClient`

And the problem is that every time we invoke any of these methods, Fabric8 KubernetesClient tries to locate one HttpClient.Factory and it seems that the logic to get one HttpClient.Factory sometimes gets the `VertxHttpClientFactory` implementation over the expected one `QuarkusHttpClientFactory`

With this solution, it avoids to find a HttpClient.Factory using the ServiceLoader logic in Fabric8 Kubernetes Client, but it provides the expected `QuarkusHttpClientFactory` implementation always.

Fix quarkusio#31476
Sgitario added a commit to Sgitario/quarkus that referenced this issue Mar 3, 2023
There were several places where a new KubernetesClient was being created:
- Some places use `Clients.fromConfig`.
- Other places use `KubernetesClientUtils.createConfig`.
- And other places use `KubernetesClientBuildItem.getClient`

And the problem is that every time we invoke any of these methods, Fabric8 KubernetesClient tries to locate one HttpClient.Factory and it seems that the logic to get one HttpClient.Factory sometimes gets the `VertxHttpClientFactory` implementation over the expected one `QuarkusHttpClientFactory`

With this solution, it avoids to find a HttpClient.Factory using the ServiceLoader logic in Fabric8 Kubernetes Client, but it provides the expected `QuarkusHttpClientFactory` implementation always.

Fix quarkusio#31476
@quarkus-bot quarkus-bot bot added this to the 3.0 - main milestone Mar 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/kubernetes kind/bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants