Fail to redeploy to Kubernetes

[kenfinnigan]

Describe the bug

When running mvn clean package -Dquarkus.kubernetes.deploy=true to update an existing deployment in Kubernetes, it fails with:

Build step io.quarkus.kubernetes.deployment.KubernetesDeployer#deploy threw an exception: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://192.168.64.75:8443/api/v1/namespaces/default/services. Message: Service "my-app" is invalid: spec.ports[0].nodePort: Invalid value: 31084: provided port is already allocated.

Ths previously worked in 1.x

Expected behavior

Image is built and updated deployment rolled out to Kubernetes

Actual behavior

Exception when running mvn clean package -Dquarkus.kubernetes.deploy=true if application already deployed

How to Reproduce?

No response

Output of uname -a or ver

No response

Output of java -version

No response

GraalVM version (if different from Java)

No response

Quarkus version or git rev

No response

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response

[quarkus-bot]

/cc @geoand, @iocanel

[geoand]

This looks like a bug in the Kubernetes Client.
Basically when we do https://github.com/quarkusio/quarkus/blob/2.2.0.Final/extensions/kubernetes/vanilla/deployment/src/main/java/io/quarkus/kubernetes/deployment/KubernetesDeployer.java#L181, the client fails with something like:

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://192.168.49.2:8443/api/v1/namespaces/default/services. Message: Service "getting-started" is invalid: spec.ports[0].nodePort: Invalid value: 30974: provided port is already allocated. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec.ports[0].nodePort, message=Invalid value: 30974: provided port is already allocated, reason=FieldValueInvalid, additionalProperties={})], group=null, kind=Service, name=getting-started, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Service "getting-started" is invalid: spec.ports[0].nodePort: Invalid value: 30974: provided port is already allocated, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
[ERROR]         at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:686)
[ERROR]         at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:625)
[ERROR]         at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:565)
[ERROR]         at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:526)
[ERROR]         at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:314)
[ERROR]         at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:753)
[ERROR]         at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:88)
[ERROR]         at io.fabric8.kubernetes.client.dsl.base.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:61)
[ERROR]         at io.fabric8.kubernetes.client.utils.CreateOrReplaceHelper.createOrReplace(CreateOrReplaceHelper.java:47)
[ERROR]         at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:339)
[ERROR]         at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:88)
[ERROR]         at io.fabric8.kubernetes.client.dsl.internal.NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableImpl.createOrReplace(NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableImpl.java:106)
[ERROR]         at io.fabric8.kubernetes.client.dsl.internal.NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableImpl.createOrReplace(NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableImpl.java:64)
[ERROR]         at io.quarkus.kubernetes.deployment.KubernetesDeployer.lambda$deploy$4(KubernetesDeployer.java:181)
[ERROR]         at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
[ERROR]         at io.quarkus.kubernetes.deployment.KubernetesDeployer.deploy(KubernetesDeployer.java:177)
[ERROR]         at io.quarkus.kubernetes.deployment.KubernetesDeployer.deploy(KubernetesDeployer.java:102)

@manusa mind taking a look please?

[manusa]

Hi,

This is related to fabric8io/kubernetes-client#3122 and probably caused by the change of behavior in fabric8io/kubernetes-client#3221 (comment).

I haven't seen (yet) how this is performed in Quarkus. In JKube we switched to a patch operation when applying/reapplying service (eclipse-jkube/jkube#756 & eclipse-jkube/jkube#757). Most likely some similar change needs to be done in the Quarkus extension (again, haven't checked yet).

Related changes in Quarkus

• Kubernetes deploy should always delete previously installed resources before installing new ones. #13953
• Make KubernetesDeployer always delete existing resources #15262
• OpenShift plugin not deploying the Quarkus App (Regression Issue) #15487
• Revert "fix: KubernetesDeployer now always deletes existing resources" #15508

[geoand]

Hm... That might be a little tricky in Quarkus as don't keep any state on whether it's the first deploy or not.

[manusa]

Hm... That might be a little tricky in Quarkus as don't keep any state on whether it's the first deploy or not.

Keeping the state is not necessary. Worst case scenario the cluster should be checked for the existence of the Service.
But most probably, if #edit was being used, the fix should be as trivial as changing to #patch. But I need to check.

[geoand]

Okay. If you need any help, let me know

[manusa]

This issue is tightly related with the change of behavior of the resource deployment that has intermittently changed both in 1.1x.x and 2.x versions before and after some regression report (#15487).

So probably this issue is also reproducible in the latest 1.x (LTS) release ¿anyone can confirm?.

[Sgitario]

I could not reproduce this issue either using a local K8s environment (kind) or using OpenShift (with the k8s and container jib extensions). The command I tried:

mvn clean package -Dquarkus.kubernetes.deploy=true -Dquarkus.container-image.registry=quay.io -Dquarkus.container-image.group=xxx

I've also tried to use different services types (from node-port to clusterip and loadbalancer).

Every time that I redeployed the app after making some changes, everything worked fine (and the changes took effect):

[WARNING] [io.quarkus.container.image.jib.deployment.JibProcessor] Base image 'fabric8/java-alpine-openjdk11-jre' does not use a specific image digest - build may not be reproducible
[INFO] [io.quarkus.container.image.jib.deployment.JibProcessor] LogEvent [level=INFO, message=Docker config auths section defines credentials for quay.io]
[INFO] [io.quarkus.container.image.jib.deployment.JibProcessor] LogEvent [level=LIFECYCLE, message=Using credentials from Docker config (/home/jcarvaja/.docker/config.json) for quay.io/jcarvaja/getting-started:1.0.0-SNAPSHOT]
[INFO] [io.quarkus.container.image.jib.deployment.JibProcessor] The base image requires auth. Trying again for fabric8/java-alpine-openjdk11-jre...
[INFO] [io.quarkus.container.image.jib.deployment.JibProcessor] Using base image with digest: sha256:b459cc59d6c7ddc9fd52f981fc4c187f44a401f2433a1b4110810d2dd9e98a07
[INFO] [io.quarkus.container.image.jib.deployment.JibProcessor] Container entrypoint set to [java, -Djava.util.logging.manager=org.jboss.logmanager.LogManager, -jar, quarkus-run.jar]
[INFO] [io.quarkus.container.image.jib.deployment.JibProcessor] Created container image quay.io/jcarvaja/getting-started:1.0.0-SNAPSHOT (sha256:c3ec0996bed3be953257d4c0eecc80c157e3010be44640203ca4fbb25668bf3c)

[INFO] [io.quarkus.kubernetes.deployment.KubernetesDeployer] Deploying to kubernetes server: https://api.ocp48.dynamic.quarkus:6443/ in namespace: jose-test.
[INFO] [io.quarkus.kubernetes.deployment.KubernetesDeployer] Applied: Service getting-started.
[INFO] [io.quarkus.kubernetes.deployment.KubernetesDeployer] Applied: Deployment getting-started.
[INFO] [io.quarkus.deployment.QuarkusAugmentor] Quarkus augmentation completed in 14167ms

I've also tried with 2.2.1.Final, 2.2.0.Final and 999-SNAPSHOT.

Any more hints about how to reproduce this issue?

[manusa]

With Minikube and Quarkus 2.2.1.Final:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  11.130 s
[INFO] Finished at: 2021-09-01T09:42:07+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal io.quarkus.platform:quarkus-maven-plugin:2.2.1.Final:build (default) on project code-with-quarkus: Failed to build quarkus application: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
[ERROR]         [error]: Build step io.quarkus.kubernetes.deployment.KubernetesDeployer#deploy threw an exception: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://192.168.39.112:8443/api/v1/namespaces/default/services. Message: Service "code-with-quarkus" is invalid: spec.ports[0].nodePort: Invalid value: 31995: provided port is already allocated. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec.ports[0].nodePort, message=Invalid value: 31995: provided port is already allocated, reason=FieldValueInvalid, additionalProperties={})], group=null, kind=Service, name=code-with-quarkus, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Service "code-with-quarkus" is invalid: spec.ports[0].nodePort: Invalid value: 31995: provided port is already allocated, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).

$ minikube version                                  
minikube version: v1.21.0
commit: 76d74191d82c47883dc7e1319ef7cebd3e00ee11
$ kubectl version          
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:59:11Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.7", GitCommit:"132a687512d7fb058d0f5890f07d4121b3f0a2e2", GitTreeState:"clean", BuildDate:"2021-05-12T12:32:49Z", GoVersion:"go1.15.12", Compiler:"gc", Platform:"linux/amd64"}

[manusa]

Any more hints about how to reproduce this issue?

Basically the issue should be reproducible by invoking mvn clean package -Dquarkus.kubernetes.deploy=true twice (no changes to the code needed)

[manusa]

I tried to reproduce on the RH Developer Sandbox (using S2I) but I couldn't (mvn clean package -Dquarkus.kubernetes.deploy=true -Dquarkus.kubernetes.service-type=NodePort)

I think that OpenShift takes care of the allocated nodePort issue, since the problem is not even reproducible when using kubectl with the generated manifests.