Openshift extension does not create a service for a management interface

[fedinskiy]

Describe the bug

I have an application, deployed on Openshift via Quarkus-openshift extension, which uses separate management interface[1].

After an update to Quarkus 3.2.0, the interface can not be accessed anymore, since this port is not exposed as a part of a service.
[1] https://quarkus.io/guides/management-interface-reference

Expected behavior

The port is exposed as a part of a service, same as for 3.1.0.

Generated file target/kubernetes/openshift.yml should contains service description like this:

---
apiVersion: v1
kind: Service
metadata:
  annotations:
<...>
  labels:
    app.kubernetes.io/name: openshift-quickstart
<...>
  name: openshift-quickstart
spec:
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: 8080
    - name: https
      port: 443
      protocol: TCP
      targetPort: 8443
    - name: management
      port: 9000
      protocol: TCP
      targetPort: 9000
  selector:
    app.kubernetes.io/name: openshift-quickstart
    app.kubernetes.io/version: 1.0.0-SNAPSHOT
  type: ClusterIP
---

and oc describe svc/openshift-quickstart should return this:

Name:              openshift-quickstart
Namespace:         fd-test
Labels:            app.kubernetes.io/managed-by=quarkus<...>
Annotations:  <...>
Selector:          app.kubernetes.io/name=openshift-quickstart,app.kubernetes.io/version=1.0.0-SNAPSHOT
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                $EDITED
IPs:               $EDITED
Port:              http  80/TCP
TargetPort:        8080/TCP
Endpoints:         $EDITED1:8080
Port:              https  443/TCP
TargetPort:        8443/TCP
Endpoints:         $EDITED:8443
Port:              management  9000/TCP
TargetPort:        9000/TCP
Endpoints:         $EDITED:9000
Session Affinity:  None
Events:            <none>

Actual behavior

The generated file looks like this:

---
apiVersion: v1
kind: Service
metadata:
  annotations:
<...>
  labels:
    app.kubernetes.io/name: openshift-quickstart
<...>
  name: openshift-quickstart
spec:
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: 8080
  selector:
    app.kubernetes.io/name: openshift-quickstart
    app.kubernetes.io/version: 1.0.0-SNAPSHOT
  type: ClusterIP

and the deployed service like that:

Name:              openshift-quickstart
Namespace:         fd-test
Labels:            app.kubernetes.io/managed-by=quarkus
<...>
Annotations:    <...>
Selector:          app.kubernetes.io/name=openshift-quickstart,app.kubernetes.io/version=1.0.0-SNAPSHOT
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                $EDITED
IPs:               $EDITED
Port:              http  80/TCP
TargetPort:        8080/TCP
Endpoints:         $EDITED:8080
Session Affinity:  None
Events:            <none>

How to Reproduce?

1. [email protected]:fedinskiy/reproducer.git -b openshift-extension-management
2. mvn clean install -Dquarkus.kubernetes.deploy=true -Dquarkus.openshift.route.expose=true -Dquarkus.kubernetes-client.trust-certs=true -Dquarkus.platform.version=3.2.0.Final

Run mvn clean install -Dquarkus.kubernetes.deploy=true -Dquarkus.openshift.route.expose=true -Dquarkus.kubernetes-client.trust-certs=true -Dquarkus.platform.version=3.1.0.Final for expected result.

Output of uname -a or ver

6.3.8-200.fc38.x86_64

Output of java -version

17.0.5, vendor: GraalVM Community

GraalVM version (if different from Java)

No response

Quarkus version or git rev

3.2.0.Final

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.8.7 (b89d5959fcde851dcb1c8946a785a163f14e1e29)

Additional information

No response

[quarkus-bot]

/cc @Sgitario (kubernetes), @geoand (kubernetes,openshift), @iocanel (kubernetes,openshift)

[Sgitario]

This is expected from Quarkus 3.2 (see #33694).
This change is also noted in the migration guide for Quarkus 3.2.
To create the service for the management interface, you would need to use:

quarkus.kubernetes.ingress.expose=true
quarkus.kubernetes.ingress.target-port=management

[fedinskiy]

@Sgitario this solution doesn't solve the problem:

1. [email protected]:fedinskiy/reproducer.git -b openshift-extension-management
2. Add the recommended properties:

printf "quarkus.openshift.ingress.expose=true
quarkus.openshift.ingress.target-port=management
quarkus.kubernetes.ingress.expose=true
quarkus.kubernetes.ingress.target-port=management
" >>src/main/resources/application.properties

(using only quarkus.openshift or quarkus.kubernetes didn't help either)
3. Deploy the app: mvn clean install -Dquarkus.kubernetes.deploy=true -Dquarkus.openshift.route.expose=true -Dquarkus.kubernetes-client.trust-certs=true -Dquarkus.platform.version=3.2.0.Final
4. Check the service:

$ oc get services
NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
openshift-quickstart   ClusterIP   172.30.196.250   <none>        80/TCP    43s

If 3.1.0.Final is used, the service looks like this:

$ oc get services
NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                   AGE
openshift-quickstart   ClusterIP   172.30.235.117   <none>        9000/TCP,80/TCP,443/TCP   11m

[Sgitario]

@Sgitario this solution doesn't solve the problem:

1. [email protected]:fedinskiy/reproducer.git -b openshift-extension-management
2. Add the recommended properties:

printf "quarkus.openshift.ingress.expose=true
quarkus.openshift.ingress.target-port=management
quarkus.kubernetes.ingress.expose=true
quarkus.kubernetes.ingress.target-port=management
" >>src/main/resources/application.properties

(using only quarkus.openshift or quarkus.kubernetes didn't help either) 3. Deploy the app: mvn clean install -Dquarkus.kubernetes.deploy=true -Dquarkus.openshift.route.expose=true -Dquarkus.kubernetes-client.trust-certs=true -Dquarkus.platform.version=3.2.0.Final 4. Check the service:

$ oc get services
NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
openshift-quickstart   ClusterIP   172.30.196.250   <none>        80/TCP    43s

If 3.1.0.Final is used, the service looks like this:

$ oc get services
NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                   AGE
openshift-quickstart   ClusterIP   172.30.235.117   <none>        9000/TCP,80/TCP,443/TCP   11m

For OpenShift, you need to use the route-specific properties:

quarkus.openshift.route.expose=true
quarkus.openshift.route.target-port=management

[fedinskiy]

@Sgitario after applying quarkus.openshift.route.target-port, it turns out, that there is no route to the main interface, which is quite unexpected (and it is hard to imagine a use-case for this configuration).

Is there a way to have management interface exposed as a service (so it can be accessed from inside the cluster. eg by other microservices) and main interface exposed through route and service (so it can be accessed by users)?

[Sgitario]

@Sgitario after applying quarkus.openshift.route.target-port, it turns out, that there is no route to the main interface, which is quite unexpected (and it is hard to imagine a use-case for this configuration).

Is there a way to have management interface exposed as a service (so it can be accessed from inside the cluster. eg by other microservices) and main interface exposed through route and service (so it can be accessed by users)?

We removed this use case on purpose since management endpoints should be the more protected the better.
Can you elaborate on why you think this use case might be useful?
FYI @cescoffier

[fedinskiy]

The first thing, that comes in mind is a dashboard, that shows the current status of several services and deployed as a service itself. It needs to check and collect statuses of services which are deployed in the same cluster. Some of these services can be user-facing ones, so they need to expose the HTTP/HTTPS port to the outside world.

I may be wrong, but it seems, that exposing management port inside the cluster is more safe, that exposing it outside the cluster.

[Sgitario]

The first thing, that comes in mind is a dashboard, that shows the current status of several services and deployed as a service itself. It needs to check and collect statuses of services which are deployed in the same cluster. Some of these services can be user-facing ones, so they need to expose the HTTP/HTTPS port to the outside world.

This is what you got with Kubernetes/OpenShift by default. Users should not reinvent the wheel. However, I want to know from @cescoffier since he was who requested removing the generated Service for the Management port.

Note that if we want to expose the Management port for a service only, we might need to add a new property for doing this.

[fedinskiy]

I am fine with management port being exposed through route, as it happens now (although it looks less secure, than in 3.1.0) but in that case we also need an option to expose HTTP(S) alongside it.

[cescoffier]

I'm slightly worried.

First, it would need an explicit (defaulting to disable) property to expose the management interface as service. Definitely not as a route.

However, due to the nature of the management endpoint, I'm worried about multi replicas deployment. So, it would need a new service object with a unique name per pod to avoid the load balancing (note that the pod may not be created when you have a scaler able to handle scale to 0).

[Sgitario]

I'm slightly worried.

First, it would need an explicit (defaulting to disable) property to expose the management interface as service. Definitely not as a route.

However, due to the nature of the management endpoint, I'm worried about multi replicas deployment. So, it would need a new service object with a unique name per pod to avoid the load balancing (note that the pod may not be created when you have a scaler able to handle scale to 0).

I propose to add a new option within the ports config to enable creating an extra Service resource:

quarkus.kubernetes.ports.management.add-service=true

This would create a new service called <app name>-management.

Wdyt? Adding @iocanel to the discussion and reopening.

[cescoffier]

It won't work when you have multiple replicas, as you will have the round-robin.
The management interface is not intended to be used using a round robin

[fedinskiy]

@cescoffier @Sgitario correct me, if I am wrong, but current configuration doesn't allow to use tools like Prometheus, which read from q/metrics, right?

[cescoffier]

It does, but you need to use the pod ip, not the service one.

BTW, there are no service on purpose.

[cescoffier]

Close as won't fix. Works as expected.