Missing open ports in COS charms when running juju status

[gustavosr98]

Bug Description

$ juju status
Model  Controller    Cloud/Region            Version  SLA          Timestamp
lma    aws-microk8s  aws-microk8s/localhost  2.9.34   unsupported  18:37:06-04:00

App           Version  Status  Scale  Charm             Channel  Rev  Address         Exposed  Message
alertmanager  0.23.0   active      1  alertmanager-k8s  edge      33  10.152.183.69   no       
grafana       8.2.6    active      1  grafana-k8s       edge      45  10.152.183.205  no       
loki          2.4.1    active      1  loki-k8s          edge      45  10.152.183.210  no       
prometheus    2.33.5   active      1  prometheus-k8s    edge      75  10.152.183.254  no       

Unit             Workload  Agent  Address      Ports  Message
alertmanager/0*  active    idle   10.1.100.12         
grafana/0*       active    idle   10.1.57.73          
loki/0*          active    idle   10.1.100.13         
prometheus/0*    active    idle   10.1.100.14         

To Reproduce

On a Microk8s cloud

$ juju add-model lma
$ juju deploy ./microk8s/lma/lma_servers.yaml --trust
$ juju status

Environment

Cloud: Microk8s v1.25/stable (Remote)
Juju: 2.9.34 (Client and Controller)

COS bundle: ./microk8s/lma/lma_servers.yaml

# https://charmhub.io/topics/canonical-observability-stack/editions/lite
# https://charmhub.io/cos-lite
# channel = latest/edge (rev 5)

bundle: kubernetes
name: cos-lite
description: >
  COS Lite is a light-weight, highly-integrated, observability stack running on Kubernetes
applications:
  # I am ignoring Traefik for now
  # since some older charms do not have 
  # juju endpoint relation "ingress" implemented 
  # 
  # traefik:
  #   charm: traefik-k8s
  #   scale: 1
  #   trust: true
  #   channel: edge
  alertmanager:
    charm: alertmanager-k8s
    scale: 1
    trust: true
    channel: edge
  prometheus:
    charm: prometheus-k8s
    scale: 1
    trust: true
    channel: edge
  grafana:
    charm: grafana-k8s
    scale: 1
    trust: true
    channel: edge
  loki:
    charm: loki-k8s
    scale: 1
    trust: true
    channel: edge

relations:
# ----- Servers
# - [traefik:ingress-per-unit, prometheus:ingress]
# - [traefik:ingress-per-unit, loki:ingress]
- [ grafana:grafana-source, prometheus:grafana-source ]
- [ grafana:grafana-source, loki:grafana-source ]
- [ grafana:grafana-source, alertmanager:grafana-source ]

- [ alertmanager:alerting, prometheus:alertmanager ]
- [ alertmanager:alerting, loki:alertmanager ]

# ----- Agents on servers
# - [traefik:metrics-endpoint, prometheus:metrics-endpoint]
- [ grafana:grafana-dashboard, loki:grafana-dashboard ]
- [ grafana:grafana-dashboard, prometheus:grafana-dashboard ]
- [ grafana:grafana-dashboard, alertmanager:grafana-dashboard ]

- [ prometheus:metrics-endpoint, alertmanager:self-metrics-endpoint ]
- [ prometheus:metrics-endpoint, loki:metrics-endpoint ]

Relevant log output

n/a

Additional context

I did not use Traefik as a ingress controller in this Microk8s deployment since I wanted to expose other k8s charm services deployed on the same Microk8s which do not have the ingress relation implemented (apps model: redis-k8s, mattermost-k8s, discourse-k8s, wordpress-k8s).

When running juju config <app> juju-external-hostname=<>; juju expose <app> for any charm in apps model I would get the ingress resource created on Microk8s when using ingress addon (kubectl get ingress --all-namespaces).

However when doing the same for COS charms no ingress resource would get created.

I would think that showing open ports for COS charms would allow to be able to expose its services with Microk8s ingress addon just like the k8s charms I have on the apps model.

[sed-i]

@rbarry82 can probably explain the ins and outs of this.

BTW, K8s itself does list the ports, for example alertmanager:

        _ports = SimpleNamespace(api=9093, ha=9094)
        # ...
        self.service_patcher = KubernetesServicePatch(
            self,
            [
                (f"{self.app.name}", self._ports.api, self._ports.api),
                (f"{self.app.name}-ha", self._ports.ha, self._ports.ha),
            ],
        )

$ k get services -A
NAMESPACE         NAME                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
wlcm2             am                       ClusterIP   10.152.183.114   <none>        9093/TCP,9094/TCP        5m48s

[simskij]

@gustavosr98

I know this does not solve your issue, but I still have some comments that I think are worthwhile to post.

I did not use Traefik as a ingress controller in this Microk8s deployment

Traefik can be used in tandem with any other ingress controller on the deployed MicroK8s. It is, right now, the only ingress controller that fully supports the functionality needed by COS Lite.

other k8s charm services deployed on the same Microk8s

I would not recommend this topology for anything production or production-like. I totally get why this is tempting to keep the amount of moving parts as low as possible, but for COS Lite - or any observability solution really - to interfere as little as possible with your workloads, and more importantly: vice versa, we recommend you to always deploy it on an isolated MicroK8s cluster.

[gustavosr98]

@sed-i Thank you that is good information to manually create the ingress resources.

Still, I am guessing Juju is be ignoring this information when running juju expose. Otherwise, It should create the ingress resources like it does with mattermost-k8s, discourse-k8s and others that show the ports on Juju.

[gustavosr98]

Traefik can be used in tandem with any other ingress controller on the deployed MicroK8s. It is, right now, the only ingress controller that fully supports the functionality needed by COS Lite.

@simskij I started deploying Traefik just like the full bundle renders. However, I got Traefik charm stuck on waiting status with a message like "not obtaining a gateway ip". I assumed it was a conflict with the existing ingress controller from Microk8s ingress addon and did not dig much into it. If Traefik can co-exist with this plugin then that might be another issue I have to take a look back.

[simskij]

Traefik can be used in tandem with any other ingress controller on the deployed MicroK8s. It is, right now, the only ingress controller that fully supports the functionality needed by COS Lite.

@simskij I started deploying Traefik just like the full bundle renders. However, I got Traefik charm stuck on waiting status with a message like "not obtaining a gateway ip". I assumed it was a conflict with the existing ingress controller from Microk8s ingress addon and did not dig much into it. If Traefik can co-exist with this plugin then that might be another issue I have to take a look back.

Yeah that error message might not be the best tbh. What it means is thay you lack a source of external IPs, like MetalLB

[gustavosr98]

I would not recommend this topology for anything production or production-like. I totally get why this is tempting to keep the amount of moving parts as low as possible, but for COS Lite - or any observability solution really - to interfere as little as possible with your workloads, and more importantly: vice versa, we recommend you to always deploy it on an isolated MicroK8s cluster.

@simskij Thank you this is valuable information to take into account for other deployments. I did not considered it for two reasons. One is that I have a really small deployment intended for demo purposes. And the other reason is I tried using COS to monitor some VM charmed workloads and hit into this caveat, which seems I would need to either use just one juju controller for the whole deployment or do some extra configuration when bootstraping the N controllers so that they can discover the pods network direction.

[rbarry82]

Traefik can definitely exist with the other ingress controller, assuming that your load balancer has more than one IP in the pool (since nginx will otherwise bind to 80/443, and Traefik won't be able to).

I've never really checked into whether juju expose works with OF/sidecar charms other than vague memories of hearing that it actually doesn't work as expected, but in any case, the default nginx ingress does "normal" k8s service-based-routing, which isn't appropriate for a lot of what's in sidecars right now for various reasons.

One controller for the whole deployment works, as long as that controller is publicly routable. The other configuration actually isn't about the pod's network direction, and more due to Juju not "knowing" what external address it should use for the controller, ala juju bootstrap microk8s coscontroller --config controller-service-type=external --config controller-external-name=mydnsname --config controller-external-ips=[x.x.x.x, y.y.y.y], so it defaults to handing you a k8s-internal, non-routable DNS name (and IP, probably) which can't be resolved from outside the cluster.

What is the actual "error" from the controller when you try to consume?

[simskij]

This is currently blocked pending a native way of opening ports using Juju/OF.

[PietroPasotti]

See canonical/operator#905

[rbarry82]

This could nominally be tested on a current version of Juju by using juju run ... to invoke open-port explicitly, to see if it actually percolates through the controller to be rendered when juju status interrogates it. At a glance, it looks like it might, but it may need a follow-up patch from Juju to resolve, even once the work is done in OF.

[simskij]

@benhoyt - do you have any suggestion as to how we could accomplish this for sidecar charms?

[benhoyt]

@simskij Yes, now that canonical/operator#905 is merged (and included in ops 2.1.0+) you should be able to use Unit.open_port() in the cos-lite charms. This will tell Juju to open those ports on the K8s end. I've just grepped the codebases for each of the cos-lite charms, and none of them are using open_port yet.

[sed-i]

There are some related juju issues still open:

• https://bugs.launchpad.net/juju/+bug/2003782
• https://bugs.launchpad.net/juju/+bug/2003783

We should use open_port, but it may still not be visible in juju status.