Add InhibitionClusterWithoutWorkerNodes for CAPA #1397
Conversation
hervenicol
force-pushed
the
capa-inhibit-noworker-clusters
branch
from
08fe221 to
3a24c8c
Compare
| annotations: | ||
| description: '{{`Cluster ({{ $labels.cluster_id }}) has no worker nodes.`}}' | ||
| expr: |- | ||
| label_replace( |
There was a problem hiding this comment.
Why is this for capa only? otherwise it looks good
There was a problem hiding this comment.
I would think that with capi_machinepool_spec_replicas and capi_machinedeployment_spec_replicas we should be good?
There was a problem hiding this comment.
Because the metrics is only for CAPA.
Ref: https://gigantic.slack.com/archives/C02HLSDH3DZ/p1729502169569249?thread_ts=1729498455.102279&cid=C02HLSDH3DZ
| "(.*)" | ||
| ) == 1 | ||
| unless on (cluster_id) ( | ||
| sum(capi_machinepool_spec_replicas{} > 0) by (cluster_id) |
There was a problem hiding this comment.
Do we accept 1 worker nodes only?
There was a problem hiding this comment.
Prometheus-agent/Alloy could run on a 1-node WC, so that's potentially ok.
There was a problem hiding this comment.
Are we sure we want to use capi_machinepool_spec_replicas? That's the number of replicas defined in the MachinePool spec, but that doesn't necessarily represent the current number of nodes. For the inhibition, I thought we would prefer to use the current number of replicas. There are other metrics like
capi_machinepool_status_replicas: Replicas is the most recently observed number of replicas.capi_machinepool_status_replicas_ready: The number of ready replicas for this MachinePool. A machine is considered ready when the node has been created and is "Ready".capi_machinepool_status_replicas_available: The number of available replicas (ready for at least minReadySeconds) for this MachinePool.
Do you think it would make sense to use one of those instead?
There was a problem hiding this comment.
Current inhibition works when the cluster has been purposely scaled down.
If the cluster should have nodes but none is ready/available, I think we should get a page.
I don't think current state of CAPI monitoring manages it, so I'd rather have a "prometheus-agent down" alert than no alert in this case.
It seems to me that there's quite a gap between vintage AWS and CAPA alerts, but I don't think it's Atlas responsibility to fix it. So I went the quickest way to solving my actual issue 😅
There was a problem hiding this comment.
ok, I thought you wanted the inhibition to avoid paging when the cluster was having other issues i.e. no ready nodes, meaning there was nothing wrong with your component.
There was a problem hiding this comment.
That would be ideal. But my first expectation is to not get paged when a cluster has no issues, for now 🤣
Towards: https://github.com/giantswarm/giantswarm/issues/31390
This PR adds an inhibition for clusters that have no worker nodes.
Checklist
oncall-kaas-cloudGitHub group).