Can't remove nodeSelector variable for packaged Helm charts

[dbth21]

After recent changes, like rancher/rke2-charts#133 packaged components have default nodeSelector which can't be removed.

I tried to remove it using default RKE2 approach

• HelmChartConfig:

nodeSelector: {}

• and even using HelmChart

  set:
    nodeSelector: {}

Both of these methods don't work, all objects rendered by Helm charts contain

  nodeSelector:
    kubernetes.io/os: linux

Environmental Info:
RKE2 Version:
rke2 version v1.21.4+rke2r2 (d5746a3)
go version go1.16.6b7

Node(s) CPU architecture, OS, and Version:
Linux k8s-cp-1 5.4.0-81-generic #91-Ubuntu SMP Thu Jul 15 19:09:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Expected behavior:
Variable nodeSelector is absent or empty when nodeSelector: {} is set in Helm variables.

[brandond]

Why do you want to remove the Linux nodeselector? These charts only work on Linux, and with the addition of Windows agent support it is important to ensure that these pods don't attempt to schedule on to the wrong OS.

[dbth21]

Hi @brandond
Thank you for your fast reply!
In my particular scenario cluster autoscaler can't scale up a node because coredns (and ingress-nginx, by the way) in RKE2 v.21 have this linux related label. RKE2 1.20 works fine with the same version and same configuration of autoscaler.
However, I think use-cases are not very important, because if we have configuration parameter for nodeSelector, this parameter should allow to define an empty value, like tolerations, for example.
tolerations: [] works fine, removes all tolerations. But nodeSelector: {} doesn't remove the linux one.

[brandond]

In my particular scenario cluster autoscaler can't scale up a node because coredns (and ingress-nginx, by the way) in RKE2 v.21 have this linux related label.

Can you explain this bit? You're saying that the cluster autoscaler won't scale up your node count because some of the pods have nodeselectors?

[dbth21]

Yes, cluster autoscaler for Hetzner cloud triggers scale up event in case pending pod has node selector like:

nodeSelector:
  hcloud/node-group: <group_name>

or doesn't have node selectors at all.

[dbth21]

@brandond here is a detailed description

1. Cluster autoscaler

- command:
  - ./cluster-autoscaler
  - --cloud-provider=hetzner
  - --expander=most-pods
  - --v=1
  - --nodes=1:5:CX11:NBG1:ingress
  - --nodes=0:5:CX11:NBG1:svctest

2. CoreDNS with helm config hodeSelectors: {}

$ kubectl get deploy  -n kube-system rke2-coredns-rke2-coredns -o yaml | grep -A2 nodeSelector
      nodeSelector:
        kubernetes.io/os: linux
      priorityClassName: system-cluster-critical

3. Cluster autoscaler logs:

I0907 01:57:29.424623       1 klogx.go:86] Pod kube-system/rke2-coredns-rke2-coredns-66c748d7f9-tdvmq is unschedulable
I0907 01:57:29.424664       1 klogx.go:86] Pod kube-system/rke2-coredns-rke2-coredns-autoscaler-5c54dd9774-cmv6t is unschedulable
I0907 01:57:29.424676       1 klogx.go:86] Pod kube-system/helm-install-rke2-ingress-nginx-w8999 is unschedulable
I0907 01:57:29.424686       1 klogx.go:86] Pod kube-system/helm-install-rke2-metrics-server-dlk7t is unschedulable
I0907 01:57:30.886846       1 scale_up.go:453] No expansion options

4. Adding Hetzner's specific label:

$ kubectl get deploy  -n kube-system rke2-coredns-rke2-coredns -o yaml | grep -A2 nodeSelector
      nodeSelector:
        hcloud/node-group: svctest
        kubernetes.io/os: linux

5. Cluster autoscaler logs, still the same:

I0907 02:04:49.814822       1 hetzner_node_group.go:421] Set node group svctest size from 0 to 0, expected delta 0
I0907 02:04:51.754527       1 klogx.go:86] Pod kube-system/rke2-coredns-rke2-coredns-5c4fc69d6c-2hhwv is unschedulable
I0907 02:04:51.754572       1 klogx.go:86] Pod kube-system/rke2-coredns-rke2-coredns-autoscaler-5c54dd9774-cmv6t is unschedulable
I0907 02:04:51.754583       1 klogx.go:86] Pod kube-system/helm-install-rke2-metrics-server-dlk7t is unschedulable
I0907 02:04:51.754592       1 klogx.go:86] Pod kube-system/helm-install-rke2-ingress-nginx-w8999 is unschedulable
I0907 02:04:52.610527       1 scale_up.go:453] No expansion options

6. Removing nodeSelectors manually, using kubectl edit deploy -n kube-system rke2-coredns-rke2-coredns

$ kubectl edit deploy -n kube-system rke2-coredns-rke2-coredns
deployment.apps/rke2-coredns-rke2-coredns edited
$  kubectl get deploy  -n kube-system rke2-coredns-rke2-coredns -o yaml | grep -A2 nodeSelector
$

7.Cluster autoscaler logs, scale up has been triggered

I0907 02:08:19.677669       1 klogx.go:86] Pod kube-system/rke2-coredns-rke2-coredns-6d885589b9-zch58 is unschedulable
I0907 02:08:19.677906       1 klogx.go:86] Pod kube-system/rke2-coredns-rke2-coredns-autoscaler-5c54dd9774-cmv6t is unschedulable
I0907 02:08:19.677936       1 klogx.go:86] Pod kube-system/helm-install-rke2-ingress-nginx-w8999 is unschedulable
I0907 02:08:19.677976       1 klogx.go:86] Pod kube-system/helm-install-rke2-metrics-server-dlk7t is unschedulable
I0907 02:08:20.571756       1 scale_up.go:468] Best option to resize: svctest
I0907 02:08:20.571860       1 scale_up.go:472] Estimated 1 nodes needed in svctest
I0907 02:08:20.571942       1 scale_up.go:586] Final scale-up plan: [{svctest 0->1 (max: 5)}]
I0907 02:08:20.571986       1 scale_up.go:675] Scale-up: setting group svctest size to 1

[brandond]

That sounds like it's worth opening an issue with the Hetzner cluster autoscaler to be able to ignore nodeselectors?

[brandond]

Based on the code here: https://github.com/kubernetes/autoscaler/blob/9d54f7b782fe9e1197ff20ed9e71e3af81ca1ae9/cluster-autoscaler/utils/labels/labels.go#L62

It appears that kubernetes.io nodeselector labels should not prevent the autoscaler from scaling the cluster?

[stale]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 180 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.