Switch to using kubelet config file for all supported flags

[brandond]

Proposed Changes

Switch to using kubelet config file for all supported flags

Types of Changes

enhancement / tech debt

Verification

See linked issue

Testing

Linked Issues

• Move to using the kubelet config file for config instead of cmd line args #2116

User-Facing Change

Further Comments

Currently using a hacky string replace when marshaling the config file to work around an upstream issue:

• kubelet config file, resolverConfig option not used kubernetes/kubernetes#79421

[codecov]

Codecov Report

Attention: Patch coverage is 66.48936% with 63 lines in your changes missing coverage. Please review.

Project coverage is 43.47%. Comparing base (b55aaeb) to head (0d6cc13).
Report is 3 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/daemons/agent/agent.go	71.05%	31 Missing and 13 partials ⚠️
pkg/daemons/agent/agent_linux.go	48.27%	13 Missing and 2 partials ⚠️
pkg/agent/config/config.go	40.00%	2 Missing and 1 partial ⚠️
pkg/etcd/etcd.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10433      +/-   ##
==========================================
- Coverage   47.83%   43.47%   -4.37%     
==========================================
  Files         181      181              
  Lines       18794    18906     +112     
==========================================
- Hits         8990     8219     -771     
- Misses       8452     9479    +1027     
+ Partials     1352     1208     -144     

Flag	Coverage Δ
e2etests	35.81% <66.48%> (-7.21%)	⬇️
inttests	18.62% <1.59%> (-16.20%)	⬇️
unittests	14.18% <0.00%> (-0.09%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dereknola]

LGTM pending RKE2 cgroup checks

[brandond]

I'm going to hold on this until the August release cycle, as I'd like more lead time on testing it.

[brandond]

@dereknola I have confirmed this works on Windows, using RKE2: rancher/rke2#6909

It looks like RKE2 adds some extra CLI flags to the kubelet, so those will need to be migrated over to config as well once this is merged: https://github.com/rancher/rke2/blob/b696280a35f87bb260f26b256d7be839f1945535/pkg/pebinaryexecutor/pebinary.go#L146-L168

root@dev02:~# kubectl get node -o wide
NAME    STATUS   ROLES                       AGE     VERSION          INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                  KERNEL-VERSION     CONTAINER-RUNTIME
dev02   Ready    control-plane,etcd,master   3h42m   v1.31.1+rke2r1   10.0.1.184    <none>        Ubuntu 24.04 LTS                          6.8.0-45-generic   containerd://1.7.21-k3s2
win02   Ready    <none>                      3h31m   v1.31.1          10.0.1.224    <none>        Windows Server 2022 Standard Evaluation   10.0.20348.2113    containerd://1.7.21-k3s2

Running kubelet --alsologtostderr=false --cloud-provider=external --config-dir=C:\var\lib\rancher\rke2\agent\etc\kubelet.conf.d --feature-gates=CloudDualStackNodeIPs=true --hostname-override=win02 --kubeconfig=C:\var\lib\rancher\rke2\agent\kubelet.kubeconfig --log-file=\var\lib\rancher\rke2\agent\logs\kubelet.log --log-file-max-size=50 --logtostderr=false --node-ip=10.0.1.224 --node-labels= --stderrthreshold=FATAL
Running RKE2 kubelet [--cgroups-per-qos=false --enforce-node-allocatable= --file-check-frequency=5s --hairpin-mode=promiscuous-bridge --resolv-conf= --sync-frequency=30s --cloud-provider=external --config-dir=C:\var\lib\rancher\rke2\agent\etc\kubelet.conf.d --feature-gates=CloudDualStackNodeIPs=true --hostname-override=win02 --kubeconfig=C:\var\lib\rancher\rke2\agent\kubelet.kubeconfig --node-ip=10.0.1.224 --node-labels=

PS C:\Users\Administrator> c:\usr\local\bin\rke2.exe --version
rke2.exe version v1.31.1+dev.045680fc (045680fc54b2e82cbb5491c9a916655535cd8fb7)
go version go1.22.6

PS C:\Users\Administrator> dir C:\var\lib\rancher\rke2\agent\etc\kubelet.conf.d\


    Directory: C:\var\lib\rancher\rke2\agent\etc\kubelet.conf.d


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----         10/5/2024  12:51 AM           1516 00-rke2-defaults.conf


PS C:\Users\Administrator> cat C:\var\lib\rancher\rke2\agent\etc\kubelet.conf.d\00-rke2-defaults.conf
address: 0.0.0.0
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: C:\var\lib\rancher\rke2\agent\client-ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: cgroupfs
clusterDNS:
- 10.43.0.10
clusterDomain: cluster.local
containerRuntimeEndpoint: npipe:////./pipe/containerd-containerd
cpuManagerReconcilePeriod: 10s
evictionHard:
  imagefs.available: 5%
  nodefs.available: 5%
evictionMinimumReclaim:
  imagefs.available: 10%
  nodefs.available: 10%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: false
fileCheckFrequency: 20s
healthzBindAddress: 127.0.0.1
httpCheckFrequency: 20s
imageMaximumGCAge: 0s
imageMinimumGCAge: 2m0s
kind: KubeletConfiguration
logging:
  flushFrequency: 5s
  format: text
  options:
    json:
      infoBufferSize: "0"
    text:
      infoBufferSize: "0"
  verbosity: 0
memorySwap: {}
nodeStatusReportFrequency: 5m0s
nodeStatusUpdateFrequency: 10s
resolverConfig: C:\var\lib\rancher\rke2\agent\etc\resolv.conf
runtimeRequestTimeout: 2m0s
serializeImagePulls: false
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: C:\var\lib\rancher\rke2\agent\pod-manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
tlsCertFile: C:\var\lib\rancher\rke2\agent\serving-kubelet.crt
tlsPrivateKeyFile: C:\var\lib\rancher\rke2\agent\serving-kubelet.key
volumeStatsAggPeriod: 1m0s

[brandond]

moving to WIP until closer to 1.32.

[caroline-suse-rancher]

Hey @brandond, are we still planning to do this for 1.32?

[brandond]

I am watching what upstream does with the drop-in config dir feature in kubernetes/enhancements#3983. If it is on by default for 1.32 then I think we can move forward.

[dereknola]

Does this need to be changed now that 1.32 is here?

[brandond]

dropped.

[brandond]

ugh nope, still required, I put it back.

root@systemd-node-1:/# grep resolv /var/lib/rancher/k3s/agent/etc/kubelet.conf.d/00-k3s-defaults.conf
resolvConf: /var/lib/rancher/k3s/agent/etc/resolv.conf

root@systemd-node-1:/# kubectl get --raw /api/v1/nodes/systemd-node-1/proxy/configz 2>/dev/null | jq . | grep resolv
    "resolvConf": "/etc/resolv.conf",

root@systemd-node-1:/# echo -e "apiVersion: kubelet.config.k8s.io/v1beta1\nkind: KubeletConfiguration\nresolverConfig: /var/lib/rancher/k3s/agent/etc/resolv.conf" > /var/lib/rancher/k3s/agent/etc/kubelet.conf.d/99-resolv.conf

root@systemd-node-1:/# systemctl restart k3s

root@systemd-node-1:/# grep -rF resolv /var/lib/rancher/k3s/agent/etc/kubelet.conf.d/
/var/lib/rancher/k3s/agent/etc/kubelet.conf.d/00-k3s-defaults.conf:resolvConf: /var/lib/rancher/k3s/agent/etc/resolv.conf
/var/lib/rancher/k3s/agent/etc/kubelet.conf.d/99-resolv.conf:resolverConfig: /var/lib/rancher/k3s/agent/etc/resolv.conf

root@systemd-node-1:/# kubectl get --raw /api/v1/nodes/systemd-node-1/proxy/configz 2>/dev/null | jq . | grep resolv
    "resolvConf": "/var/lib/rancher/k3s/agent/etc/resolv.conf",