deb packages for kubelet on pkgs.k8s.io seem to include a systemd service definition for redhat machines

[timricese]

What happened:

After updating to 1.28.2 and switching from apt.kubernetes.io to the pkgs.k8s.io repository, KUBELET_EXTRA_ARGS specified in /etc/default/kubelet (as documented: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/kubelet-integration/#the-kubelet-drop-in-file-for-systemd) no longer were being applied at runtime.

Instead, the systemd service dropin file referred to /etc/sysconfig/kubelet, which is the path used on redhat/rpm machines.

The file that should be included in the deb package is presumably:

release/cmd/kubepkg/templates/latest/deb/kubeadm/10-kubeadm.conf

Line 9 in ec317cd

EnvironmentFile=-/etc/default/kubelet

however it appears to actually be:

release/cmd/kubepkg/templates/latest/rpm/kubeadm/10-kubeadm.conf

Line 9 in ec317cd

EnvironmentFile=-/etc/sysconfig/kubelet

What you expected to happen:

the same EnvironmentFile as documented and previously used in packages from apt.kubernetes.io should still be used.

How to reproduce it (as minimally and precisely as possible):

apt install kubelet=1.28.2-1.1
systemctl cat kubelet.service

Anything else we need to know?:

https://kubernetes.slack.com/archives/C09NXKJKA/p1694790202990129

Environment:

• Cloud provider or hardware configuration: bare metal
• OS (e.g: cat /etc/os-release): Ubuntu 22.04
• Kernel (e.g. uname -a): -
• Others: kubelet==1.28.2-1.1

[xmudrii]

This also caused the issue described in #3219 (comment)
I think that we should look into this as soon as possible. We can't alter existing releases, but we can fix this for future releases. This is mostly like caused by reusing same spec files for both RPM and Debian packages.

/priority critical-urgent
/assign

[xmudrii]

I think this can be easily fixed in the spec file with some sed magic ✨ I'll propose a PR for that on Monday.

[afbjorklund]

Maybe it needs a test ?

• Create test jobs to validate that debs/rpms can be installed #821

[xmudrii]

@afbjorklund We have an issue to track creating tests for the new implementation: #3168

[yogeek]

We just stumbled upon this issue after migrating to the new repo [pkgs.k8s.io](https://kubernetes.io/blog/2023/08/15/pkgs-k8s-io-introduction/) with the following package : 1.26.9-1-1

here are the steps we followed on ubuntu 22.04 :

export KUBEADM_VERSION="1.26.9-1.1"
export KUBECTL_VERSION="1.26.9-1.1"
export MINOR_K8S_VERSION=$(echo $KUBECTL_VERSION | grep -o -E "^[0-9]\\.[0-9]+")

apt-get update
apt-get -o Dpkg::Options::="--force-confnew" -y install  curl unzip nfs-common golang-cfssl gpg

# Add new k8s apt repository
curl -fsSL https://pkgs.k8s.io/core:/stable:/v${MINOR_K8S_VERSION}/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

# Add kube's apt repository
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v${MINOR_K8S_VERSION}/deb/ /" | tee /etc/apt/sources.list.d/kubernetes.list

# update repos
apt-get update

# Install packages
apt-get install -y --allow-unauthenticated  "kubeadm=${KUBEADM_VERSION}" "kubectl=${KUBECTL_VERSION}"

and when I check the kubelet environment file, it is the wrong one : ("sysconfig" instead of "default")

cat /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf | grep EnvironmentFile=
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
EnvironmentFile=-/etc/sysconfig/kubelet

We also tried with the latest patched version for 1.26 (1.26.15-1-1) but same result

I am not familiar with the k8s release process so I am not sure how can we benefit from the fix in #3279 to install the 1.26 packages please ?

[neolit123]

We also tried with the latest patched version for 1.26 (1.26.15-1-1) but same result

@xmudrii IIUC, #3279 does not need a backport and it would apply to all k8s packages that we release with the new repo process, is that correct?

[xmudrii]

That's correct, no backport is needed. I'm also not able to reproduce the issue for kubeadm releases created after the fix was merged. This what I'm getting and I believe this is correct:

root@efb6b74d04b8:/# cat /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf | grep EnvironmentFile=
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
EnvironmentFile=-/etc/default/kubelet

[jorhett]

Yes, however the package itself places the defaults file in the RedHat location:

$ dpkg -L kubelet |grep /kubelet
/etc/sysconfig/kubelet

So this was halfway fixed, now the package contents need to be fixed.

[xmudrii]

@jorhett This issue should be fixed by #3625