[release-1.28] Remove GA feature-gates

[brandond]

Backports for:

• k3s sets KubeletCredentialProviders feature gate which is is GA and removed in 1.28 #8941
• K3s sets JobTrackingWithFinalizers feature gate which is is GA and will be removed in a future release #8969

[VestigeJ]

##Environment Details
Reproduced using VERSION=v1.28.5+k3s1
Validated using COMMIT=6224ea62af78b44db3f2f881e55590c61463dbcd
Infrastructure

• Cloud

Node(s) CPU architecture, OS, and version:

Linux 5.14.21-150500.53-default x86_64 GNU/Linux
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP5"

Cluster Configuration:

NAME               STATUS   ROLES                       AGE   VERSION
ip-21-31-13-12     Ready    control-plane,etcd,master   21m   v1.28.5+k3s-6224ea62

Config.yaml:

write-kubeconfig-mode: 644
debug: true
cluster-init: true
token: YOUR_TOKEN_HERE

Reproduction

$ curl https://get.k3s.io --output install-"k3s".sh
$ sudo chmod +x install-"k3s".sh
$ sudo groupadd --system etcd && sudo useradd -s /sbin/nologin --system -g etcd etcd
$ sudo modprobe ip_vs_rr
$ sudo modprobe ip_vs_wrr
$ sudo modprobe ip_vs_sh
$ sudo printf "on_oovm.panic_on_oom=0 \nvm.overcommit_memory=1 \nkernel.panic=10 \nkernel.panic_ps=1 \nkernel.panic_on_oops=1 \n" > ~/90-kubelet.conf
$ sudo cp 90-kubelet.conf /etc/sysctl.d/
$ sudo systemctl restart systemd-sysctl
$ sudo INSTALL_K3S_VERSION=v1.28.5+k3s1 INSTALL_K3S_EXEC=server ./install-k3s.sh
$ set_kubefig
$ sudo journalctl -u k3s | grep -i "jobtrackingwithfinalizers"
$ sudo journalctl -u k3s | grep -i "feature-gates"
$ sudo journalctl -u k3s | grep -i "kubeletcredentialproviders"
$ sudo journalctl --rotate 
$ sudo journalctl --vacuum-time=1s
$ sudo mkdir -p /var/lib/rancher/credentialprovider/bin/
$ wget https://github.com/rancher/wharfie/releases/download/v0.6.5/config-amd64.yaml
$ wget https://github.com/rancher/wharfie/releases/download/v0.6.5/ecr-credential-provider-amd64
$ sudo cp config-amd64.yaml /var/lib/rancher/credentialprovider/config.yaml
$ chmod +x ecr-credential-provider-amd64
$ sudo cp ecr-credential-provider-amd64 /var/lib/rancher/credentialprovider/bin/
$ sudo INSTALL_K3S_COMMIT=6224ea62af78b44db3f2f881e55590c61463dbcd INSTALL_K3S_EXEC=server ./install-k3s.sh
$ sudo journalctl -u k3s | grep -i "kubeletcredentialproviders"
$ sudo journalctl -u k3s | grep -i "feature-gates"
$ sudo journalctl -u k3s | grep -i "jobtrackingwithfinalizers"

Results:

$ sudo journalctl -u k3s | grep -i "feature-gates"

--feature-gates=JobTrackingWithFinalizers=true
.
.
--feature-gates=CloudDualStackNodeIPs=true
.
.

Validated using COMMIT_ID=6224ea62af78b44db3f2f881e55590c61463dbcd

$ sudo journalctl -u k3s | grep -i "feature-gates"

 --feature-gates=CloudDualStackNodeIPs=true................

$ sudo journalctl -u k3s | grep -i "credentialprovider"

 level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --feature-gates=CloudDualStackNodeIPs=true --healthz-bind-address=127.0.0.1 --hostname-override=ip-21-31-13-12 --image-credential-provider-bin-dir=/var/lib/rancher/credentialprovider/bin --image-credential-provider-config=/var/lib/rancher/credentialprovider/config.yaml --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --node-ip=21.31.13.12 --node-labels= --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --read-only-port=0 --resolv-conf=/etc/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key"