kube-proxy fails when in proxy-mode=nftables mode

[vaskozl]

The nftables kube-proxy backend is planned to one day be the default backend for kube-proxy. Support was added in 1.29

Uppon adding --feature-gates=NFTablesProxyMode=true --proxy-mode=nftables to iptables it to add any rules:

│     /dev/stdin:892:87-100: Error: Could not process rule: No such file or directory
│     add rule ip kube-proxy external-3BXM2ZZ4-haproxy/external-kubernetes-ingress/tcp/http fib saddr type local jump mark-for-masquerade comment "masquerade local traffic" 
│                                                                                           ^^^^^^^^^^^^^^ 
│     /dev/stdin:893:87-100: Error: Could not process rule: No such file or directory  

I believe this is due to these unset kernel variables:

CONFIG_NFT_FIB_IPV4=m
CONFIG_NFT_FIB_IPV6=m

N.B. nftables masquerade was recently added to `flannel.

You can read more about why and what in the KEP: https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/3866-nftables-proxy/README.md

[frezbo]

I think we can make this the default for 1.8

[smira]

Verified to work with the updated kernel.

I0507 11:39:31.916310       1 server_linux.go:304] "Using nftables Proxier"

e2e status passed:7 skipped:7194 

[vaskozl]

I can confirm that the nftables backend now works fine with 1.7.2