Fix access to hostNetwork port on NodeIP when egress-selector-mode=agent

[PaulSD]

With egress-selector-mode: agent (the default), and egress.X.io/cluster: true node labels (also a default), if any Pods are running with hostNetwork: true (and therefore use the Node IP), then the API Server will attempt to use agent tunnels to communicate with those Pods, but tunnel authorization will fail, resulting in connection failures.

This commit fixes the issue by adjusting the API Server such that the egress.X.io/cluster: true node labels are ignored with egress-selector-mode: agent. This ensures that agent tunnels are only used when connecting to kubelet ports on Node IPs, and will not be used when connecting to hostNetwork Pod ports on Node IPs.

This commit fixes the issue by removing the egress.X.io/cluster: true node labels unless egress-selector-mode is cluster or pod. When egress-selector-mode is agent, this ensures that agent tunnels are only used when connecting to kubelet ports on Node IPs, and will not be used when connecting to hostNetwork Pod ports on Node IPs.

Linked Issue

• apiserver egress proxy attempts to use tunnel for connections to hostnetwork pods in agent mode #6830

More details

This is loosely related to the following issues: #5637 rancher/rke2#3016

When deciding whether to tunnel connections, the server (prior to this commit):

• Always skips tunneling for connections to the local node:

  k3s/pkg/daemons/control/tunnel.go

  Lines 256 to 257 in b411864

  	if nodeName == t.config.ServerNodeName {
  	useTunnel = false

• Always tunnels to other nodes accessed using the node name:

  k3s/pkg/daemons/control/tunnel.go

  Lines 243 to 247 in b411864

  	} else {
  	// Destination is a node by name, it is safe to use the tunnel.
  	nodeName = host
  	toKubelet = true
  	useTunnel = true

• Checks the destination against a list of CIDRs:

  k3s/pkg/daemons/control/tunnel.go

  Lines 227 to 230 in b411864

  	if ip := net.ParseIP(host); ip != nil {
  	// Destination is an IP address, which could be either a pod, or node by IP.
  	// We can only use the tunnel for egress to pods if the agent supports it.
  	if nets, err := t.cidrs.ContainingNetworks(ip); err == nil && len(nets) > 0 {

  • This list is not populated if egress-selector-mode is disabled:

    k3s/pkg/daemons/control/tunnel.go

    Lines 111 to 112 in b411864

    	if t.config.EgressSelectorMode == config.EgressSelectorModeDisabled {
    	return

  • If egress-selector-mode is agent then this list is populated with Node IPs:

    k3s/pkg/daemons/control/tunnel.go

    Line 130 in b411864

    func (t *TunnelServer) onChangeNode(nodeName string, node *v1.Node) (*v1.Node, error) {

  • If egress-selector-mode is pod or cluster then this list is populated with both Node and Pod IPs:

    k3s/pkg/daemons/control/tunnel.go

    Line 155 in b411864

    func (t *TunnelServer) onChangePod(podName string, pod *v1.Pod) (*v1.Pod, error) {

• Tunnels if the destination matches the CIDR list and either:
  • The connection is to the kubelet port:

    k3s/pkg/daemons/control/tunnel.go

    Lines 233 to 235 in b411864

    	if n.IsReservedPort(port) {
    	toKubelet = true
    	useTunnel = true

  • The destination node is labeled with egress.X.io/cluster: true :

    k3s/pkg/daemons/control/tunnel.go

    Line 237 in b411864

    useTunnel = t.egress[nodeName]

• Otherwise does not tunnel

When deciding whether to accept a tunnel connection, the server:

• Always accepts connections to the kubelet port:

  k3s/pkg/agent/tunnel/tunnel.go

  Lines 343 to 344 in 76729d8

  	if a.isKubeletPort(proto, host, port) {
  	return true

• Checks the destination against a list of CIDRs, and accepts the connection if a match is found:

  k3s/pkg/agent/tunnel/tunnel.go

  Lines 346 to 347 in 76729d8

  	if ip := net.ParseIP(host); ip != nil {
  	if nets, err := a.cidrs.ContainingNetworks(ip); err == nil && len(nets) > 0 {

  • This list is not populated if egress-selector-mode is disabled or agent:

    k3s/pkg/agent/tunnel/tunnel.go

    Line 107 in 76729d8

    switch tunnel.mode {

  • If egress-selector-mode is cluster then this list is populated with Node IPs and Cluster CIDRs:

    k3s/pkg/agent/tunnel/tunnel.go

    Line 174 in 76729d8

    func (a *agentTunnel) clusterAuth(config *daemonconfig.Node) {

  • If egress-selector-mode is pod then this list is populated with Node and Pod IPs. If the destination matches a Node IP, then an additional check is made on the destination port, and the connection is accepted only if the destination port belongs to a Pod that has hostNetwork: true:

    k3s/pkg/agent/tunnel/tunnel.go

    Line 190 in 76729d8

    func (a *agentTunnel) watchPods(ctx context.Context, apiServerReady <-chan struct{}, config *daemonconfig.Node) {

    k3s/pkg/agent/tunnel/tunnel.go

    Lines 349 to 350 in 76729d8

    	if p.hostNet {
    	return proto == "tcp" && a.ports[port]

• Otherwise blocks the tunneled connection

In my particular case

I'm running RKE2 with the ingress configured with hostNetwork: true. (This was configured to permit IPv6 access to the ingress from outside the cluster before we had IPv6 working within the cluster. It may no longer be necessary now that we have IPv6 working within the cluster.)

At some point a few months ago, ingress creation/configuration became unreliable. It would eventually succeed if we retried it enough times, but it would usually fail with the following error:

Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": failed to call webhook: Post "https://rke2-ingress-nginx-controller-admission.kube-system.svc:443/networking/v1/ingresses?timeout=30s": EOF

(We now know that it would succeed if the API Server happened to connect to the ingress on the local node, but would fail if the API Server connected to the ingress on any other node.)

The rke2 server log on the source node included:

error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF
Proxy error: read failed: tunnel disconnect

The rke2 server log on the destination node included:

msg="Remotedialer proxy error" error="connection not allowed"

User-Facing Change

Fixed an issue that would cause the apiserver egress proxy to attempt to use the agent tunnel to connect to service endpoints even in agent or disabled mode.

[brandond]

I updated this PR to use the revised approach. Thanks for taking a shot at it @PaulSD - I'd like to get this fixed for this month's release.

[brandond]

For QA:

• Test that by default, the egress.k3s.io/cluster label is absent (egress-selector-mode is agent or disabled)
• Test that the egress.k3s.io/cluster label is present when k3s is in pod or cluster mode.