Add support for full node mesh

[raffaelespazzoli]

Full node mesh can be support with the wireguard VPN implementation.
Full node mesh adds the following:

1. maximum theoretical bandwidth exploitation
2. best possible resilience to node failure, in fact with n active gateways (Epic: Add support for multiple active gateways #34 ) a node failure causes a drop in bandwidth of 1/n. But with full node mesh a node failure causes a drop in bandwidth of zero as the only workload that is impacted by it is also failed.

[mangelajo]

How would full-node mesh work in enviroments were not all nodes have PublicIPs ? would we create a load balancer per worker node?

The AWS Implementation of the K8s network load balancer works particularly bad when multiple ports are added (if that is an alternative), at least it's what I have been experiencing empirically.

[raffaelespazzoli]

this old project gives an idea of how it would be possible: https://github.com/Huang-Wei/shared-loadbalancer . I realize there is some work do to there. In the immediate, having the feature when the nodes are reachable (VPC peering, bare metal on prem), would be still a win.

[dfarrell07]

@sridhargaddam mentioned there is a dependency on Load Balancer work here.

[nyechiel]

This was discussed in the Submariner community and the general feeling is that this is not aligned with our short-to-medium term goals.