Cilium L2 Announcements compatibility

[vixns]

We are using EnvoyGateway on cilium load balancer with L2 Announcements.

Cilium uses leases to assign the IP to a node, as L2 LB is more a fail over IP than a real load balancer.
When there is no envoy(gateway) pod on this node, the gateway is not reachable on the external/LB IP.

Our current current workaround is setting envoyDeployment.replicas to the dataplane's node count.

One way to address this would be to run envoy as DaemonSet, simple and efficient, but we may not want to run EnvoyGateway on all/some nodes.

A better way would be to automatically run an Envoy pod on all nodes matching the corresponding CiliumL2AnnouncementPolicy CR's selector.

https://docs.cilium.io/en/stable/network/l2-announcements/

[shawnh2]

this is quite similar to #2969

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[zvlb]

#3092 - run envoy-gatewat like daemonset already merged

[zirain]

#3092 should help this, please reopen if need more work.