Add support deployment on bare metal or VM

[lizj3624]

Description:
I hope gateway support deployment on bare metal or VM

[arkodg]

+1 this can be supported using a File based Provider or a Docker/Container Provider

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[lizj3624]

+1

[tommie]

I'm looking into using envoy-gateway for running on VPS without external LB. Just bootstrapping a node or two. Being able to set externalIP on the generated services would be nice. I see there's movement in KubernetesServiceSpec that could be used to add that.

Has anyone used MetalLB (or similar) with the gateway?

Is there anything else that goes into "supporting deployment" that I don't understand?

Edit #360 seems related to the externalIP approach.

[arkodg]

@tommie would running EnvoyProxy as a docker container satisfy your use case ?

[tommie]

Preferably not. I'm using docker-compose and a cron'd update now, but my (wet) dream is to have a proper orchestrator where I can stop treating my two servers as precious. Trying to build a bottlerocket+kubeadm+flannel+envoyproxy(?)+... solution that can go from bootstrapping on a single cheap node, while also being easy to scale.

I could do something special for ingress, but since K8s is supposed to abstract that, I'd love to avoid it. And since I'm just starting to play with K8s, using Gateway seems preferable over Ingress, muddying the waters with LBs, IIUC.

When I tried patching in externalIPs into the Gateway's Service, that seemed to work until it was "healed", so I think I'm in favor of #360. It requires me to maintain a list of ingress machines, and I'd probably want to abstract that with node labels, but that's much nicer than having to run stuff outside of K8s.

Since Hetzner's cheap servers don't give me a controllable SDN, I don't think MetalLB will work for me (based on a casual investation.) I would basically just move the externalIPs assignment problem one layer down the stack.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[tommie]

Follow-up on #1322, and looking for input on next steps.

The generated gateway services set externalTrafficPolicy: Local, with a comment saying it's to preserve the client address. However, this causes Kubernetes to generate iptables rules like

-A KUBE-EXTERNAL-SERVICES -d 10.0.2.16/32 -p tcp -m comment --comment "envoy-gateway-system/envoy-default-http-64656661:http has no local endpoints" -m tcp --dport 80 -j DROP

on nodes that don't have such a pod running. Probably from here: https://github.com/kubernetes/kubernetes/blob/52353a2e4ac121ccbfc0fc243e5738888286cc96/pkg/proxy/iptables/proxier.go#L1070

Since I'm trying to run Envoy without an external LB, I think I need a way for Gateway to run a pod on (at least) each node that is referenced in gateway.spec.addresses. Currently, it seems replica count is static, and there is no node affinity that could link the assigned external IPs to the replicas.

I wonder if the Envoy Deployment should be a DaemonSet in this case instead. We could use a special AddressType that identifies a set of nodes by label, and then this label could be used for finding nodes both for the Service.spec.externalIPs and the DaemonSet node selector.

I've built a controller to update Gateway addresses based on labelled nodes, which does the first part, but I don't think the Envoy configmap is generic enough right now to switch to a DaemonSet.

[tommie]

Some references about DaemonSet:

The nginx-ingress Helm chart allows you to specify the controller kind: https://github.com/kubernetes/ingress-nginx/blob/24cd56d27c60b550ee103bff1ccbef980f614c7e/charts/ingress-nginx/templates/controller-daemonset.yaml#L1

They discuss it as a way to avoid unschedulable pods, rather than as ingress using node IP addresses: https://kubernetes.github.io/ingress-nginx/deploy/baremetal/#via-the-host-network

Seems there is some precedent for DaemonSet, at least.

[tommie]

Here's a PoC: https://github.com/envoyproxy/gateway/compare/main...tommie:envoyproxy-gateway:daemonset?expand=1

By setting replicas: -1, the Deployment becomes a DaemonSet. I've also added nodeSelector to the pod template:

# https://github.com/envoyproxy/gateway/blob/main/docs/latest/user/customize-envoyproxy.md
apiVersion: config.gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: envoy-config
  namespace: envoy-gateway-system
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        replicas: -1  # DaemonSet
        pod:
          nodeSelector:
            somelabel: ""

[arkodg]

@tommie if you're running without any LB support, then you should be using a different service type such as NodePort https://gateway.envoyproxy.io/v0.4.0/api/config_types.html#kubernetesservicespec

[tommie]

@tommie if you're running without any LB support, then you should be using a different service type such as NodePort https://gateway.envoyproxy.io/v0.4.0/api/config_types.html#kubernetesservicespec

Thanks, yes, good point. I think LoadBalancer is the default, right? And AFAICT, it does what NodePort does, and since I don't have a cloud provider configured, it shouldn't do anything else. The only difference may be that a Gateway status isn't updated if there is no external address populated by the cloud provider?

[arkodg]

yes, LoadBalancer is a superset of Nodeport but it looks it EG is making some opinionated settings/optimizations for LoadBalancer.
Have you tried the Nodeport option ?, it should be available in the latest version
https://gateway.envoyproxy.io/latest/api/config_types.html#kubernetesservicespec
was introduced as part of #1378

[tommie]

Have you tried the Nodeport option ?

Yes, though I didn't realize you are suggesting it's enough to solve the problem.

Thinking out loud: Let's say I set replicas to 2. EG Controller schedules Proxy pods on some random nodes. I will at least want a nodeSelector to ensure I have control over which nodes are public gateways. I thought I needed to guarantee that Proxy was running on all selected nodes (not constrained by replicas,) but actually, ExternalDNS is looking at the routes I have, so as long as EG Controller keeps status addresses up-to-date, perhaps I don't need DaemonSet... I made the comment about DaemonSet before I had ExternalDNS configured, so perhaps that additional controller removes the DaemonSet need.

Am I getting that right?

Edit: ... before ExternalDNS I just noted that I couldn't control where the replicas: 1 Deployment ends up, so I'd just have to know it somehow. But that problem is solved with a layer of indirection.

[arkodg]

yah it looks like this can work

• Set EnvoyProxy.Spec.Provider.Kubernetes.EnvoyService.Type to NodePort
• Set EnvoyProxy.Spec.Provider.Kubernetes.EnvoyDeployment.Pod.Affinity https://gateway.envoyproxy.io/latest/api/config_types.html#kubernetespodspec to ensure the pods run on specific nodes
• EG should hopefully update Gateway.Status.Addresses with the Node Addresses
  

  gateway/internal/status/gateway.go

  Line 53 in b1a7520

  if svc.Spec.Type == corev1.ServiceTypeNodePort {

  
  This will populate it with all Node Addresses, in case you have some private addresses, im not sure how you can limit that
• Integrate ExternalDNS

[tommie]

@arkodg Thank you! Closed the PR. I'll play around with this some more.

[arkodg]

cool, please feel free to raise GH issues, if you notice anything missing

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[arkodg]

closed in favor of #1393

[vladimirfx]

closed in favor of #1393

How does running out of cluster solve in-cluster deployment issue? For bare metal providers L2 load balancers may not be available (Hetzner). Such providers provide failover IP machinery (using BGP or proprietary API) and is up to the user how to ingest traffic. External DNS is slow compared to switching failover IP. In addition, BGP routing may balance traffic with the same IP to many nodes at the same time.
All of this requires running a proxy on a selected set of nodes. So... DaemonSet is the direct solution.
Currently, we implemented a workaround using specific replica count and affinity. But this is manual and error-prone.