Add MetalLB to bare-metal deployment page

[antoineco]

What this PR does / why we need it: second part of #3027
Adds a usage example for MetalLB.

Which issue this PR fixes: closes #3055

[kfox1111]

lgtm. :)

[antoineco]

/hold

Some user reported on Slack it was sufficient to set externalIPs on the ingress-nginx Service. This apparently makes kube-proxy add iptables NAT entries that forward all incoming traffic for the externalIP:servicePort combination to the internal Service.

That would be new to me, but let me test.

edit: it does indeed work. Now I definitely feel ignorant. I'll add it to the docs and probably remove the section about NodePorts.

[aledbf]

@antoineco the problem with that approach is that you lose the source IP address
https://kubernetes.io/docs/concepts/services-networking/service/#shortcomings

[antoineco]

@aledbf the docs mention the userspace proxy, but nowadays iptables is the default (and soon ipvs). Wouldn't the Local externalTrafficPolicy solve that issue (with the limitations that we know)?

[aledbf]

@antoineco I am not sure because externalTrafficPolicy is another iptables hack. Also, I think it doesn't work with extrernalips kubernetes/kubernetes#62537 (comment)

[antoineco]

@aledbf fair point. Let me do some more extensive testing. I will also mention that limitation in the MetalLB section (which is also another iptables trick on the node 😄)

[kfox1111]

I have used externalIP without metallb as well, in production. It works well. That is in additional option to using metallb.

the externalIP method has no HA option by itself, nor a way to put the ip on a host unless it is already there. You could add keepalived to do this.

metallb will move the ip around in case of failure.

[aledbf]

You could add keepalived to do this.

Same issue, you lose the source IP address.

[kfox1111]

Yup. But can be worth it for the HA.

You can get close to seamless by using metallb in bgp mode with nginx-ingress with daemonset and the service externalTrafficPolicy=Local. But sticky connections might roll around due to the rehashing.

Really, I think currently there is a complete tradeoff between getting seamless rolling upgrades of nginx-ingress and getting external ip's in the logs. You get one or the other. :(

[antoineco]

/hold cancel

PR and live preview updated with what we discussed.

[aledbf]

/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aledbf, antoineco

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [aledbf,antoineco]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment