Support multiple load_balancer entries

[svrc]

Hi all, here's a scenario , let me know what you think of it

1. It is good to have split horizon DNS for a CF instance, i.e. on the devbox you have a named/bind9 server that resolves *.cf.azurelovecf.com to 10.10.16.4 inside the VNet, and the Public IP outside the VNet. This avoids hairpin NAT routing to the load balancer public IP address which sometimes can be slower and flaky than remaining within the VNet.
2. In a high availability enterprise scenario, we should use an Azure Load Balancer created ahead of time and have it load balance to two HAproxy instances annotated with the load_balancer cloud property.
3. However an azure load balancer can only have a Public OR a Private IP address, it cannot have both.
4. Therefore I think it makes sense to be able to have two (or more?) load_balancer entries, one for a Private load balancer and one for a Public load balancer.

Let me know if this is a reasonable enhancement. Right now as workaround, I'd need to create two resource pools and jobs for haproxy.

[AbelHu]

@svrc-pivotal It is a good idea. Let us do some investigation.

[AbelHu]

Related to Azure/azure-powershell#2042

[AbelHu]

@svrc-pivotal Now Azure has supported to create one external LB and one internal LB in the same NIC. Could you have a try?

[AbelHu]

@svrc-pivotal Need to correct. Azure can support to create create one external LB and one internal LB in the same NIC but the port rules cannot be same. We are still investigating.

[gossion]

There is a question and answer at this link about multiple load balancers, saying that:

1. Azure NICs can participate in multiple load-balancer: one external and one internal.
2. LB rules on the external and internal load-balancers cannot use the same backend port.
3. Only the primary NIC can be used in load-balancer backend pools.

I had tried to create rule tcp 22:22 for both external and internal load balancers and assigned those two lbs to the same VM, I got error code "RulesUseSameBackendPortProtocolAndIPConfig" when creating the VM.

Hi @svrc-pivotal, are you expecting to use same backend port in load balancers in CF? Azure does not support such feature now.

[svrc]

Hi @gossion yeah the idea was that I could reduce the number of HA proxy instances.. they would use the same backend port. I guess we'll have to wait to see if Azure improves LBs to enable the same backend ports.

[gossion]

Hi @svrc-pivotal ,

Now in Azure the secondary NICs can also be used in LB backend pools, so this feature is doable. However, the secondary NICs can't be probed by LB because by default the outbound traffic will go to gateway of primary NIC. In order to make LB work with the secondary NICs we need to configure a policy route in the VM, which means we need a bosh release to configure the network additionally. I had a PR in networking-release, trying to implement the policy routing for this scenario, but it is not reviewed / merged yet.

It looks like this is not a common scenario, and the issue has last for a long time. Not sure if you have already solved the issue by any other methods, so my question is - do you still need this feature?

[svrc]

@gossion At the moment we deploy separate VMs if we need separate load balancers. I have at least two large Azure customers deploying separate VMs for internal vs. external facing HAproxies for example.

I guess my point was that Azure is the only cloud I know (out of AWS, OpenStack, GCP, vSphere) that requires multiple NICs for multiple load balancers, so this should be a common case. Multi-NIC could be a workaround but seems too early.

We can close this issue if it doesn't look like Azure LBs will ever allow a single NIC to expose to multiple load balancers, as it's a larger concern than just this CPI :)

[svrc]

This should work with standard LB skus and maybe should be revisited, especially considering SNAT with standard LBs requires a public LB with unused inbound rules to denote SNAT outbound IPs/ports (scenario 2 here - https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-outbound-connections ).

I could forsee a case where if using standard LBs that every VM in a BOSH deployment gets a public LB assigned for outbound SNAT purposes, and then a subset of VMs require a second LBs for internal purposes. (Or is is preferred to mix internal/external frontends on the same LB in a standard case? admittedly I don't know)

[daviddob]

Any updates regarding supporting multiple LBs attached to the same VM orchestrating via BOSH? We currently are accomplishing this using a separate piece of automation to attach the VMs after deployment to the second LB (one internal, one external), however it would be nice to simplify and eliminate the need for out of band changes.

[bosh-admin-bot]

This issue was marked as Stale because it has been open for 21 days without any activity. If no activity takes place in the coming 7 days it will automatically be close. To prevent this from happening remove the Stale label or comment below.

[daviddob]

Not stale - waiting on updates from the BOSH team.

[bosh-admin-bot]

This issue was marked as Stale because it has been open for 21 days without any activity. If no activity takes place in the coming 7 days it will automatically be close. To prevent this from happening remove the Stale label or comment below.

[bosh-admin-bot]

This issue was closed because it has been labeled Stale for 7 days without subsequent activity. Feel free to re-open this issue at any time by commenting below.

[MSSedusch]

this issue should be fixed with PR #638 which was released with v37.6.0.