Support for application gateways with multiple backend pools

[Justin-W]

Is your feature request related to a problem? Please describe.

A single Azure Application Gateway can have multiple backend pools of VMs, and route traffic to different pools in various ways (e.g. path-based routing) so that a single APG front-end handles all traffic for multiple backends. However, it appears that the CPI's existing support for adding VMs to APGs (via the application_gateway property of vm_types/vm_extensions config data) which was added in #169 and #312 always attaches any Bosh-created VMs to the first backend pool, making it impossible to configure Bosh to manage the VMs of any APG backend pool(s) except the first pool, which currently forces us to configure a separate APG (with a separate frontend, etc. per APG) for each Bosh-managed backend pool.

Describe the solution you'd like

I would like to be able to use Bosh to manage multiple backend pools of VMs, where:

1. multiple pools are attached to the same APG; (e.g. a VM needs to be attached to a single pool, but it's not the AP's first pool)
2. some VMs may need to be attached to multiple backend pools (of the same APG); (e.g. such VMs handle traffic for more than one backend port/protocol/path/etc.)
3. (Optionally) some VMs may need to be attached to (single or multiple) backend pools of multiple APGs; (e.g. such VMs handle traffic for multiple APGs/frontends; and also potentially multiple backends of some or all of those APGs, as in requirement 2)

The first 2 requirements above would seem to be most easily addressed by adding support for a new application_gateway_backend_pools property (as a list/array, to address requirement 2 mentioned above) in the vm_types/vm_extensions config data.

And although the 3rd requirement adds additional value hypothetically, my current use case doesn't technically require it (although I've addressed it in one of the alternative implementations discussed below).

Describe alternatives you've considered

Alternative 1: We could change the existing application_gateway property of vm_types/vm_extensions config data from a String to a Map/Hash which would support 2 properties (e.g. name and backend_pools), but that would be backwards-incompatible with existing configurations which already pass a String value.

Alternative 2: We could (maybe?) change the existing application_gateway property of vm_types/vm_extensions config data to allow either a String or a Map/Hash (as described in the previous alternative), since that would address backwards-compatibility. But this might be more complex to implement, test, and document/understand. Also, I'm not sure if multi-typed config properties are possible and/or permitted, as I don't recall any other such multi-typed properties in this or other CPIs' cloud properties. So I figured that this alternative was probably infeasible.

Alternative 3: We could add support for a new application_gateway_backend_pool property (as a String instead of a List), but that would not address requirement 2 (which is the most important of the 3 requirements for my use cases).

Alternative 4: we could add a new application_gateways property (as a List/Array of Maps/Hashes like those described in alternative 2 above) of vm_types/vm_extensions config data, as a more flexible alternative to the existing application_gateway String property. This would enable VMs to be attached to multiple backend pools of multiple APGs (optional requirement 3 above). And the config schema would seem consistent with how some other sections of Bosh config (e.g. the az and azs properties documented here) currently handle multi-typed config data alternatives, and would be backwards-compatible. But this alternative would require the CPI implementation support, test, etc. both the old and new properties, including extending the existing APG functionality to support VMs being attached to both multiple backends (requirement 2) and multiple APGs (requirement 3), and I'm not certain how much more complex it would be to implement support for the 3rd requirement (which I don't currently have a direct need for). However, if anyone will have a legitimate need for the 3rd requirement, then supporting it now would probably be better than adding support for it later (which would probably involve even more backwards-compatibility-related schema issues).

Additional context

• Initial support for APGs (first backend pool only) was added by allow use to associate application gateways in vm cloud properties #169 and Support application gateway #312
• Support for multiple Load Balancers (but only the first backend pool of each LB) was recently added by Support multiple load_balancer entries #111 and Add support for multiple load balancers #638, and is included in v37.6.0.
• APGs support multiple backend pools: https://docs.microsoft.com/en-us/azure/application-gateway/application-gateway-components#backend-pools
• APG path-based routing docs: https://docs.microsoft.com/en-us/azure/application-gateway/url-route-overview

[Justin-W]

FYI: I'm planning on trying to implement this feature myself ASAP, but given the config schema backwards-compatibility issues I mentioned above, I was hoping for a bit of guidance on the design/implementation alternatives I outlined above.

I'm thinking that the primary design I recommended above (i.e. a new application_gateway_backend_pools property) or else alternative 4 are the most feasible (especially from a backwards-compatibility perspective). However, alternatives 1 and 2 may also be feasible if someone can provide me with a bit of guidance (about how to proceed with backwards-incompatible config schema changes and/or multi-typed String/Hash config schema elements, respectively).

@klakin-pivotal @rkoster @bgandon Any thoughts or tips before I start work on a PR?

[donghoang89]

We are facing the same issue. This would be a great feature to have.

[rkoster]

I'm in favor of alternative 4, since, as you pointed out already, it is in line with how similar changes have been rolled out before (the az, azs example being one of those).

[beyhan]

+1
We should also mark the old way as deprecated in case this feature gets implemented and add a short example in the docs how to switch from the old property to the new one.

[bosh-admin-bot]

This issue was marked as Stale because it has been open for 21 days without any activity. If no activity takes place in the coming 7 days it will automatically be close. To prevent this from happening remove the Stale label or comment below.

[Justin-W]

FYI: PR #651 is a combination of previously-discussed (see above) alternatives 2 and 4. Once I dug into the code, I realized that the vm_type/load_balancer config already allowed either a String or a Hash (and also allowed a hacky version of multiple LBs via a delimited String, instead of an Array). Since there is much symmetry between the vm_type/load_balancer and vm_type/application_gateway config sections, it seemed best to maintain that symmetry (by making both sections capable of supporting an Array of Hashes), and making both sections support the same 3 sub-properties.

[Justin-W]

FYI: PR cloudfoundry/docs-bosh#761 updates the CPI docs to be consistent with the changes in #651 (and also some previous PRs; see cloudfoundry/docs-bosh/pull/761 descriptions for details).

We should also mark the old way as deprecated in case this feature gets implemented and add a short example in the docs how to switch from the old property to the new one.

Note that I didn't make the "old" String format deprecated (in either the CPI code or docs PRs), because:

• the old String-only format continues to work fine for some use cases, and involves simpler config than the newer, more powerful Hash and Array formats
• the original and newer formats are both supported by the new implementation
• there are still many specs, documentation pages, etc. that continue to use the original String-only format
• I ended up not needing to create a new, separate config entry, so there is no possibility of conflicts from configs containing both formats (unless they have dupe config properties with different formats, which would be more of a YAML correctness issue than a Bosh config issue)

[bosh-admin-bot]

This issue was marked as Stale because it has been open for 21 days without any activity. If no activity takes place in the coming 7 days it will automatically be close. To prevent this from happening remove the Stale label or comment below.

[bosh-admin-bot]

This issue was marked as Stale because it has been open for 21 days without any activity. If no activity takes place in the coming 7 days it will automatically be close. To prevent this from happening remove the Stale label or comment below.