fix(envoy): create a CDS cluster per model

[driev]

What this PR does / why we need it:

When scaling a model replica count up, existing clusters were removed and new ones were added in the delta response, as the cluster name changed. This resulted in the downstream receiving 503s due to no cluster being found.

Instead of changing the cluster name everytime the number of replicas changes, just keep the cluster name static, so clusters will be updated in place.

Which issue(s) this PR fixes:

Fixes #
INFRA-1150

Special notes for your reviewer:

[sakoush]

In principle this looks that it will solve the reported 503 issue. However I am not sure how this will scale with the number of models. Any thoughts on how envoy can handle clusters in the 100s-1000s?

Supporting this scale might not be of immediate concern though (yet).

[driev]

In principle this looks that it will solve the reported 503 issue. However I am not sure how this will scale with the number of models. Any thoughts on how envoy can handle clusters in the 100s-1000s?

Supporting this scale might not be of immediate concern though (yet).

There's no limit on the number of clusters envoy can handle, once resources are allocated appropriately - since these clusters don't have active health checking the overhead of having one per model shouldn't be too expensive.

[driev]

And for reference, performance will likely be impacted by the number of stats generated by the clusters.

envoyproxy/envoy#19946

[sakoush]

LGTM! should we also remove computeHashKeyForList util as it is not being used anymore (we can always recover from history if required)?

[driev]

LGTM! should we also remove computeHashKeyForList util as it is not being used anymore (we can always recover from history if required)?

It's gone - check the first change in the list.

[sakoush]

LGTM! should we also remove computeHashKeyForList util as it is not being used anymore (we can always recover from history if required)?

It's gone - check the first change in the list.

Thanks for some reason I missed it, I should have looked more carefully.