Add more extensible/custom load balancers

[lvermeulen]

What should we add or change to make your life better?

More load balancing strategies.

In short, I would like to:

• Have a generic ILoadBalancingStrategy
• Have several implementations of this ILoadBalancingStrategy
• Make this extensible somehow

Why is this important to you?

I would like to have more choices when I choose the load balancing strategy.

The scope would initially be:

• A callback load balancing strategy: to allow a function to decide which endpoint gets picked next
• A deficit round robin load balancing strategy: to allow round robin taking into account a quantum (see Deficit round robin) and thus providing a "fairer" round robin load balancing strategy
• A failover load balancing strategy: to allow an availability predicate to decide which endpoint gets picked next
• A traffic allocation load balancing strategy: to allow inspection into traffic allocation of the endpoints (no matter how that is measured)

I have submitted the algorithms described above (with passing tests). The ILoadBalacingStrategy interface has a single Balance method which takes as parameters the list of available endpoints and the load balancing options.

I am unsure how to handle the configuration for the algorithms. Right now I have added configuration to the BackEndConfig.BackEndLoadBalancingOptions, which felt a little odd. :-)

[samsp-msft]

Updated title to be clearer.

[Tratcher]

Why these algorithms specifically? Have you had situations elsewhere that these were able to address?

• A callback load balancing strategy: to allow a function to decide which endpoint gets picked next

If all this does is run a custom user callback then there are several other ways to do that like the new middleware model (#50). It's not clear that it gains anything by being integrated directly into the LoadBalancer.

• A deficit round robin load balancing strategy: to allow round robin taking into account a quantum (see Deficit round robin) and thus providing a "fairer" round robin load balancing strategy

The description sounds like a credit based LeastRequests strategy. I don't see any obvious benefits over the existing algorithms?

• A failover load balancing strategy: to allow an availability predicate to decide which endpoint gets picked next

We expect to address this with health checks, retries, etc., not as part of load balancing.

• A traffic allocation load balancing strategy: to allow inspection into traffic allocation of the endpoints (no matter how that is measured)

Is this an abstract version of LeastBusy with a customizable score? It's hard to know how useful this would be without some examples of how the score might be calculated and if the right primitives are available to do that calculation.

I am unsure how to handle the configuration for the algorithms. Right now I have added configuration to the BackEndConfig.BackEndLoadBalancingOptions, which felt a little odd. :-)

Yes, we do need to figure that out. RoundRobin is the only one so far that's needed any state, and that ones one property.

The ILoadBalancingStrategy abstraction I would hold off on until we worked out how you might flow such implementations and their configurations through the config system.

[lvermeulen]

The ILoadBalancingStrategy abstraction I would hold off on until we worked out how you might flow such implementations and their configurations through the config system.

I will hold off on it until then.

[Tratcher]

When designing session affinity (#174) we worked out an extensibility model that we should be able to apply here as well. Here's a comparison for what it might look like:

• AffinitizedDestinationLookupMiddleware -> LoadBalancingMiddleware
• ISessionAffinityProvider -> ILoadBalancingPolicy (new)
• SessionAffinityOptions -> LoadBalancingOptions

ILoadBalancer would go away, LoadBalancingMiddleware would become responsible for resolving an IEnumrable<ILoadBalancingPolicy> from DI and finding the right policy for the request/cluster. The LoadBalancingMode enum would go away, we'd use a string name on the options to look up the policy. The five load balancing algorithms we currently support would each be moved to their own ILoadBalancingPolicy implementation. The default algorithm wouldn't change, we'd still use PowerOfTwoChoices.

Load balancing would also get it's own docs article that described how to select a policy, the available policies, and the extensibility mode.

Out of scope: new load balancing strategies. For now we're concerned with providing the extensibility.