Setting load-balancing policies in the era of CRDs

[zivnevo]

Background

In the proposed CRD-based management, load-balancing policies are defined as a field in the Spec of the Import CRD. The rational here is to allow assigning a load-balancing scheme to each remote service separately, which makes a lot of sense, because remote services are deployed differently and exhibit various performance patterns.
The field type is currently undefined.

In this discussion we would like to decide:

1. Is Import the right place to define load-balancing schemes? Another option would be a dedicated load-balancing policy, as we have today.
2. If we stay with Import, what should be the type of the field? A simple string? a map from client-name to a scheme? A more complex type?

Note that if we choose to define the load-balancing scheme inside Import and use just a simple string, we lose the ability to set the scheme based on the requesting client. This might be important, as demonstrated by the following use-case.

A possible use-case for setting a per-client scheme

We have two clients:

1. The high-priority client is doing some important job and requires as-fast-as-possible responses to its request
2. The low-priority client is doing some maintenance stuff, and is OK with waiting a bit for responses to its request
   If we have a per-client scheme, we may allow the high-priority client to access the high-availability remote services, while the low-priority client can always access the same low-latency peer.

This use-case hints towards a need for a load-balancing policy to assign the load-balancing scheme to a <client, service> pair, based on their attributes. We don't really want to handle a herd of clients by specifying their names (as if they were pets).

Additional things to consider

• Do we need more complex schemes, e.g., weighted random, or ring hash (as in envoy)? These may require setting more data fields (weights, hash key)

[orozery]

I think we should optimize the API for the most common use-case, which in my opinion is the following order (starting from most common):

1. Single source (no load balancing)
2. Primary source with fallback
3. Client-dependent load-balancing / Random/weighted/other load-balancing (can't tell which one should be more common)

I feel like 1 will be 90% of the cases, and 2 will probably be at least twice as common as 3.
Assuming this, I suggest we can remove the load-balancing spec from the import, and instead, unless a specific LB object tells otherwise, use the Sources array as an ordered fallback list.