Make load-balancing extensible

docs/docfx/articles/load-balancing.md

@@ -0,0 +1,116 @@
+# Load Balancing
+
+Introduced: preview8
+
+## Introduction
+
+Whenever there are multiple healthy destinations available, YARP has to decide which one to use for a given request.
+YARP ships with built-in load-balancing algorithms, but also offers extensibility for any custom load balancing approach.
+
+## Configuration
+
+### Services and middleware registration
+
+Load balancing policies are registered in the DI container via the `AddLoadBalancingPolicies()` method, which is automatically called by `AddReverseProxy()`.
+
+The middleware is added with `UseProxyLoadBalancing()`, which is included by default in the parameterless `MapReverseProxy` method.
+
+### Cluster configuration
+
+The algorithm used to determine the destination can be configured by setting the `LoadBalancingPolicy`.
+
+If no policy is specified, `PowerOfTwoChoices` will be used.
+
+#### File example
+
+```JSON
+"ReverseProxy": {
+    "Clusters": {
+        "cluster1": {
+            "LoadBalancingPolicy": "RoundRobin",
+            "Destinations": {
+                "cluster1/destination1": {
+                    "Address": "https://localhost:10000/"
+                },
+                "cluster1/destination2": {
+                    "Address": "https://localhost:10010/"
+                }
+            }
+        }
+    }
+}
+```
+
+#### Code example
+
+```C#
+var clusters = new[]
+{
+    new Cluster()
+    {
+        Id = "cluster1",
+        LoadBalancingPolicy = LoadBalancingPolicies.RoundRobin,
+        Destinations =
+        {
+            { "destination1", new Destination() { Address = "https://localhost:10000" } },
+            { "destination2", new Destination() { Address = "https://localhost:10010" } }
+        }
+    }
+};
+```
+
+## Built-in policies
+
+YARP ships with the following built-in policies:
+- `First`
+
+    Select the first destination without considering load. This is useful for dual destination fail-over systems.
+- `Random`
+
+    Select a destination randomly.
+- `PowerOfTwoChoices` (default)
+
+    Select two random destinations and then select the one with the least assigned requests.
+    This avoids the overhead of `LeastRequests` and the worst case for `Random` where it selects a busy destination.
+- `RoundRobin`
+
+    Select a destination by cycling through them in order.
+- `LeastRequests`
+
+    Select the destination with the least assigned requests. This requires examining all destinations.
+
+## Extensibility
+
+`ILoadBalancingPolicy` is responsible for picking a destination from a list of available healthy destinations.
+
+A custom implementation can be provided in DI.
+
+```c#
+// Implement the ILoadBalancingPolicy
+public sealed class LastLoadBalancingPolicy : ILoadBalancingPolicy
+{
+    public string Name => "Last";
+
+    public DestinationInfo PickDestination(HttpContext context, IReadOnlyList<DestinationInfo> availableDestinations)
+    {
+        return availableDestinations[^1];
+    }
+}
+
+// Register it in DI
+services.AddSingleton<ILoadBalancingPolicy, LastLoadBalancingPolicy>();
+
+// Set the LoadBalancingPolicy on the cluster
+cluster.LoadBalancingPolicy = "Last";
+```
+
+Other information that may be necessary to decide on a destination, such as cluster configuration, can be accessed from the `HttpContext`:
+
+```c#
+public DestinationInfo PickDestination(HttpContext context, IReadOnlyList<DestinationInfo> availableDestinations)
+{
+    var proxyFeature = context.Features.Get<IReverseProxyFeature>();
+    var clusterConfig = proxyFeature.ClusterConfig;
+    // ...
+}
+```

docs/docfx/articles/proxyhttpclientconfig.md

@@ -99,9 +99,7 @@ The below example shows 2 samples of HTTP client and request configurations for
 {
     "Clusters": {
         "cluster1": {
-            "LoadBalancing": {
-                "Mode": "Random"
-            },
+            "LoadBalancingPolicy": "Random",
             "HttpClient": {
                 "SslProtocols": [
                     "Tls11",

docs/docfx/articles/service-fabric-int.md

@@ -73,7 +73,7 @@ YARP integration is enabled and configured per each SF service. The configuratio
 These are the supported parameters:
 - `YARP.Enable` - indicates whether the service opt-ins to serving traffic through YARP. Default `false`
 - `YARP.EnableDynamicOverrides` - indicates whether application parameters replacement is enabled on the service. Default `false`
-- `YARP.Backend.LoadBalancing.Mode` - configures YARP load balancing mode. Optional parameter
+- `YARP.Backend.LoadBalancingPolicy` - configures YARP load balancing policy. Optional parameter
 - `YARP.Backend.SessionAffinity.*` - configures YARP session affinity. Available parameters and their meanings are provided on [the respective documentation page](session-affinity.md). Optional parameter
 - `YARP.Backend.HttpRequest.*` - sets proxied HTTP request properties. Available parameters and their meanings are provided on [the respective documentation page](proxyhttpclientconfig.md) in 'HttpRequest' section. Optional parameter
 - `YARP.Backend.HealthCheck.Active.*` - configures YARP active health checks to be run against the given service. Available parameters and their meanings are provided on [the respective documentation page](dests-health-checks.md). There is one label in this group `YARP.Backend.HealthCheck.Active.ServiceFabric.ListenerName` which is not covered by that document because it's SF specific. Its purpose is explained below. Optional parameter

docs/docfx/articles/session-affinity.md

@@ -7,7 +7,7 @@ Session affinity is a mechanism to bind (affinitize) a causally related request
 
 ## Configuration
 ### Services and middleware registration
-Session affinity services are registered in the DI container via `AddSessionAffinityProvider()` method which is automatically called by `AddReverseProxy()`. The middleware `UseAffinitizedDestinationLookup()` and `UseRequestAffinitizer()` are included by default in the paramterless MapReverseProxy method. If you are customizing the proxy pipeline, place the first middleware **before** adding `LoadBalancingMiddleware` and the second **after** load balancing.
+Session affinity services are registered in the DI container via `AddSessionAffinityProvider()` method which is automatically called by `AddReverseProxy()`. The middleware `UseAffinitizedDestinationLookup()` and `UseRequestAffinitizer()` are included by default in the parameterless MapReverseProxy method. If you are customizing the proxy pipeline, place the first middleware **before** adding `LoadBalancingMiddleware` and the second **after** load balancing.
 
 Example:
 ```C#

samples/ReverseProxy.Config.Sample/appsettings.json

@@ -20,9 +20,7 @@
   "ReverseProxy": {
     "Clusters": {
       "cluster1": {
-        "LoadBalancing": {
-          "Mode": "Random"
-        },
+        "LoadBalancingPolicy": "Random",
         "SessionAffinity": {
           "Enabled": "true",
           "Mode": "Cookie"

src/ReverseProxy.ServiceFabric/ServiceDiscovery/Util/LabelsParser.cs

@@ -165,17 +165,14 @@ internal static Cluster BuildCluster(Uri serviceName, Dictionary<string, string>
 
             var clusterId = GetClusterId(serviceName, labels);
 
-            var loadBalancingModeLabel = GetLabel<string>(labels, "YARP.Backend.LoadBalancing.Mode", null);
             var versionLabel = GetLabel<string>(labels, "YARP.Backend.HttpRequest.Version", null);
 #if NET
             var versionPolicyLabel = GetLabel<string>(labels, "YARP.Backend.HttpRequest.VersionPolicy", null);
 #endif
             var cluster = new Cluster
             {
                 Id = clusterId,
-                LoadBalancing = !string.IsNullOrEmpty(loadBalancingModeLabel)
-                    ? new LoadBalancingOptions { Mode = (LoadBalancingMode)Enum.Parse(typeof(LoadBalancingMode), loadBalancingModeLabel) }
-                    : null,
+                LoadBalancingPolicy = GetLabel<string>(labels, "YARP.Backend.LoadBalancingPolicy", null),
                 SessionAffinity = new SessionAffinityOptions
                 {
                     Enabled = GetLabel(labels, "YARP.Backend.SessionAffinity.Enabled", false),

src/ReverseProxy/Abstractions/ClusterDiscovery/Contract/Cluster.cs

@@ -21,9 +21,9 @@ public sealed class Cluster : IDeepCloneable<Cluster>
         public string Id { get; set; }
 
         /// <summary>
-        /// Load balancing options.
+        /// Load balancing policy.
         /// </summary>
-        public LoadBalancingOptions LoadBalancing { get; set; }
+        public string LoadBalancingPolicy { get; set; }
 
         /// <summary>
         /// Session affinity options.
@@ -61,7 +61,7 @@ Cluster IDeepCloneable<Cluster>.DeepClone()
             return new Cluster
             {
                 Id = Id,
-                LoadBalancing = LoadBalancing?.DeepClone(),
+                LoadBalancingPolicy = LoadBalancingPolicy,
                 SessionAffinity = SessionAffinity?.DeepClone(),
                 HealthCheck = HealthCheck?.DeepClone(),
                 HttpClient = HttpClient?.DeepClone(),
@@ -84,7 +84,7 @@ internal static bool Equals(Cluster cluster1, Cluster cluster2)
             }
 
             return string.Equals(cluster1.Id, cluster2.Id, StringComparison.OrdinalIgnoreCase)
-                && LoadBalancingOptions.Equals(cluster1.LoadBalancing, cluster2.LoadBalancing)
+                && string.Equals(cluster1.LoadBalancingPolicy, cluster2.LoadBalancingPolicy, StringComparison.OrdinalIgnoreCase)
                 && SessionAffinityOptions.Equals(cluster1.SessionAffinity, cluster2.SessionAffinity)
                 && HealthCheckOptions.Equals(cluster1.HealthCheck, cluster2.HealthCheck)
                 && ProxyHttpClientOptions.Equals(cluster1.HttpClient, cluster2.HttpClient)

src/ReverseProxy/Abstractions/ClusterDiscovery/Contract/LoadBalancingPolicies.cs

@@ -0,0 +1,37 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+namespace Microsoft.ReverseProxy.Abstractions.ClusterDiscovery.Contract
+{
+    /// <summary>
+    /// Names of built-in load balancing policies.
+    /// </summary>
+    public static class LoadBalancingPolicies
+    {
+        /// <summary>
+        /// Select the first destination without considering load. This is useful for dual destination fail-over systems.
+        /// </summary>
+        public static string First => nameof(First);
+
+        /// <summary>
+        /// Select a destination randomly.
+        /// </summary>
+        public static string Random => nameof(Random);
+
+        /// <summary>
+        /// Select a destination by cycling through them in order.
+        /// </summary>
+        public static string RoundRobin => nameof(RoundRobin);
+
+        /// <summary>
+        /// Select the destination with the least assigned requests. This requires examining all destinations.
+        /// </summary>
+        public static string LeastRequests => nameof(LeastRequests);
+
+        /// <summary>
+        /// Select two random destinations and then select the one with the least assigned requests.
+        /// This avoids the overhead of LeastRequests and the worst case for Random where it selects a busy destination.
+        /// </summary>
+        public static string PowerOfTwoChoices => nameof(PowerOfTwoChoices);
+    }
+}

src/ReverseProxy/Configuration/ConfigurationConfigProvider.cs

@@ -146,7 +146,7 @@ private Cluster CreateCluster(IConfigurationSection section)
             var cluster = new Cluster
             {
                 Id = section.Key,
-                LoadBalancing = CreateLoadBalancingOptions(section.GetSection(nameof(Cluster.LoadBalancing))),
+                LoadBalancingPolicy = section[nameof(Cluster.LoadBalancingPolicy)],
                 SessionAffinity = CreateSessionAffinityOptions(section.GetSection(nameof(Cluster.SessionAffinity))),
                 HealthCheck = CreateHealthCheckOptions(section.GetSection(nameof(Cluster.HealthCheck))),
                 HttpClient = CreateProxyHttpClientOptions(section.GetSection(nameof(Cluster.HttpClient))),
@@ -224,19 +224,6 @@ private static RouteHeader CreateRouteHeader(IConfigurationSection section)
             return routeHeader;
         }
 
-        private static LoadBalancingOptions CreateLoadBalancingOptions(IConfigurationSection section)
-        {
-            if (!section.Exists())
-            {
-                return null;
-            }
-
-            return new LoadBalancingOptions
-            {
-                Mode = section.ReadEnum<LoadBalancingMode>(nameof(LoadBalancingOptions.Mode)) ?? LoadBalancingMode.PowerOfTwoChoices,
-            };
-        }
-
         private static SessionAffinityOptions CreateSessionAffinityOptions(IConfigurationSection section)
         {
             if (!section.Exists())

src/ReverseProxy/Configuration/DependencyInjection/BuilderExtensions/IReverseProxyBuilderExtensions.cs

@@ -8,8 +8,8 @@
 using Microsoft.ReverseProxy.Service;
 using Microsoft.ReverseProxy.Service.Config;
 using Microsoft.ReverseProxy.Service.HealthChecks;
+using Microsoft.ReverseProxy.Service.LoadBalancing;
 using Microsoft.ReverseProxy.Service.Management;
-using Microsoft.ReverseProxy.Service.Proxy;
 using Microsoft.ReverseProxy.Service.Proxy.Infrastructure;
 using Microsoft.ReverseProxy.Service.Routing;
 using Microsoft.ReverseProxy.Service.SessionAffinity;
@@ -48,12 +48,27 @@ public static IReverseProxyBuilder AddProxy(this IReverseProxyBuilder builder)
         {
             builder.Services.TryAddSingleton<ITransformBuilder, TransformBuilder>();
             builder.Services.TryAddSingleton<IProxyHttpClientFactory, ProxyHttpClientFactory>();
-            builder.Services.TryAddSingleton<ILoadBalancer, LoadBalancer>();
             builder.Services.TryAddSingleton<IRandomFactory, RandomFactory>();
+
             builder.Services.AddHttpProxy();
             return builder;
         }
 
+        public static IReverseProxyBuilder AddLoadBalancingPolicies(this IReverseProxyBuilder builder)
+        {
+            builder.Services.TryAddSingleton<IRandomFactory, RandomFactory>();
+
+            builder.Services.TryAddEnumerable(new[] {
+                new ServiceDescriptor(typeof(ILoadBalancingPolicy), typeof(FirstLoadBalancingPolicy), ServiceLifetime.Singleton),
+                new ServiceDescriptor(typeof(ILoadBalancingPolicy), typeof(LeastRequestsLoadBalancingPolicy), ServiceLifetime.Singleton),
+                new ServiceDescriptor(typeof(ILoadBalancingPolicy), typeof(RandomLoadBalancingPolicy), ServiceLifetime.Singleton),
+                new ServiceDescriptor(typeof(ILoadBalancingPolicy), typeof(PowerOfTwoChoicesLoadBalancingPolicy), ServiceLifetime.Singleton),
+                new ServiceDescriptor(typeof(ILoadBalancingPolicy), typeof(RoundRobinLoadBalancingPolicy), ServiceLifetime.Singleton)
+            });
+
+            return builder;
+        }
+
         public static IReverseProxyBuilder AddSessionAffinityProvider(this IReverseProxyBuilder builder)
         {
             builder.Services.TryAddEnumerable(new[] {

src/ReverseProxy/Configuration/DependencyInjection/ReverseProxyServiceCollectionExtensions.cs

@@ -42,6 +42,7 @@ public static IReverseProxyBuilder AddReverseProxy(this IServiceCollection servi
                 .AddSessionAffinityProvider()
                 .AddActiveHealthChecks()
                 .AddPassiveHealthCheck()
+                .AddLoadBalancingPolicies()
                 .AddProxy();
 
             services.AddDataProtection();