Kubernetes hosting integration

[ReubenBond]

This PR adds experimental support for integrating more deeply with Kubernetes.

Users make a call to siloBuilder.UseKubernetesHosting() to enable the following:

• Early in the startup process and at runtime, the silo will probe Kubernetes to find which silos do not have corresponding pods and mark those silos as dead
• Configure ClusterOptions.ServiceId and ClusterOptions.ClusterId based on labels set on the pod in Kubernetes
• Configure SiloOptions.SiloName based on the pod name
• Configure EndpointOptions.AdvertisedIPAddress, EndpointOptions.SiloListeningEndpoint, and EndpointOptions.GatewayListeningEndpoint based upon the pod's PodIP and the configured SiloPort and GatewayPort (default values 11111, 30000)

In a future update, we could consider optionally instructing Kubernetes to delete pods which correspond to dead silos (eg, because the pod is not responding - perhaps the process has become a zombie).

To reduce load on Kubernetes' API server (which is apparently a big issue), only a subset of silos will monitor Kubernetes. The default value is 2 silos per cluster, and to reduce churn, the two oldest silos in the cluster are chosen.

There are some requirements on how Orleans is deployed into Kubernetes when using this plugin. For example:

• Silo names must match pod names (the abovementioned method will configure that)
• Pods must have a serviceId and clusterId label which corresponds to the silo's ServiceId and ClusterId. The abovementioned method will propagate those labels into the corresponding options in Orleans from Env vars.
• Pods must have environment variables set for ServiceId, ClusterId, PodName, and PodNamespace (see example below)

Here is an example YAML file to deploy such a silo:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dictionary-app
  labels:
    serviceId: dictionary-app
spec:
  selector:
    matchLabels:
      serviceId: dictionary-app
  replicas: 3
  template:
    metadata:
      labels:
        # The serviceId label is used to identify the service to Orleans
        serviceId: dictionary-app

        # The clusterId label is used to identify an instance of a cluster to Orleans.
        # Typically, this will be the same value as serviceId or any fixed value.
        # In cases where you are not using rolling deployments (for example, blue/green deployments),
        # this value can allow for distinct clusters which do not communicate directly with each others,
        # but which still share the same storage and other resources.
        clusterId: dictionary-app
    spec:
      containers:
        - name: main
          image: reuben.azurecr.io/dictionary-app
          imagePullPolicy: Always
          ports:
          # Define the ports which Orleans uses
          - containerPort: 11111
          - containerPort: 30000
          env:
          # The Azure Storage connection string for clustering is injected as an environment variable
          # It must be created separately using a command such as:
          # > kubectl create secret generic az-storage-acct --from-file=key=./az-storage-acct.txt
          - name: STORAGE_CONNECTION_STRING
            valueFrom:
              secretKeyRef:
                name: az-storage-acct
                key: key
          # Configure settings to let Orleans know which cluster it belongs to and which pod it is running in
          - name: ORLEANS_SERVICE_ID
            valueFrom:
              fieldRef:
                fieldPath: metadata.labels['serviceId']
          - name: ORLEANS_CLUSTER_ID
            valueFrom:
              fieldRef:
                fieldPath: metadata.labels['clusterId']
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name: DOTNET_SHUTDOWNTIMEOUTSECONDS
            value: "120"
          request:
            # Set resource requests
      terminationGracePeriodSeconds: 180
      imagePullSecrets:
        - name: reuben-acr
  minReadySeconds: 60
  strategy:
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1

A minimal, do-nothing program then looks like this:

using Microsoft.Extensions.Hosting;
using Orleans.Hosting;
using System;

var storageConnectionString = Environment.GetEnvironmentVariable("STORAGE_CONNECTION_STRING");

await Host.CreateDefaultBuilder()
    .UseOrleans(siloBuilder =>
    {
        siloBuilder.UseKubernetesHosting();
        siloBuilder.UseAzureStorageClustering(options => options.ConnectionString = storageConnectionString);
    })
    .RunConsoleAsync();

[johnazariah]

Love this! :D

[ReubenBond]

TODO: annotations instead of labels, include sample

[benjaminpetit]

This will be a breaking change for 3.3, no?

[ReubenBond]

Shouldn't be, since this isn't sent over the wire

[benjaminpetit]

Gotcha, you get the name from MembershipEntry

[aelij]

Might also want to mention that a role binding is required for this on RBAC-enabled clusters, e.g.

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: pod-reader
rules:
- apiGroups: [ "" ]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: pod-reader-binding
subjects:
- kind: ServiceAccount
  name: default
  apiGroup: ''
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: ''

[NikolayBieliashov]

That works for me in this case!