work in progress for #173, documentation changes for kubernetes deplo…

…yment
idaholab · Apr 24, 2023 · f4ae512 · f4ae512
1 parent c40976a
commit f4ae512
Showing 1 changed file with 169 additions and 1 deletion.
diff --git a/docs/kubernetes.md b/docs/kubernetes.md
@@ -1,17 +1,185 @@
 # <a name="Kubernetes"></a>Deploying Malcolm with Kubernetes
 
 * [Deploying Malcolm with Kubernetes](#Kubernetes)
+    - [System](#System)
+        + [Ingress Controller](#Ingress)
     - [Configuration](#Config)
         + [PersistentVolumeClaim definitions](#PVC)
 * [Future Enhancements](#Future)
     - [Live Traffic Analysis](#FutureLiveCap)
     - [Horizontal Scaling](#FutureScaleOut)
     - [Helm Chart](#FutureHelmChart)
 
-Malcolm can be
+## <a name="System"></a> System
+
+### <a name="Ingress"></a> Ingress Controller
+
+Malcolm's [ingress controller manifest]({{ site.github.repository_url }}/blob/{{ site.github.build_revision }}/kubernetes/00-ingress.yml) uses the [Ingress-NGINX controller for Kubernetes](https://github.com/kubernetes/ingress-nginx). A few Malcolm features require some customization when installing and configuring the Ingress-NGINX controller:
+
+* To [forward](malcolm-hedgehog-e2e-iso-install.md#HedgehogConfigForwarding) logs from a remote instance of [Hedgehog Linux](hedgehog.md):
+    - See ["Exposing TCP and UDP services"](https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/) in the Ingress-NGINX documentation.
+    - You must configure the controller to start up with the `--tcp-services-configmap=ingress-nginx/tcp-services` flag:
+```yml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+…
+  name: ingress-nginx-controller
+  namespace: ingress-nginx
+spec:
+…
+  template:
+…
+    spec:
+      containers:
+        * args:
+            * /nginx-ingress-controller
+            * --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
+            * --election-id=ingress-nginx-leader
+            * --controller-class=k8s.io/ingress-nginx
+            * --ingress-class=nginx
+            * --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
+            * --validating-webhook=:8443
+            * --validating-webhook-certificate=/usr/local/certificates/cert
+            * --validating-webhook-key=/usr/local/certificates/key
+            * --enable-ssl-passthrough
+            * --tcp-services-configmap=ingress-nginx/tcp-services
+…
+```
+    - You must add the appropriate ports (minimally TCP ports 5044 and 9200) to the `ingress-nginx-controller` load-balancer service definition:
+```yml
+---
+apiVersion: v1
+kind: Service
+metadata:
+…
+  name: ingress-nginx-controller
+  namespace: ingress-nginx
+spec:
+  externalTrafficPolicy: Local
+  ipFamilies:
+    + IPv4
+  ipFamilyPolicy: SingleStack
+  ports:
+    + appProtocol: http
+      name: http
+      port: 80
+      protocol: TCP
+      targetPort: http
+    + appProtocol: https
+      name: https
+      port: 443
+      protocol: TCP
+      targetPort: https
+    + appProtocol: tcp
+      name: lumberjack
+      port: 5044
+      targetPort: 5044
+      protocol: TCP
+    + appProtocol: tcp
+      name: tcpjson
+      port: 5045
+      targetPort: 5045
+      protocol: TCP
++ appProtocol: tcp
+      name: opensearch
+      port: 9200
+      targetPort: 9200
+      protocol: TCP
+…
+  type: LoadBalancer
+```
+    - You must add the appropriate ports (minimally TCP ports 5044 and 9200) to the `ingress-nginx-controller` deployment container's definition:
+```yml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+…
+  name: ingress-nginx-controller
+  namespace: ingress-nginx
+spec:
+…
+  template:
+…
+    spec:
+      containers:
+…
+          ports:
+            - containerPort: 80
+              name: http
+              protocol: TCP
+            - containerPort: 443
+              name: https
+              protocol: TCP
+            - containerPort: 8443
+              name: webhook
+              protocol: TCP
+            - name: lumberjack
+              containerPort: 5044
+              protocol: TCP
+            - name: tcpjson
+              containerPort: 5045
+              protocol: TCP
+            - name: opensearch
+              containerPort: 9200
+              protocol: TCP
+…
+```
+* To use [SSL Passthrough](https://kubernetes.github.io/ingress-nginx/user-guide/tls/) to have the Kubernetes gateway use Malcolm's TLS certificates rather than its own:
+    - You must configure the controller to start up with the `--enable-ssl-passthrough` flag.
+```
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+…
+  name: ingress-nginx-controller
+  namespace: ingress-nginx
+spec:
+…
+  template:
+…
+    spec:
+      containers:
+        - args:
+            - /nginx-ingress-controller
+            - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
+            - --election-id=ingress-nginx-leader
+            - --controller-class=k8s.io/ingress-nginx
+            - --ingress-class=nginx
+            - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
+            - --validating-webhook=:8443
+            - --validating-webhook-certificate=/usr/local/certificates/cert
+            - --validating-webhook-key=/usr/local/certificates/key
+            - --enable-ssl-passthrough
+            - --tcp-services-configmap=ingress-nginx/tcp-services
+…
+```
+    - You must modify Malcolm's [ingress controller manifest]({{ site.github.repository_url }}/blob/{{ site.github.build_revision }}/kubernetes/00-ingress.yml) to specify the `host:` value and use [host-based routing](https://kubernetes.github.io/ingress-nginx/user-guide/basic-usage/):
+```
+…
+spec:
+  rules:
+  - host: malcolm.example.org
+    http:
+      paths:
+      - path: /
+        pathType: Prefix
+        backend:
+          service:
+            name: nginx-proxy
+            port:
+              number: 443
+…
+```
+
+### <a name="Limits"></a> System Limits
 
 ## <a name="Config"></a> Configuration
 
+The steps to configure and tune Malcolm for a Kubernetes deployment are [very similar](malcolm-config.md#ConfigAndTuning) to those for a Docker-based deployment. Both methods use [environment variable files](malcolm-config.md#MalcolmConfigEnvVars) for Malcolm's runtime configuration.
+
+Malcolm's configuration and runtime scripts (e.g., `./scripts/configure`, `./scripts/auth_setup`, `./scripts/start`, etc.) are used for both Docker- and Kubernetes-based deployments. To indicate to these scripts that you're working with Kubernetes rather than `docker-compose`, provide the script with the [kubeconfig file](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/) used to communicate with the API server of the Kubernetes cluster (e.g., `./scripts/configure -f k3s.yaml` or `./scripts/start -f kubeconfig.yaml`, etc.). The scripts will detect whether YAML file specified is a kubeconfig file or a Docker compose file and act accordingly
+
 ### <a name="PVC"></a> PersistentVolumeClaim definitions
 
 ## <a name="Running"></a> Running Malcolm