Operator sidecar (#838)

This adds information about Operator component changes in v5.0.x.

Closes #791 

Staged:
http://192.241.195.202:9000/staging/operator-sidecar/operations/installation.html

source/includes/k8s/deploy-operator.rst

@@ -14,43 +14,62 @@ Deploy the MinIO Operator
 Overview
 --------
 
-MinIO is a Kubernetes-native high performance object store with an S3-compatible
-API. The MinIO Kubernetes Operator supports deploying MinIO Tenants onto private
-and public cloud infrastructures ("Hybrid" Cloud).
-
-The following procedure installs the latest stable version
-(|operator-version-stable|) of the MinIO Operator and MinIO Plugin on Kubernetes
-infrastructure:
-
-- The MinIO Operator installs a :kube-docs:`Custom Resource Document (CRD)
-  <concepts/extend-kubernetes/api-extension/custom-resources/#customresourcedefinitions>`
-  to support describing MinIO tenants as a Kubernetes :kube-docs:`object
-  <concepts/overview/working-with-objects/kubernetes-objects/>`. See the MinIO
-  Operator :minio-git:`CRD Reference <operator/blob/master/docs/tenant_crd.adoc>` for
-  complete documentation on the MinIO CRD.
-
-- The MinIO Kubernetes Plugin brings native support for deploying and managing
-  MinIO tenants on a Kubernetes cluster using the :mc:`kubectl minio` command. 
-
-This documentation assumes familiarity with all referenced Kubernetes
-concepts, utilities, and procedures. While this documentation *may* 
-provide guidance for configuring or deploying Kubernetes-related resources 
-on a best-effort basis, it is not a replacement for the official
-:kube-docs:`Kubernetes Documentation <>`.
+MinIO is a Kubernetes-native high performance object store with an S3-compatible API. 
+The MinIO Kubernetes Operator supports deploying MinIO Tenants onto private and public cloud infrastructures ("Hybrid" Cloud).
+
+The following procedure installs the latest stable version (|operator-version-stable|) of the MinIO Operator and MinIO Plugin on Kubernetes infrastructure:
+
+- The MinIO Operator installs a :kube-docs:`Custom Resource Definition (CRD) <concepts/extend-kubernetes/api-extension/custom-resources/#customresourcedefinitions>` to support describing MinIO tenants as a Kubernetes :kube-docs:`object <concepts/overview/working-with-objects/kubernetes-objects/>`. 
+  See the MinIO Operator :minio-git:`CRD Reference <operator/blob/master/docs/tenant_crd.adoc>` for complete documentation on the MinIO CRD.
+
+- The MinIO Kubernetes Plugin brings native support for deploying and managing MinIO tenants on a Kubernetes cluster using the :mc:`kubectl minio` command. 
+
+This documentation assumes familiarity with referenced Kubernetes concepts, utilities, and procedures. 
+While this documentation *may* provide guidance for configuring or deploying Kubernetes-related resources on a best-effort basis, it is not a replacement for the official :kube-docs:`Kubernetes Documentation <>`.
+
+MinIO Operator Components
+-------------------------
+
+The MinIO Operator exists in its own namespace.
+
+Within the Operator's namespace, the MinIO Operator utilizes two pods:
+- The Operator pod for the base Operator functions to deploy, manage, modify, and maintain tenants.
+- Console pod for the Operator's Graphical User Interface, the Operator Console.
+
+When you use the Operator to create a tenant, the tenant *must* have its own namespace.
+Within that namespace, the Operator generates the pods required by the tenant configuration.
+
+Each pod runs three containers:
+
+- MinIO Container that runs all of the standard MinIO functions, equivalent to basic MinIO installation on baremetal.
+  This container stores and retrieves objects in the provided mount points (persistent volumes). 
+
+- InitContainer that only exists during the launch of the pod to manage configuration secrets during startup.
+  Once startup completes, this container terminates. 
+
+- SideCar container that monitors configuration secrets for the tenant and updates them as they change.
+  This container also monitors for root credentials and creates an error if it does not find root credentials. 
+
+The tenant utilizes Persistent Volume Claims to talk to the Persistent Volumes that store the objects.
+
+.. image:: /images/k8s/OperatorsComponent-Diagram.png
+   :width: 600px
+   :alt: A diagram of the namespaces and pods used by or maintained by the MinIO Operator.
+   :align: center
 
 Prerequisites
 -------------
 
 Kubernetes Version 1.19.0
 ~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Starting with v4.0.0, the MinIO Operator and MinIO Kubernetes Plugin require
-Kubernetes 1.19.0 and later. The Kubernetes infrastructure *and* the 
-``kubectl`` CLI tool must have the same version of 1.19.0+.
+Starting with v4.0.0, the MinIO Operator and MinIO Kubernetes Plugin **require** Kubernetes 1.19.0 and later. 
+The Kubernetes infrastructure *and* the ``kubectl`` CLI tool must have the same version of 1.19.0+.
+
+Prior to v4.0.0, the MinIO Operator and Plugin required Kubernetes 1.17.0. 
+You *must* upgrade your Kubernetes infrastructure to 1.19.0 or later to use the MinIO Operator or Plugin v4.0.0 or later.
 
-Prior to v4.0.0, the MinIO Operator and Plugin required Kubernetes 1.17.0. You 
-*must* upgrade your Kubernetes infrastructure to 1.19.0 or later to use 
-the MinIO Operator or Plugin v4.0.0 or later.
+Starting with v5.0.0, MinIO *recommends* Kubernetes 1.21.0 or later for both the infrastructure and the ``kubectl`` CLI tool.
 
 ``kubectl`` Configuration
 ~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -62,9 +81,18 @@ This procedure assumes that your local host machine has both the correct version
 Kubernetes TLS Certificate API
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-The MinIO Operator automatically generates TLS Certificate Signing Requests (CSR) and uses the Kubernetes ``certificates.k8s.io`` :kube-docs:`TLS certificate management API <tasks/tls/managing-tls-in-a-cluster/>` to create signed TLS certificates.
+.. versionchanged:: Operator v.5.0.0
 
-The MinIO Operator therefore *requires* that the Kubernetes ``kube-controller-manager`` configuration include the following :kube-docs:`configuration settings <reference/command-line-tools-reference/kube-controller-manager/#options>`:
+   The MinIO Operator manages TLS Certificate Signing Requests (CSR) using the Kubernetes ``certificates.k8s.io`` :kube-docs:`TLS certificate management API <tasks/tls/managing-tls-in-a-cluster/>` to create signed TLS certificates in the following circumstances:
+
+   - When ``autoCert`` is enabled.
+   - For the MinIO Console when the :envvar:`OPERATOR_CONSOLE_TLS_ENABLE` environment variable is set to ``on``.
+   - For :ref:`STS service <minio-security-token-service>` when :envvar:`OPERATOR_STS_ENABLED` environment variable is set to ``on``.
+   - For retrieving the health of the cluster.
+
+   The MinIO Operator reads certificates inside the ``operator-ca-tls`` secret and syncs this secret within the tenant namespace to trust private certificate authorities, such as when using cert-manager.
+
+For any of these circumstances, the MinIO Operator *requires* that the Kubernetes ``kube-controller-manager`` configuration include the following :kube-docs:`configuration settings <reference/command-line-tools-reference/kube-controller-manager/#options>`:
 
 - ``--cluster-signing-key-file`` - Specify the PEM-encoded RSA or ECDSA private key used to sign cluster-scoped certificates.
 
@@ -74,22 +102,21 @@ The Kubernetes TLS API uses the CA signature algorithm for generating new TLS ce
 MinIO recommends ECDSA (e.g. `NIST P-256 curve <https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.186-4.pdf>`__) or EdDSA (e.g. :rfc:`Curve25519 <7748>`) TLS private keys/certificates due to their lower computation requirements compared to RSA.
 See :ref:`minio-TLS-supported-cipher-suites` for a complete list of supported TLS Cipher Suites.
 
-The Operator cannot complete initialization if the Kubernetes cluster is 
-not configured to respond to a generated CSR. Certain Kubernetes 
-providers do not specify these configuration values by default. 
+If the Kubernetes cluster is not configured to respond to a generated :abbr:`CSR (Certificate Signing Request)`, the Operator cannot complete initialization. 
+Some Kubernetes providers do not specify these configuration values by default. 
 
-To verify whether the ``kube-controller-manager`` has the required 
-settings, use the following command. Replace ``$CLUSTER-NAME`` with the name 
-of the Kubernetes cluster:
+To check whether the ``kube-controller-manager`` specifies the cluster signing key and certificate files, use the following command:
 
 .. code-block:: shell
    :class: copyable
 
    kubectl get pod kube-controller-manager-$CLUSTERNAME-control-plane \ 
      -n kube-system -o yaml
 
-Confirm that the output contains the highlighted lines. The output of 
-the example command above may differ from the output in your terminal:
+- Replace ``$CLUSTERNAME`` with the name of the Kubernetes cluster.
+
+Confirm that the output contains the highlighted lines. 
+The output of the example command above may differ from the output in your terminal:
 
 .. code-block:: shell
    :emphasize-lines: 12,13
@@ -111,18 +138,60 @@ the example command above may differ from the output in your terminal:
 
 .. important::
 
-   The MinIO Operator automatically generates TLS certificates for all 
-   MinIO Tenant pods using the specified Certificate Authority (CA).
-   Clients external to the Kubernetes cluster must trust the  
-   Kubernetes cluster CA to connect to the MinIO Operator or MinIO Tenants. 
+   The MinIO Operator automatically generates TLS certificates for all MinIO Tenant pods using the specified Certificate Authority (CA).
+   Clients external to the Kubernetes cluster must trust the Kubernetes cluster CA to connect to the MinIO Operator or MinIO Tenants. 
 
-   Clients which cannot trust the Kubernetes cluster CA can try disabling TLS 
-   validation for connections to the MinIO Operator or a MinIO Tenant. 
+   Clients which cannot trust the Kubernetes cluster CA can disable TLS validation for connections to the MinIO Operator or a MinIO Tenant. 
 
-   Alternatively, you can generate x.509 TLS certificates signed by a known
-   and trusted CA and pass those certificates to MinIO Tenants. 
+   Alternatively, you can generate x.509 TLS certificates signed by a known and trusted CA and pass those certificates to MinIO Tenants. 
    See :ref:`minio-tls` for more complete documentation.
 
+Configure MinIO Operator to Trust Custom Certificates
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you use custom certificates for your deployment, add the certificate so that MinIO Operator trusts it.
+
+This procedure assumes you have an existing custom certificate.
+
+1. Use the following command to generate a secret from the certificate:
+
+   .. code-block:: shell
+      :class: copyable
+   
+      kubectl create secret generic MY-CUSTOM-TLS -n MY-CLUSTER-NAMESPACE --from-file=<path/to/public.crt>
+   
+   Replace the following placeholders in the above command:
+   - ``MY-CUSTOM-TLS`` with the name of your secrets file
+   - ``MY-CLUSTER-NAMESPACE`` with your cluster's namespace
+   - ``<path/to/public.crt>`` with the relative path to the public certificate to use to create the secret
+
+2. Add a volume to the yaml for your cluster under ``.spec.template.spec``
+
+   .. code-block:: yaml
+      :class: copyable
+
+      volumes:
+        - name: tls-certificates
+          projected:
+            defaultMode: 420
+            sources:
+              - secret:
+                  items:
+                    - key: public.crt
+                      path: CAs/custom-public.crt
+                  name: MY-CUSTOM-TLS
+
+   - replace ``MY-CUSTOM-TLS`` with the name of your secrets file.
+
+3. Add a ``volumeMount`` to the yaml for your cluster under ``.spec.template.spec.container[0]``
+
+   .. code-block:: yaml
+      :class: copyable
+
+      volumeMounts:
+        - mountPath: /tmp/certs
+          name: tls-certificates
+
 Procedure
 ---------
 

source/includes/openshift/deploy-minio-on-openshift.rst

@@ -176,10 +176,9 @@ The Operator Console still requires using the generated JWT token for access, wh
 After deploying the MinIO Operator, you can create a new MinIO Tenant.
 To deploy a MinIO Tenant using OpenShift, see :ref:`deploy-minio-tenant-redhat-openshift`.
 
-.. uncomment this block when upgrade operator procedure is complete
 
-   .. toctree::
-      :titlesonly:
-      :hidden:
+.. toctree::
+   :titlesonly:
+   :hidden:
 
-      /operations/install-deploy-manage/upgrade-minio-operator
+   /operations/install-deploy-manage/upgrade-minio-operator

source/operations/install-deploy-manage/minio-operator-console.rst

@@ -43,7 +43,12 @@ The MinIO Operator Console supports deploying, managing, and monitoring MinIO Te
 
 You can :ref:`deploy a MinIO Tenant <minio-k8s-deploy-minio-tenant>` through the Operator Console.
 
-The Operator Console automatically detects MinIO Tenants deployed on the cluster when provisioned through the Operator Console, the :ref:`MinIO Kubernetes Plugin <minio-k8s-deploy-minio-tenant-commandline>`, Helm, or Kustomize.
+The Operator Console automatically detects MinIO Tenants deployed on the cluster when provisioned through:
+
+- Operator Console
+- :ref:`MinIO Kubernetes Plugin <minio-k8s-deploy-minio-tenant-commandline>`
+- Helm
+- Kustomize
 
 Select a listed tenant to open an in-browser view of that tenant's MinIO Console. 
 You can use this view to directly manage, modify, expand, upgrade, and delete the tenant through the Operator UI.
@@ -56,12 +61,12 @@ You can use this view to directly manage, modify, expand, upgrade, and delete th
    The ZIP archive contains status, events, and log information for each pool on the deployment.
    The archive also includes a summary yaml file describing the deployment.
 
-   |subnet| users can upload this file for analysis by MinIO Engineers.
+   |subnet| users relying on the commercial license should register the MinIO tenants to their SUBNET account, which can be done through the Operator Console.
 
 Tenant Registration
 -------------------
 
-|subnet| users relying on the commercial license should register your MinIO tenants to your SUBNET account, which can be done through the Operator Console.
+|subnet| users relying on the commercial license should register the MinIO tenants to their SUBNET account, which can be done through the Operator Console.
 
 .. image:: /images/k8s/operator-console-register.png
    :align: center

source/operations/install-deploy-manage/upgrade-minio-operator.rst

@@ -65,7 +65,7 @@ The following changes apply for Operator v5.0.0 or later:
   This change impacts the Tenant :abbr:`CRD <CustomResourceDefinition>` and only impacts users editing a tenant YAML directly, such as through Helm or Kustomize.
 - Both the **Log Search API** (``.spec.log``) and **Prometheus** (``.spec.prometheus``) deployments have been removed.
   However, existing deployments are left running as standalone deployments / statefulsets with no connection to the Tenant CR.
-  If the Tenant CRD is deleted, this does not cascade to the log or Prometheus deployments.
+  Deleting the Tenant :abbr:`CRD (Custom Resource Definition)` does **not** cascade to the log or Prometheus deployments.
 
   .. important::
 
@@ -170,7 +170,7 @@ Upgrade Operator to |operator-version-stable|
          "name": "minio-operator"
       }
 
-3. Download the Latest Stable Version of the MinIO Kubernetes Plugin
+3. Download the latest stable version of the MinIO Kubernetes Plugin
 
    .. include:: /includes/k8s/install-minio-kubectl-plugin.rst
 

source/reference/kubectl-minio-plugin.rst

@@ -93,7 +93,30 @@ Environment Variables
 ---------------------
 
 The :ref:`MinIO Operator <minio-operator-installation>` uses the following environment variables during startup to set configuration settings.
-Configure these variables on the machine running the kubectl plugin.
+Configure these variables in the ``minio-operator`` container.
+
+Setting Environment Variables in Kubernetes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To set these environment variables, modify the operator container's yaml at ``.spec.env`` or use the following ``kubectl`` command syntax:
+
+.. code-block:: shell
+   :class: copyable
+
+   kubectl set env -n minio-operator deployment/minio-operator <ENV_VARIABLE>=<value> ... <ENV_VARIABLE2>=<value2>
+
+Replace:
+
+- ``minio-operator`` with the namespace for your Operator, if not using the default value.
+- ``deployment/minio-operator`` with the deployment for your Operator, if not the default value.
+  (This is not common.)
+- ``<ENV_VARIABLE>`` with the environment variable to set or modify.
+- ``<value>`` with the value to use for the environment variable.
+
+You can set or modify multiple environment variables by separating each ``VARIABLE=value`` pair with a space.
+
+Available MinIO Operator Environment Variables
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 .. envvar:: MINIO_OPERATOR_CERTIFICATES_VERSION
 
@@ -126,7 +149,7 @@ Configure these variables on the machine running the kubectl plugin.
 
 .. envvar:: MINIO_OPERATOR_DEPLOYMENT_NAME
 
-   Specifies the namespace to create and use for Operator
+   Specifies the namespace to create and use for Operator.
 
    When not specified, the default value is ``minio-operator``.