From bdd1fd05376b60c5d9ff3d8e3425f597ca2acded Mon Sep 17 00:00:00 2001 From: Daryl White <53910321+djwfyi@users.noreply.github.com> Date: Tue, 13 Aug 2024 14:06:39 -0400 Subject: [PATCH 01/11] Starting to add cert manager info --- source/operations/cert-manager.rst | 424 +++++++++++++++++++++++++++++ 1 file changed, 424 insertions(+) create mode 100644 source/operations/cert-manager.rst diff --git a/source/operations/cert-manager.rst b/source/operations/cert-manager.rst new file mode 100644 index 00000000..0ca0721d --- /dev/null +++ b/source/operations/cert-manager.rst @@ -0,0 +1,424 @@ +.. _minio-certmanager: + +============ +Cert Manager +============ + +.. default-domain:: minio + +.. contents:: Table of Contents + :local: + :depth: 1 + +TLS certificate management with cert-manager +-------------------------------------------- + +`cert-manager `__ works as a controller for managing certificates within Kubernetes clusters. +cert-manager obtains valid certificates from an ``Issuer`` or ``ClusterIssuer`` and can renew certificates prior to expiration. + +The MinIO Operator supports using cert-manager for certificates as an alternative to the MinIO Operator managing certificates for itself and its tenants. + +On a Kubernetes cluster, cert-manager requires a global level ``ClusterIssuer`` to generate intermediary ``Issuers`` and certificates. +At the namespace level, cert-manager issues certificates derived from an ``Issuer``. +A ``ClusterIssuer`` can issue certificates for multiple namespaces. +An ``Issuer`` only issues certificates for its own namespace. + +To learn more about cert-manager Issuers, refer to their `Issuer documentation `__. + +Below is a logical view of a Kubernetes cluster: + +![Cert-manager-namespaces.png](images%2FCert-manager-namespaces.png) + +This cluster contains 4 namespaces: + +- minio-operator +- tenant-1 +- tenant-2 +- other-namespace + +This guide shows you how to set up different Certificate Authorities (CA) in each namespace, all of them referencing a global `Cluster Issuer`. + +![Cert-manager Issuers.png](images%2FCert-manager%20Issuers.png) + +Cert Manager is installed in the `cert-manager` namespace. + +The global `Cluster Issuer` is created in the default namespace. + +A local `Issuer` is created in each tenant namespace. + +An `Issuer` is also created in the `minio-operator` namespace. More about services that require TLS certificates +in the `minio-operator` namespace are covered below in [MinIO Operator services with cert-manager](#minio-operator-services-with-cert-manager).. + +> [!NOTE] +> This guide uses a self-signed `Cluster Issuer`. You can also use [other Issuers supported by Cert Manager](https://cert-manager.io/docs/configuration/issuers/). +> The main difference is that you must provide the `Issuer` CA certificate to minio, instead of the CA's mentioned in this guide. + +## Getting Started + +### Prerequisites + +- Kubernetes version `+v1.21`. While cert-manager + supports [earlier K8s versions](https://cert-manager.io/docs/installation/supported-releases/), MinIO Operator requires 1.21 or later. +- [kustomize](https://kustomize.io/) installed +- `kubectl` access to your `k8s` cluster + +### Setup Cert-manager + +Install [cert-manager](https://cert-manager.io/docs/releases/release-notes/release-notes-1.12/) 1.12.X LTS is preferred, +or install latest, for example using kubectl. + +```bash +kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.12.9/cert-manager.yaml +``` + +For more details on the Cert Manager refer to https://cert-manager.io/docs/installation/. + +### Create Cluster Self-signed root Issuer + +The `Cluster Issuer` is the cluster level Issuer all other certificates are derived from. Request Cert Manager to +generate this by creating a `ClusterIssuer` resource: + +```yaml +# selfsigned-root-clusterissuer.yaml +apiVersion: cert-manager.io/v1 +kind: ClusterIssuer +metadata: + name: selfsigned-root +spec: + selfSigned: {} +``` + +```shell +kubectl apply -f selfsigned-root-clusterissuer.yaml +``` + +# MinIO Operator services with cert-manager + +MinIO Operator manages the TLS certificate issuing for the services hosted in the `minio-operator` namespace. That is the Secure Token Service `sts`. + +This section describes how to generate the `sts` TLS certificate with Cert Manager. +These certificates must be issued before installing Operator. +Be sure to follow step [Create Cluster Self-signed root Issuer](#create-cluster-self-signed-root-issuer) mentioned above. + +## Secure Token Service (STS) + +MinIO STS is a service included with MinIO Operator that provides Native IAM Authentication for Kubernetes. In essence +this service allows you to control access to your MinIO tenant from your kubernetes applications without having to explicitly create credentials +for each application. For more information on the Service see the MinIO docs at https://min.io/docs/minio/kubernetes/upstream/developers/sts-for-operator.html. +There is also an [STS](https://github.com/minio/operator/blob/master/docs/STS.md) guide in the docs and example client code in +https://github.com/minio/operator/tree/master/examples/kustomization/sts-example. + +For the purpose of this guide, STS Service can be considered a webserver presented with a TLS certificate for https traffic. +This guide covers how to **disable** the automatic generation of the certificate in MinIO Operator and issue the certificate using +Cert Manager instead. + +### Create minio-operator namespace CA Issuer + +To create the Certificate Authority (CA) certificate used to issue certificates for services in the `minio-operator` +namespace, first create the minio-operator namespace + +```shell +kubectl create ns minio-operator +``` + +Request a Certificate with `spec.isCA: true` specified. This is our CA for the `minio-operator` namespace. + +```yaml +# operator-ca-tls-secret.yaml +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: minio-operator-ca-certificate + namespace: minio-operator +spec: + isCA: true + commonName: operator + secretName: operator-ca-tls + duration: 70128h # 8y + privateKey: + algorithm: ECDSA + size: 256 + issuerRef: + name: selfsigned-root + kind: ClusterIssuer + group: cert-manager.io +``` + +```shell +kubectl apply -f operator-ca-tls-secret.yaml +``` +A new secret with the name `operator-ca-tls` is created in the `minio-operator` namespace, this is the CA issuing TLS certificates for the services in the `minio-operator` namespace. + +> [!IMPORTANT] +> Make sure to trust this certificate in your applications that need to interact with the `sts` service. + + +Now create the `Issuer`: + +```yaml +# operator-ca-issuer.yaml +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: minio-operator-ca-issuer + namespace: minio-operator +spec: + ca: + secretName: operator-ca-tls +``` + +```shell +kubectl apply -f operator-ca-issuer.yaml +``` + +### Create TLS certificate for STS service + +Request Cert Manager to issue a new certificate containing following DNS domains: + +```shell +sts +sts.minio-operator.svc. +sts.minio-operator.svc. +``` + +> [!IMPORTANT] +> Replace `` with the actual values for your MinIO tenant. +> `cluster domain` is the internal root DNS domain assigned in your Kubernetes cluster. Typically this is `cluster.local`, check on your coredns +> configuration for the correct value for your Kubernetes cluster. For example, using `kubectl get configmap coredns -n kube-system -o jsonpath="{.data}"`. +> The way the root DNS domain is managed can vary depending on the Kubernetes distribution (Openshift, Rancher, EKS, etc.) + +Create a `Certificate` for the domains mentioned above: + +```yaml +# sts-tls-certificate.yaml +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: sts-certmanager-cert + namespace: minio-operator +spec: + dnsNames: + - sts + - sts.minio-operator.svc + - sts.minio-operator.svc.cluster.local + secretName: sts-tls + issuerRef: + name: minio-operator-ca-issuer +``` + +```shell +kubectl apply -f sts-tls-certificate.yaml +``` +This creates a secret called `sts-tls` in the `minio-operator` namespace. + +> [!IMPORTANT] +> The secret name is not optional. Make sure the secret name is `sts-tls` by setting `spec.secretName: sts-tls` as in the example above. + +### Install Operator with Auto TLS disabled for STS + +When installing the Operator deployment, make sure to set `OPERATOR_STS_AUTO_TLS_ENABLED: off` env variable in the `minio-operator` container. This prevents +MinIO Operator from issuing the certificate for STS and instead wait for you to provide the TLS certificate issued by Cert Manager. + +> [!WARNING] +> Missing to provide the secret `sts-tls` containing the TLS certificate or providing an invalid key-pair in the secret will +> prevent the STS service from start. + +A way to make sure that the env variables are properly set is using kustomization to patch the `minio-operator` deployment: + +```yaml +# minio-operator/kustomization.yaml +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: +- github.com/minio/operator/resources + +patches: +- patch: |- + apiVersion: apps/v1 + kind: Deployment + metadata: + name: minio-operator + namespace: minio-operator + spec: + template: + spec: + containers: + - name: minio-operator + env: + - name: OPERATOR_STS_AUTO_TLS_ENABLED + value: "off" + - name: OPERATOR_STS_ENABLED + value: "on" +``` + +```shell +kubectl apply -k minio-operator +``` + +# MinIO tenant TLS certificates with cert-manager + +## Create tenant-1 namespace CA Issuer + +To create the Certificate Authority (CA) certificate in the namespace `tenant-1`, first create the namespace + +```shell +kubectl create ns tenant-1 +``` + +Next, request a Certificate with `spec.isCA: true` specified: + +```yaml +# tenant-1-ca-certificate.yaml +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: tenant-1-ca-certificate + namespace: tenant-1 +spec: + isCA: true + commonName: tenant-1-ca + secretName: tenant-1-ca-tls + duration: 70128h # 8y + privateKey: + algorithm: ECDSA + size: 256 + issuerRef: + name: selfsigned-root + kind: ClusterIssuer + group: cert-manager.io +``` + +```shell +kubectl apply -f tenant-1-ca-certificate.yaml +``` + + +Then create the `Issuer`: + +```yaml +# tenant-1-ca-issuer.yaml +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: tenant-1-ca-issuer + namespace: tenant-1 +spec: + ca: + secretName: tenant-1-ca-tls +``` + +```shell +kubectl apply -f tenant-1-ca-issuer.yaml +``` + +## Deploy the tenant + +### Create a certificate for Tenant + +Request Cert Manager issue a new TLS server certificate for MinIO that includes the following DNS domains: + +```shell +minio. +minio..svc +minio..svc. +*.-hl..svc. +*..svc. +*..minio..svc.' +``` + +> [!IMPORTANT] +> Replace `` with the actual values for your MinIO tenant. +> * `` is the internal root DNS domain assigned in your Kubernetes cluster. Typically this is `cluster.local`, check on your coredns +> configuration for the correct value for your Kubernetes cluster. For example, using `kubectl get configmap coredns -n kube-system -o jsonpath="{.data}"`. +> The way the root DNS domain is managed can vary depending on the Kubernetes distribution (Openshift, Rancher, EKS, etc.) +> * `tenant-name` is the name provided to your tenant in the `metadata.name` of the Tenant YAML. For this example it is `myminio`. +> * `namespace` is the namespace where the tenant is created, the `metadata.namespace` notes that in the Tenant YAML. For this example it is `tenant-1`. + +Create a `Certificate` for the domains mentioned above: + +```yaml +# tenant-1-minio-certificate.yaml +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: tenant-certmanager-cert + namespace: tenant-1 +spec: + dnsNames: + - "minio.tenant-1" + - "minio.tenant-1.svc" + - 'minio.tenant-1.svc.cluster.local' + - '*.minio.tenant-1.svc.cluster.local' + - '*.myminio-hl.tenant-1.svc.cluster.local' + - '*.myminio.minio.tenant-1.svc.cluster.local' + secretName: myminio-tls + issuerRef: + name: tenant-1-ca-issuer +``` + +> [!TIP] +> For this example, the Tenant name is `myminio`. We recommend naming the secret in the field `spec.secretName` as `-tls`, following the naming +> convention MinIO Operator uses when creates certificates with Autocert enabled (`spec.requestAutoCert: true`). + +```shell +kubectl apply -f tenant-1-minio-certificate.yaml +``` + +### Configure the tenant to use the certificate created by cert-manager + +In the tenant spec, do the following: + +* Disable Autocert `spec.requestAutoCert: false`. This instructs Operator to not attempt to issue certificates and instead rely on Cert Manager to provide them in a secret. +* Reference the Secret containing the TLS certificate from the previous step in `spec.externalCertSecret`. + +```yaml +apiVersion: minio.min.io/v2 +kind: Tenant +metadata: + name: myminio + namespace: tenant-1 +spec: +... + ## Disable default tls certificates. + requestAutoCert: false + ## Use certificates generated by cert-manager. + externalCertSecret: + - name: myminio-tls + type: cert-manager.io/v1 +... +``` + +## Trust tenant-1 CA in MinIO Operator + +MinIO Operator can trust as many CA certificates as provided. To do this, create a secret with the prefix `operator-ca-tls-` +followed by a unique identifier in the `minio-operator` namespace. + +MinIO Operator mounts and trusts all certificates issued by the provided CA's. This is required because +MinIO Operator performs health checks using the */minio/health/cluster* endpoint. + +If Operator is not correctly configured to trust the MinIO Certificate (or its CA), you will see an error message like the following in the Operator Pod logs: + +```error +Failed to get cluster health: Get "https://minio.tenant-1.svc.cluster.local/minio/health/cluster": +x509: certificate signed by unknown authority +``` +For more details about health checks, see https://min.io/docs/minio/Kubernetes/upstream/operations/monitoring/healthcheck-probe.html#cluster-write-quorum. + +### Create `operator-ca-tls-tenant-1` secret + +Copy the Cert Manager generated CA public key (ca.crt) into the `minio-operator` namespace. This allows Operator to trust +the cert-manager issued CA and the derived certificates. + +Create a `ca.crt` file containing the CA: +```sh +kubectl get secrets -n tenant-1 tenant-1-ca-tls -o=jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt +``` + +Create the secret: +```sh +kubectl create secret generic operator-ca-tls-tenant-1 --from-file=ca.crt -n minio-operator +``` +> [!TIP] +> In this example we choose a secret name of `operator-ca-tls-tenant-1`. Note the tenant namespace +> `tenant-1` is used as suffix for easy identification of which namespace the CA is coming from. \ No newline at end of file From a2f6475b6db5ca848857089d7c3d637f05fed0b9 Mon Sep 17 00:00:00 2001 From: Daryl White <53910321+djwfyi@users.noreply.github.com> Date: Wed, 11 Sep 2024 16:29:36 -0400 Subject: [PATCH 02/11] Adds cert-manager documentation for kubernetes outputs Closes #1245 Partially addresses #1273 --- source/images/k8s/cert-manager-cluster.svg | 373 ++++++++ source/images/k8s/cert-manager-namespaces.svg | 530 +++++++++++ .../k8s/file-transfer-protocol-k8s.rst | 2 +- source/index.rst | 1 + source/operations/cert-manager.rst | 881 ++++++++++-------- .../monitoring/healthcheck-probe.rst | 2 + source/operations/network-encryption.rst | 6 + 7 files changed, 1430 insertions(+), 365 deletions(-) create mode 100644 source/images/k8s/cert-manager-cluster.svg create mode 100644 source/images/k8s/cert-manager-namespaces.svg diff --git a/source/images/k8s/cert-manager-cluster.svg b/source/images/k8s/cert-manager-cluster.svg new file mode 100644 index 00000000..6c58eb1d --- /dev/null +++ b/source/images/k8s/cert-manager-cluster.svg @@ -0,0 +1,373 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/source/images/k8s/cert-manager-namespaces.svg b/source/images/k8s/cert-manager-namespaces.svg new file mode 100644 index 00000000..251e5bfb --- /dev/null +++ b/source/images/k8s/cert-manager-namespaces.svg @@ -0,0 +1,530 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/source/includes/k8s/file-transfer-protocol-k8s.rst b/source/includes/k8s/file-transfer-protocol-k8s.rst index 408db992..6096eebc 100644 --- a/source/includes/k8s/file-transfer-protocol-k8s.rst +++ b/source/includes/k8s/file-transfer-protocol-k8s.rst @@ -164,7 +164,7 @@ If SFTP is enabled, the output resembles the following: enableSFTP: true -.. _minio-certificate-key-file-sftp-k8s +.. _minio-certificate-key-file-sftp-k8s: Connect to MinIO Using SFTP with a Certificate Key File ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/source/index.rst b/source/index.rst index e3d376e4..4b970693 100644 --- a/source/index.rst +++ b/source/index.rst @@ -106,6 +106,7 @@ For more about connecting to ``play``, see :ref:`MinIO Console play Login `__. +To learn more about cert-manager Issuers, refer to the `Issuer documentation `__. -Below is a logical view of a Kubernetes cluster: +Below is a logical view of a Kubernetes cluster with four namespaces: -![Cert-manager-namespaces.png](images%2FCert-manager-namespaces.png) +- ``minio-operator`` +- ``tenant-1`` +- ``tenant-2`` +- ``other-namespace`` -This cluster contains 4 namespaces: +.. image:: /images/k8s/cert-manager-namespaces.svg + :width: 600px + :alt: A graphic depiction of a kubernetes cluster with four namespaces. Each namespace in its own white box. One is labeled minio-operator with contents two sts content items. A second is labeled tenant-1 with a MinIO logo and four drives of a pool. A third is labeled tenant-2 with similar contents to the tenant-1 box. The fourth is labeled other namespace with contents of other pods. + :align: center -- minio-operator -- tenant-1 -- tenant-2 -- other-namespace +This guide shows you how to set up a different Certificate Authority (CA) in each namespace. +Each namespace's CA references the global ``Cluster Issuer``. -This guide shows you how to set up different Certificate Authorities (CA) in each namespace, all of them referencing a global `Cluster Issuer`. -![Cert-manager Issuers.png](images%2FCert-manager%20Issuers.png) +The following graphic depicts how various namespaces make use of either an ``Issuer`` or ``ClusterIssuer`` type. -Cert Manager is installed in the `cert-manager` namespace. +- cert-manager is installed in the ``cert-manager`` namespace, which does not have either an ``issuer`` or a ``ClusterIssuer``. +- The ``default`` namespace receives the global ``Cluster Issuer``. +- Each tenant's namespace receives a local ``Issuer``. +- The ``minio-operator`` namespace receives a local ``Issuer``. + More about services that require TLS certificates in the ``minio-operator`` namespace are covered :ref:`below `. -The global `Cluster Issuer` is created in the default namespace. +.. image:: /images/k8s/cert-manager-cluster.svg + :width: 600px + :alt: A graphic depiction of a kubernetes cluster with five separate namespaces represented by different boxes. One is labeled minio-operator with a text box that says "minio-operator: issuer". A second is labeled tenant-1 with text box that says "tenant-1: issuer". A third is labeled default with a text box that says "root:ClusterIssuer". A fourth is labeled tenant-2 with a text box that says "tenant-2: issuer". The fifth is labeled cert-manager with a text box that says "minio-operator: issuer". + :align: center -A local `Issuer` is created in each tenant namespace. +.. note:: -An `Issuer` is also created in the `minio-operator` namespace. More about services that require TLS certificates -in the `minio-operator` namespace are covered below in [MinIO Operator services with cert-manager](#minio-operator-services-with-cert-manager).. + This guide uses a self-signed ``Cluster Issuer``. + You can also use `other Issuers supported by cert-manager `__. + The main difference is that you must provide the ``Issuer`` CA certificate to MinIO, instead of the CA's mentioned in this guide. -> [!NOTE] -> This guide uses a self-signed `Cluster Issuer`. You can also use [other Issuers supported by Cert Manager](https://cert-manager.io/docs/configuration/issuers/). -> The main difference is that you must provide the `Issuer` CA certificate to minio, instead of the CA's mentioned in this guide. -## Getting Started +Prerequisites +------------- -### Prerequisites +- A `supported version of Kubernetes `__. + + While cert-manager supports `earlier K8s versions `__, the MinIO Operator requires a currently supported version. +- `kustomize `__ installed +- ``kubectl`` access to your ``k8s`` cluster -- Kubernetes version `+v1.21`. While cert-manager - supports [earlier K8s versions](https://cert-manager.io/docs/installation/supported-releases/), MinIO Operator requires 1.21 or later. -- [kustomize](https://kustomize.io/) installed -- `kubectl` access to your `k8s` cluster +Getting Started +--------------- -### Setup Cert-manager +Setup cert-manager +~~~~~~~~~~~~~~~~~~ -Install [cert-manager](https://cert-manager.io/docs/releases/release-notes/release-notes-1.12/) 1.12.X LTS is preferred, -or install latest, for example using kubectl. +Install cert-manager. +`Release 1.12.X LTS `__ is preferred, but you may install latest. -```bash -kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.12.9/cert-manager.yaml -``` +The following command installs version 1.12.13 using ``kubectl``. -For more details on the Cert Manager refer to https://cert-manager.io/docs/installation/. +.. code-block:: shell + :class: copyable + + kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.12.13/cert-manager.yaml -### Create Cluster Self-signed root Issuer +For more details on installing cert-manager, see their `installation instructions `__. -The `Cluster Issuer` is the cluster level Issuer all other certificates are derived from. Request Cert Manager to -generate this by creating a `ClusterIssuer` resource: +Create a self-signed root Issuer for the cluster +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -```yaml -# selfsigned-root-clusterissuer.yaml -apiVersion: cert-manager.io/v1 -kind: ClusterIssuer -metadata: - name: selfsigned-root -spec: - selfSigned: {} -``` +The ``Cluster Issuer`` is the top level Issuer from which all other certificates in the cluster derive. -```shell -kubectl apply -f selfsigned-root-clusterissuer.yaml -``` +1. Request cert-manager to generate this by creating a ``ClusterIssuer`` resource. -# MinIO Operator services with cert-manager + Create a file called ``selfsigned-root-clusterissuer.yaml`` with the following contents: -MinIO Operator manages the TLS certificate issuing for the services hosted in the `minio-operator` namespace. That is the Secure Token Service `sts`. + .. code-block:: yaml + :class: copyable + + # selfsigned-root-clusterissuer.yaml + apiVersion: cert-manager.io/v1 + kind: ClusterIssuer + metadata: + name: selfsigned-root + spec: + selfSigned: {} -This section describes how to generate the `sts` TLS certificate with Cert Manager. -These certificates must be issued before installing Operator. -Be sure to follow step [Create Cluster Self-signed root Issuer](#create-cluster-self-signed-root-issuer) mentioned above. +2. Apply the resource to the cluster: -## Secure Token Service (STS) + .. code-block:: shell + :class: copyable -MinIO STS is a service included with MinIO Operator that provides Native IAM Authentication for Kubernetes. In essence -this service allows you to control access to your MinIO tenant from your kubernetes applications without having to explicitly create credentials -for each application. For more information on the Service see the MinIO docs at https://min.io/docs/minio/kubernetes/upstream/developers/sts-for-operator.html. -There is also an [STS](https://github.com/minio/operator/blob/master/docs/STS.md) guide in the docs and example client code in -https://github.com/minio/operator/tree/master/examples/kustomization/sts-example. + kubectl apply -f selfsigned-root-clusterissuer.yaml -For the purpose of this guide, STS Service can be considered a webserver presented with a TLS certificate for https traffic. -This guide covers how to **disable** the automatic generation of the certificate in MinIO Operator and issue the certificate using -Cert Manager instead. +.. _minio-operator-services-with-cert-manager: -### Create minio-operator namespace CA Issuer +MinIO Operator services with cert-manager +----------------------------------------- -To create the Certificate Authority (CA) certificate used to issue certificates for services in the `minio-operator` -namespace, first create the minio-operator namespace +MinIO Operator manages the TLS certificate issuing for the services hosted in the ``minio-operator`` namespace. +That is the :ref:`Secure Token Service (sts) `. -```shell -kubectl create ns minio-operator -``` +This section describes how to generate the ``sts`` TLS certificate with cert-manager. -Request a Certificate with `spec.isCA: true` specified. This is our CA for the `minio-operator` namespace. +- These certificates **must** be issued *before* installing Operator. +- The cluster's self-signed root ``ClusterIssuer`` certificate must already exist, as described above. -```yaml -# operator-ca-tls-secret.yaml -apiVersion: cert-manager.io/v1 -kind: Certificate -metadata: - name: minio-operator-ca-certificate - namespace: minio-operator -spec: - isCA: true - commonName: operator - secretName: operator-ca-tls - duration: 70128h # 8y - privateKey: - algorithm: ECDSA - size: 256 - issuerRef: - name: selfsigned-root - kind: ClusterIssuer - group: cert-manager.io -``` - -```shell -kubectl apply -f operator-ca-tls-secret.yaml -``` -A new secret with the name `operator-ca-tls` is created in the `minio-operator` namespace, this is the CA issuing TLS certificates for the services in the `minio-operator` namespace. - -> [!IMPORTANT] -> Make sure to trust this certificate in your applications that need to interact with the `sts` service. - - -Now create the `Issuer`: - -```yaml -# operator-ca-issuer.yaml -apiVersion: cert-manager.io/v1 -kind: Issuer -metadata: - name: minio-operator-ca-issuer - namespace: minio-operator -spec: - ca: - secretName: operator-ca-tls -``` - -```shell -kubectl apply -f operator-ca-issuer.yaml -``` - -### Create TLS certificate for STS service - -Request Cert Manager to issue a new certificate containing following DNS domains: - -```shell -sts -sts.minio-operator.svc. -sts.minio-operator.svc. -``` - -> [!IMPORTANT] -> Replace `` with the actual values for your MinIO tenant. -> `cluster domain` is the internal root DNS domain assigned in your Kubernetes cluster. Typically this is `cluster.local`, check on your coredns -> configuration for the correct value for your Kubernetes cluster. For example, using `kubectl get configmap coredns -n kube-system -o jsonpath="{.data}"`. -> The way the root DNS domain is managed can vary depending on the Kubernetes distribution (Openshift, Rancher, EKS, etc.) - -Create a `Certificate` for the domains mentioned above: - -```yaml -# sts-tls-certificate.yaml -apiVersion: cert-manager.io/v1 -kind: Certificate -metadata: - name: sts-certmanager-cert - namespace: minio-operator -spec: - dnsNames: - - sts - - sts.minio-operator.svc - - sts.minio-operator.svc.cluster.local - secretName: sts-tls - issuerRef: - name: minio-operator-ca-issuer -``` - -```shell -kubectl apply -f sts-tls-certificate.yaml -``` -This creates a secret called `sts-tls` in the `minio-operator` namespace. - -> [!IMPORTANT] -> The secret name is not optional. Make sure the secret name is `sts-tls` by setting `spec.secretName: sts-tls` as in the example above. - -### Install Operator with Auto TLS disabled for STS - -When installing the Operator deployment, make sure to set `OPERATOR_STS_AUTO_TLS_ENABLED: off` env variable in the `minio-operator` container. This prevents -MinIO Operator from issuing the certificate for STS and instead wait for you to provide the TLS certificate issued by Cert Manager. - -> [!WARNING] -> Missing to provide the secret `sts-tls` containing the TLS certificate or providing an invalid key-pair in the secret will -> prevent the STS service from start. - -A way to make sure that the env variables are properly set is using kustomization to patch the `minio-operator` deployment: - -```yaml -# minio-operator/kustomization.yaml -apiVersion: kustomize.config.k8s.io/v1beta1 -kind: Kustomization - -resources: -- github.com/minio/operator/resources - -patches: -- patch: |- - apiVersion: apps/v1 - kind: Deployment - metadata: - name: minio-operator - namespace: minio-operator - spec: - template: - spec: - containers: - - name: minio-operator - env: - - name: OPERATOR_STS_AUTO_TLS_ENABLED - value: "off" - - name: OPERATOR_STS_ENABLED - value: "on" -``` - -```shell -kubectl apply -k minio-operator -``` - -# MinIO tenant TLS certificates with cert-manager - -## Create tenant-1 namespace CA Issuer - -To create the Certificate Authority (CA) certificate in the namespace `tenant-1`, first create the namespace - -```shell -kubectl create ns tenant-1 -``` - -Next, request a Certificate with `spec.isCA: true` specified: - -```yaml -# tenant-1-ca-certificate.yaml -apiVersion: cert-manager.io/v1 -kind: Certificate -metadata: - name: tenant-1-ca-certificate - namespace: tenant-1 -spec: - isCA: true - commonName: tenant-1-ca - secretName: tenant-1-ca-tls - duration: 70128h # 8y - privateKey: - algorithm: ECDSA - size: 256 - issuerRef: - name: selfsigned-root - kind: ClusterIssuer - group: cert-manager.io -``` - -```shell -kubectl apply -f tenant-1-ca-certificate.yaml -``` - - -Then create the `Issuer`: - -```yaml -# tenant-1-ca-issuer.yaml -apiVersion: cert-manager.io/v1 -kind: Issuer -metadata: - name: tenant-1-ca-issuer - namespace: tenant-1 -spec: - ca: - secretName: tenant-1-ca-tls -``` - -```shell -kubectl apply -f tenant-1-ca-issuer.yaml -``` - -## Deploy the tenant - -### Create a certificate for Tenant - -Request Cert Manager issue a new TLS server certificate for MinIO that includes the following DNS domains: - -```shell -minio. -minio..svc -minio..svc. -*.-hl..svc. -*..svc. -*..minio..svc.' -``` - -> [!IMPORTANT] -> Replace `` with the actual values for your MinIO tenant. -> * `` is the internal root DNS domain assigned in your Kubernetes cluster. Typically this is `cluster.local`, check on your coredns -> configuration for the correct value for your Kubernetes cluster. For example, using `kubectl get configmap coredns -n kube-system -o jsonpath="{.data}"`. -> The way the root DNS domain is managed can vary depending on the Kubernetes distribution (Openshift, Rancher, EKS, etc.) -> * `tenant-name` is the name provided to your tenant in the `metadata.name` of the Tenant YAML. For this example it is `myminio`. -> * `namespace` is the namespace where the tenant is created, the `metadata.namespace` notes that in the Tenant YAML. For this example it is `tenant-1`. - -Create a `Certificate` for the domains mentioned above: - -```yaml -# tenant-1-minio-certificate.yaml -apiVersion: cert-manager.io/v1 -kind: Certificate -metadata: - name: tenant-certmanager-cert - namespace: tenant-1 -spec: - dnsNames: - - "minio.tenant-1" - - "minio.tenant-1.svc" - - 'minio.tenant-1.svc.cluster.local' - - '*.minio.tenant-1.svc.cluster.local' - - '*.myminio-hl.tenant-1.svc.cluster.local' - - '*.myminio.minio.tenant-1.svc.cluster.local' - secretName: myminio-tls - issuerRef: - name: tenant-1-ca-issuer -``` - -> [!TIP] -> For this example, the Tenant name is `myminio`. We recommend naming the secret in the field `spec.secretName` as `-tls`, following the naming -> convention MinIO Operator uses when creates certificates with Autocert enabled (`spec.requestAutoCert: true`). - -```shell -kubectl apply -f tenant-1-minio-certificate.yaml -``` - -### Configure the tenant to use the certificate created by cert-manager +Secure Token Service (STS) +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +MinIO STS is a service included with MinIO Operator that provides Native IAM Authentication for Kubernetes. +This service allows you to control access to your MinIO tenant from your kubernetes applications without having to explicitly create credentials for each application. +For more information, see the :ref:`Secure Token Service documentation `. + +Think of the STS Service as a webserver presented with a TLS certificate for ``https`` traffic. +This guide covers how to **disable** the automatic generation of the certificate in MinIO Operator and issue the certificate using cert-manager instead. + +Create a CA Issuer for the ``minio-operator`` namespace ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +The ``minio-operator`` namespace needs to have its own certificate authority (CA), derived from the cluster's ``ClusterIssuer`` certificate create earlier. + +1. If it does not exist, create the ``minio-operator`` namespace + + .. code-block:: shell + :class: copyable + + kubectl create ns minio-operator + +2. Request a new Certificate with ``spec.isCA: true`` specified. + + This is our :abbr:`CA (Certificate Authority)` for the `minio-operator` namespace. + + Create a file called ``operator-ca-tls-secret.yaml`` with the following contents: + + .. code-block:: yaml + :class: copyable + :emphasize-lines: 7,8 + + # operator-ca-tls-secret.yaml + apiVersion: cert-manager.io/v1 + kind: Certificate + metadata: + name: minio-operator-ca-certificate + namespace: minio-operator + spec: + isCA: true + commonName: operator + secretName: operator-ca-tls + duration: 70128h # 8y + privateKey: + algorithm: ECDSA + size: 256 + issuerRef: + name: selfsigned-root + kind: ClusterIssuer + group: cert-manager.io + +3. Apply the resource to the cluster + + .. code-block:: shell + :class: copyable + + kubectl apply -f operator-ca-tls-secret.yaml + +Kubernetes creates a new secret with the name ``operator-ca-tls`` in the ``minio-operator`` namespace. +This certificate serves as the :abbr:`CA (Certificate Authority)` that issues TLS certificates only for the services in the ``minio-`operator`` namespace. + +.. important:: + + Make sure to trust this certificate in any applications that need to interact with the ``sts`` service. + + +Use the secret to create the `Issuer` ++++++++++++++++++++++++++++++++++++++ + +Use the secret created above to add an ``Issuer`` resource for the ``minio-operator`` namespace. + +1. Create a file called ``operator-ca-issuer.yaml`` with the following contents: + + .. code-block:: yaml + + # operator-ca-issuer.yaml + apiVersion: cert-manager.io/v1 + kind: Issuer + metadata: + name: minio-operator-ca-issuer + namespace: minio-operator + spec: + ca: + secretName: operator-ca-tls + + +2. Apply the resource to the cluster + + .. code-block:: shell + + kubectl apply -f operator-ca-issuer.yaml + +Create TLS certificate for STS service +++++++++++++++++++++++++++++++++++++++ + +Now that the ``Issuer`` exists in the ``minio-operator`` namespace, cert-manager can add a certificate. + +The certificate from cert-manager must be valid for the following DNS domains: + +- ``sts`` +- ``sts.minio-operator.svc.`` +- ``sts.minio-operator.svc.`` + + .. important:: + + Replace ```` with the actual values for your MinIO tenant. + ``cluster domain`` is the internal root DNS domain assigned in your Kubernetes cluster. + Typically, this is ``cluster.local``, but confirm the value by checking your coredns configuration for the correct value for your Kubernetes cluster. + + For example: + + .. code-block:: shell + :class: copyable + + kubectl get configmap coredns -n kube-system -o jsonpath="{.data}" + + Different Kubernetes providers manage the root domain differently. + Check with your Kubernetes provider for more information. + +1. Create a ``Certificate`` for the domains mentioned above: + + Create a file named ``sts-tls-certificate.yaml`` with the following contents: + + .. code-block:: yaml + :class: copyable + :emphasize-lines: 7,12 + + # sts-tls-certificate.yaml + apiVersion: cert-manager.io/v1 + kind: Certificate + metadata: + name: sts-certmanager-cert + namespace: minio-operator + spec: + dnsNames: + - sts + - sts.minio-operator.svc + - sts.minio-operator.svc.cluster.local + secretName: sts-tls + issuerRef: + name: minio-operator-ca-issuer + + .. important:: + + The ``spec.secretName`` is not optional! + + The secret name **must be** ``sts-tls``. + Confirm this by setting ``spec.secretName: sts-tls`` as highlighted above. + + +2. Apply the resource to the cluster: + + .. code-block:: shell + :class: copyable + + kubectl apply -f sts-tls-certificate.yaml + +This creates a secret called ``sts-tls`` in the ``minio-operator`` namespace. + +.. warning:: + + Failing to provide the ``sts-tls`` secret containing the TLS certificate or providing an invalid key-value pair in the secret will prevent the STS service from starting. + +Install Operator with Auto TLS disabled for STS +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When installing the Operator deployment, set the ``OPERATOR_STS_AUTO_TLS_ENABLED`` environment variable to ``off`` in the ``minio-operator`` container. + +Disabling this environment variable prevents the MinIO Operator from issuing the certificate for STS. +Instead, Operator waits for the TLS certificate issued by cert-manager. + +There are several options for defining an environment variable. +The steps below define the variable with kustomize. + +1. Create a kustomization patch file called ``kustomization.yaml`` with the below contents: + + .. code-block:: yaml + :class: copyable + + # minio-operator/kustomization.yaml + apiVersion: kustomize.config.k8s.io/v1beta1 + kind: Kustomization + + resources: + - github.com/minio/operator/resources + + patches: + - patch: |- + apiVersion: apps/v1 + kind: Deployment + metadata: + name: minio-operator + namespace: minio-operator + spec: + template: + spec: + containers: + - name: minio-operator + env: + - name: OPERATOR_STS_AUTO_TLS_ENABLED + value: "off" + - name: OPERATOR_STS_ENABLED + value: "on" + +2. Apply the kustomization resource to the cluster: + + .. code-block:: shell + :class: copyable + + kubectl apply -k minio-operator + +Manage tenant TLS certificates with cert-manager +------------------------------------------------ + +The following procedures create and apply the resources necessary to use cert-manager for the TLS certificates within a tenant. + +The procedures use ``tenant-1`` as the name of the tenant. +Replace the string ``tenant-1`` throughout the procedures to reflect the name of your tenant. + +Create the tenant namespace CA Issuer +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Before deploying a new tenant, create a Certificate Authority and Issuer for the tenant's namespace. + +1. If necessary, create the tenant's namespace. + + .. code-block:: shell + :class: copyable + + kubectl create ns tenant-1 + +2. Request a Certificate for a new Certificate Authority. + + Create a file called ``tenant-1-ca-certificate.yaml`` with the following contents: + + .. code-block:: yaml + :class: copyable + :emphasize-lines: 7,8 + + # tenant-1-ca-certificate.yaml + apiVersion: cert-manager.io/v1 + kind: Certificate + metadata: + name: tenant-1-ca-certificate + namespace: tenant-1 + spec: + isCA: true + commonName: tenant-1-ca + secretName: tenant-1-ca-tls + duration: 70128h # 8y + privateKey: + algorithm: ECDSA + size: 256 + issuerRef: + name: selfsigned-root + kind: ClusterIssuer + group: cert-manager.io + + .. important:: + + The ``spec.isCA`` field must be set to ``true`` to create this certificate as a certificate authority. + See the emphasized lines above. + +3. Apply the request file to the cluster. + + .. code-block:: shell + :class: copyable + + kubectl apply -f tenant-1-ca-certificate.yaml + +4. Generate a resource definition for an ``Issuer``. + + Create a file called ``tenant-1-ca-issuer.yaml`` with the following contents: + + .. code-block:: yaml + :class: copyable + + # tenant-1-ca-issuer.yaml + apiVersion: cert-manager.io/v1 + kind: Issuer + metadata: + name: tenant-1-ca-issuer + namespace: tenant-1 + spec: + ca: + secretName: tenant-1-ca-tls + +5. Apply the ``Issuer`` resource definition to the cluster. + + .. code-block:: shell + :class: copyable + + kubectl apply -f tenant-1-ca-issuer.yaml + +Deploy the tenant +~~~~~~~~~~~~~~~~~ + +With the Certificate Authority and ``Issuer`` in place for the tenant's namespace, you can now deploy the object store tenant. + +Create a certificate for the tenant ++++++++++++++++++++++++++++++++++++ + +Request that cert-manager issue a new TLS server certificate for MinIO. +The certificate must be valid for the following DNS domains: + +- ``minio.`` +- ``minio..svc`` +- ``minio..svc.`` +- ``*.-hl..svc.`` +- ``*..svc.`` +- ``*..minio..svc.'`` + +.. important:: + + Replace the filler strings (````) with values for your tenant: + + - ```` is the internal root DNS domain assigned in your Kubernetes cluster. + Typically, this is ``cluster.local``, but confirm the value by checking your coredns configuration for the correct value for your Kubernetes cluster. + + For example: + + .. code-block:: shell + :class: copyable + + kubectl get configmap coredns -n kube-system -o jsonpath="{.data}" + + Different Kubernetes providers manage the root domain differently. + Check with your Kubernetes provider for more information. + + - ``tenant-name`` is the name provided to your tenant in the ``metadata.name`` of the Tenant YAML. + For this example it is ``myminio``. + + - ``namespace`` is the namespace where the tenant is created, the ``metadata.namespace`` notes that in the Tenant YAML. + For this example it is ``tenant-1``. + +1. Request a ``Certificate`` for the domains mentioned above + + Create a file called ``tenant-1-minio-certificate.yaml`` with the following contents: + + .. code-block:: yaml + :class: copyable + + # tenant-1-minio-certificate.yaml + apiVersion: cert-manager.io/v1 + kind: Certificate + metadata: + name: tenant-certmanager-cert + namespace: tenant-1 + spec: + dnsNames: + - "minio.tenant-1" + - "minio.tenant-1.svc" + - 'minio.tenant-1.svc.cluster.local' + - '*.minio.tenant-1.svc.cluster.local' + - '*.myminio-hl.tenant-1.svc.cluster.local' + - '*.myminio.minio.tenant-1.svc.cluster.local' + secretName: myminio-tls + issuerRef: + name: tenant-1-ca-issuer + + .. tip:: + + For this example, the Tenant name is ``myminio``. + We recommend naming the secret in the field ``spec.secretName`` as ``-tls``, following the naming convention the MinIO Operator uses when creating certificates without cert-manager. + +2. Apply the certificate resource to the cluster. + + .. code-block:: shell + :class: copyable + + kubectl apply -f tenant-1-minio-certificate.yaml + +Configure the tenant to use the certificate created by cert-manager ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ In the tenant spec, do the following: -* Disable Autocert `spec.requestAutoCert: false`. This instructs Operator to not attempt to issue certificates and instead rely on Cert Manager to provide them in a secret. -* Reference the Secret containing the TLS certificate from the previous step in `spec.externalCertSecret`. - -```yaml -apiVersion: minio.min.io/v2 -kind: Tenant -metadata: - name: myminio - namespace: tenant-1 -spec: -... - ## Disable default tls certificates. - requestAutoCert: false - ## Use certificates generated by cert-manager. - externalCertSecret: - - name: myminio-tls - type: cert-manager.io/v1 -... -``` - -## Trust tenant-1 CA in MinIO Operator - -MinIO Operator can trust as many CA certificates as provided. To do this, create a secret with the prefix `operator-ca-tls-` -followed by a unique identifier in the `minio-operator` namespace. - -MinIO Operator mounts and trusts all certificates issued by the provided CA's. This is required because -MinIO Operator performs health checks using the */minio/health/cluster* endpoint. - -If Operator is not correctly configured to trust the MinIO Certificate (or its CA), you will see an error message like the following in the Operator Pod logs: - -```error -Failed to get cluster health: Get "https://minio.tenant-1.svc.cluster.local/minio/health/cluster": -x509: certificate signed by unknown authority -``` -For more details about health checks, see https://min.io/docs/minio/Kubernetes/upstream/operations/monitoring/healthcheck-probe.html#cluster-write-quorum. - -### Create `operator-ca-tls-tenant-1` secret - -Copy the Cert Manager generated CA public key (ca.crt) into the `minio-operator` namespace. This allows Operator to trust -the cert-manager issued CA and the derived certificates. - -Create a `ca.crt` file containing the CA: -```sh -kubectl get secrets -n tenant-1 tenant-1-ca-tls -o=jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt -``` - -Create the secret: -```sh -kubectl create secret generic operator-ca-tls-tenant-1 --from-file=ca.crt -n minio-operator -``` -> [!TIP] -> In this example we choose a secret name of `operator-ca-tls-tenant-1`. Note the tenant namespace -> `tenant-1` is used as suffix for easy identification of which namespace the CA is coming from. \ No newline at end of file +- Disable Autocert by setting the ``spec.requestAutoCert`` field to ``false``. + + This instructs the MinIO Operator to not attempt to issue certificates and instead rely on cert-manager to provide them in a secret. +- Reference the Secret containing the TLS certificate from the previous procedure in `spec.externalCertSecret`. + + +1. Modify the tenant YAML ``spec`` section to reflect the above + + .. code-block:: yaml + :emphasize-lines: 6,9,11 + + apiVersion: minio.min.io/v2 + kind: Tenant + metadata: + name: myminio + namespace: tenant-1 + spec: + ... + ## Disable default tls certificates. + requestAutoCert: false + ## Use certificates generated by cert-manager. + externalCertSecret: + - name: myminio-tls + type: cert-manager.io/v1 + ... + +Trust the tenant's CA in MinIO Operator +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +MinIO Operator can trust as many CA certificates as provided. + +To do this, create a secret with the prefix ``operator-ca-tls-`` followed by a unique identifier in the `minio-operator` namespace. + +MinIO Operator mounts and trusts **all** certificates issued by the provided Certificate Authorities. +This is required because the MinIO Operator performs health checks using the ``/minio/health/cluster`` endpoint. + +If Operator is not correctly configured to trust the MinIO tenant's Certificate (or its CA), you will see an error message like the following in the Operator Pod logs: + +.. code-block:: shell + + Failed to get cluster health: Get "https://minio.tenant-1.svc.cluster.local/minio/health/cluster": + x509: certificate signed by unknown authority + +For more details about health checks, refer to :ref:`Healthcheck API `. + +Create ``operator-ca-tls-tenant-1`` secret +++++++++++++++++++++++++++++++++++++++++++ + +Copy the tenant's cert-manager generated CA public key (``ca.crt``) into the `minio-operator` namespace. +This allows Operator to trust the cert-manager issued CA and all certificates derived from it. + +1. Create a `ca.crt` file containing the CA: + + .. code-block:: shell + :class: copyable + + kubectl get secrets -n tenant-1 tenant-1-ca-tls -o=jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt + +2. Create the secret: + + .. code-block:: shell + :class: copyable + + kubectl create secret generic operator-ca-tls-tenant-1 --from-file=ca.crt -n minio-operator + +.. tip:: + + In this example we chose a secret name of ``operator-ca-tls-tenant-1``. + Note the tenant namespace ``tenant-1`` is used as suffix for easy identification of which namespace the CA is coming from. + Use the name of your tenant namespace for easier linking secrets to the related resources. \ No newline at end of file diff --git a/source/operations/monitoring/healthcheck-probe.rst b/source/operations/monitoring/healthcheck-probe.rst index 60a8ebfe..db59d8ba 100644 --- a/source/operations/monitoring/healthcheck-probe.rst +++ b/source/operations/monitoring/healthcheck-probe.rst @@ -39,6 +39,8 @@ a Prometheus :ref:`alert ` using the ``minio_cluster_nodes_offline_total`` metric to detect whether one or more MinIO nodes are offline. +.. _minio-cluster-write-quorum: + Cluster Write Quorum -------------------- diff --git a/source/operations/network-encryption.rst b/source/operations/network-encryption.rst index 9a1bfe06..d526b5be 100644 --- a/source/operations/network-encryption.rst +++ b/source/operations/network-encryption.rst @@ -64,6 +64,12 @@ Enabling TLS If you have a custom Subject Alternative Name (SAN) certificate that is *not* also a wildcard cert, the TLS certificate SAN **must** apply to the hostname for its parent node. Without a wildcard, the SAN must match exactly to be able to connect to the tenant. + Certificate Management with cert-manager + ---------------------------------------- + + Rather than then MinIO Operator managing certificates, you can configure the deployment to use `cert-manager `__. + For instructions for deploying the MinIO Operator and tenants using cert-manager, refer to the :ref:`cert-manager page `. + .. cond:: linux From 1054e0d44083d303b900abb7c691bcb901e4b21b Mon Sep 17 00:00:00 2001 From: Daryl White <53910321+djwfyi@users.noreply.github.com> Date: Tue, 17 Sep 2024 13:55:19 -0400 Subject: [PATCH 03/11] Restructing cert-manager docs --- source/operations/cert-manager.rst | 512 +----------------- .../cert-manager/cert-manager-operator.rst | 239 ++++++++ .../cert-manager/cert-manager-tenants.rst | 242 +++++++++ 3 files changed, 502 insertions(+), 491 deletions(-) create mode 100644 source/operations/cert-manager/cert-manager-operator.rst create mode 100644 source/operations/cert-manager/cert-manager-tenants.rst diff --git a/source/operations/cert-manager.rst b/source/operations/cert-manager.rst index 8f093bff..3c2a9454 100644 --- a/source/operations/cert-manager.rst +++ b/source/operations/cert-manager.rst @@ -13,33 +13,13 @@ cert-manager TLS certificate management with cert-manager -------------------------------------------- -`cert-manager `__ works as a controller for managing certificates within Kubernetes clusters. -cert-manager obtains valid certificates from an ``Issuer`` or ``ClusterIssuer`` and can renew certificates prior to expiration. - +`cert-manager `__ manages certificates within Kubernetes clusters. The MinIO Operator supports using cert-manager for certificates as an alternative to the MinIO Operator managing certificates for itself and its tenants. -On a Kubernetes cluster, cert-manager requires a global level ``ClusterIssuer`` to generate intermediary ``Issuers`` and certificates. -At the namespace level, cert-manager issues certificates derived from an ``Issuer``. -A ``ClusterIssuer`` can issue certificates for multiple namespaces. -An ``Issuer`` only issues certificates for its own namespace. - -To learn more about cert-manager Issuers, refer to the `Issuer documentation `__. - -Below is a logical view of a Kubernetes cluster with four namespaces: - -- ``minio-operator`` -- ``tenant-1`` -- ``tenant-2`` -- ``other-namespace`` - -.. image:: /images/k8s/cert-manager-namespaces.svg - :width: 600px - :alt: A graphic depiction of a kubernetes cluster with four namespaces. Each namespace in its own white box. One is labeled minio-operator with contents two sts content items. A second is labeled tenant-1 with a MinIO logo and four drives of a pool. A third is labeled tenant-2 with similar contents to the tenant-1 box. The fourth is labeled other namespace with contents of other pods. - :align: center - -This guide shows you how to set up a different Certificate Authority (CA) in each namespace. -Each namespace's CA references the global ``Cluster Issuer``. +cert-manager obtains valid certificates from an ``Issuer`` or ``ClusterIssuer`` and can renew certificates prior to expiration. +A ``ClusterIssuer`` issues certificates for multiple namespaces. +An ``Issuer`` only mints certificates for its own namespace. The following graphic depicts how various namespaces make use of either an ``Issuer`` or ``ClusterIssuer`` type. @@ -47,13 +27,14 @@ The following graphic depicts how various namespaces make use of either an ``Iss - The ``default`` namespace receives the global ``Cluster Issuer``. - Each tenant's namespace receives a local ``Issuer``. - The ``minio-operator`` namespace receives a local ``Issuer``. - More about services that require TLS certificates in the ``minio-operator`` namespace are covered :ref:`below `. .. image:: /images/k8s/cert-manager-cluster.svg :width: 600px :alt: A graphic depiction of a kubernetes cluster with five separate namespaces represented by different boxes. One is labeled minio-operator with a text box that says "minio-operator: issuer". A second is labeled tenant-1 with text box that says "tenant-1: issuer". A third is labeled default with a text box that says "root:ClusterIssuer". A fourth is labeled tenant-2 with a text box that says "tenant-2: issuer". The fifth is labeled cert-manager with a text box that says "minio-operator: issuer". :align: center +This guide shows you how to install cert-manager and set up the global ``Cluster Issuer``. + .. note:: This guide uses a self-signed ``Cluster Issuer``. @@ -65,19 +46,16 @@ Prerequisites ------------- - A `supported version of Kubernetes `__. - - While cert-manager supports `earlier K8s versions `__, the MinIO Operator requires a currently supported version. - `kustomize `__ installed - ``kubectl`` access to your ``k8s`` cluster -Getting Started ---------------- +.. _minio-setup-certmanager: Setup cert-manager -~~~~~~~~~~~~~~~~~~ +------------------ -Install cert-manager. -`Release 1.12.X LTS `__ is preferred, but you may install latest. +Install cert-manager +~~~~~~~~~~~~~~~~~~~~ The following command installs version 1.12.13 using ``kubectl``. @@ -86,10 +64,11 @@ The following command installs version 1.12.13 using ``kubectl``. kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.12.13/cert-manager.yaml +`Release 1.12.X LTS `__ is preferred, but you may install the latest version. For more details on installing cert-manager, see their `installation instructions `__. -Create a self-signed root Issuer for the cluster -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Create a self-signed Cluster Issuer for the cluster +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The ``Cluster Issuer`` is the top level Issuer from which all other certificates in the cluster derive. @@ -115,463 +94,14 @@ The ``Cluster Issuer`` is the top level Issuer from which all other certificates kubectl apply -f selfsigned-root-clusterissuer.yaml -.. _minio-operator-services-with-cert-manager: - -MinIO Operator services with cert-manager ------------------------------------------ - -MinIO Operator manages the TLS certificate issuing for the services hosted in the ``minio-operator`` namespace. -That is the :ref:`Secure Token Service (sts) `. - -This section describes how to generate the ``sts`` TLS certificate with cert-manager. - -- These certificates **must** be issued *before* installing Operator. -- The cluster's self-signed root ``ClusterIssuer`` certificate must already exist, as described above. - -Secure Token Service (STS) -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -MinIO STS is a service included with MinIO Operator that provides Native IAM Authentication for Kubernetes. -This service allows you to control access to your MinIO tenant from your kubernetes applications without having to explicitly create credentials for each application. -For more information, see the :ref:`Secure Token Service documentation `. - -Think of the STS Service as a webserver presented with a TLS certificate for ``https`` traffic. -This guide covers how to **disable** the automatic generation of the certificate in MinIO Operator and issue the certificate using cert-manager instead. - -Create a CA Issuer for the ``minio-operator`` namespace -+++++++++++++++++++++++++++++++++++++++++++++++++++++++ - -The ``minio-operator`` namespace needs to have its own certificate authority (CA), derived from the cluster's ``ClusterIssuer`` certificate create earlier. - -1. If it does not exist, create the ``minio-operator`` namespace - - .. code-block:: shell - :class: copyable - - kubectl create ns minio-operator - -2. Request a new Certificate with ``spec.isCA: true`` specified. - - This is our :abbr:`CA (Certificate Authority)` for the `minio-operator` namespace. - - Create a file called ``operator-ca-tls-secret.yaml`` with the following contents: - - .. code-block:: yaml - :class: copyable - :emphasize-lines: 7,8 - - # operator-ca-tls-secret.yaml - apiVersion: cert-manager.io/v1 - kind: Certificate - metadata: - name: minio-operator-ca-certificate - namespace: minio-operator - spec: - isCA: true - commonName: operator - secretName: operator-ca-tls - duration: 70128h # 8y - privateKey: - algorithm: ECDSA - size: 256 - issuerRef: - name: selfsigned-root - kind: ClusterIssuer - group: cert-manager.io - -3. Apply the resource to the cluster - - .. code-block:: shell - :class: copyable - - kubectl apply -f operator-ca-tls-secret.yaml - -Kubernetes creates a new secret with the name ``operator-ca-tls`` in the ``minio-operator`` namespace. -This certificate serves as the :abbr:`CA (Certificate Authority)` that issues TLS certificates only for the services in the ``minio-`operator`` namespace. - -.. important:: - - Make sure to trust this certificate in any applications that need to interact with the ``sts`` service. - - -Use the secret to create the `Issuer` -+++++++++++++++++++++++++++++++++++++ - -Use the secret created above to add an ``Issuer`` resource for the ``minio-operator`` namespace. - -1. Create a file called ``operator-ca-issuer.yaml`` with the following contents: - - .. code-block:: yaml - - # operator-ca-issuer.yaml - apiVersion: cert-manager.io/v1 - kind: Issuer - metadata: - name: minio-operator-ca-issuer - namespace: minio-operator - spec: - ca: - secretName: operator-ca-tls - - -2. Apply the resource to the cluster - - .. code-block:: shell - - kubectl apply -f operator-ca-issuer.yaml - -Create TLS certificate for STS service -++++++++++++++++++++++++++++++++++++++ - -Now that the ``Issuer`` exists in the ``minio-operator`` namespace, cert-manager can add a certificate. - -The certificate from cert-manager must be valid for the following DNS domains: - -- ``sts`` -- ``sts.minio-operator.svc.`` -- ``sts.minio-operator.svc.`` - - .. important:: - - Replace ```` with the actual values for your MinIO tenant. - ``cluster domain`` is the internal root DNS domain assigned in your Kubernetes cluster. - Typically, this is ``cluster.local``, but confirm the value by checking your coredns configuration for the correct value for your Kubernetes cluster. - - For example: - - .. code-block:: shell - :class: copyable - - kubectl get configmap coredns -n kube-system -o jsonpath="{.data}" - - Different Kubernetes providers manage the root domain differently. - Check with your Kubernetes provider for more information. - -1. Create a ``Certificate`` for the domains mentioned above: - - Create a file named ``sts-tls-certificate.yaml`` with the following contents: - - .. code-block:: yaml - :class: copyable - :emphasize-lines: 7,12 - - # sts-tls-certificate.yaml - apiVersion: cert-manager.io/v1 - kind: Certificate - metadata: - name: sts-certmanager-cert - namespace: minio-operator - spec: - dnsNames: - - sts - - sts.minio-operator.svc - - sts.minio-operator.svc.cluster.local - secretName: sts-tls - issuerRef: - name: minio-operator-ca-issuer - - .. important:: - - The ``spec.secretName`` is not optional! - - The secret name **must be** ``sts-tls``. - Confirm this by setting ``spec.secretName: sts-tls`` as highlighted above. - - -2. Apply the resource to the cluster: - - .. code-block:: shell - :class: copyable - - kubectl apply -f sts-tls-certificate.yaml - -This creates a secret called ``sts-tls`` in the ``minio-operator`` namespace. - -.. warning:: - - Failing to provide the ``sts-tls`` secret containing the TLS certificate or providing an invalid key-value pair in the secret will prevent the STS service from starting. - -Install Operator with Auto TLS disabled for STS -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -When installing the Operator deployment, set the ``OPERATOR_STS_AUTO_TLS_ENABLED`` environment variable to ``off`` in the ``minio-operator`` container. - -Disabling this environment variable prevents the MinIO Operator from issuing the certificate for STS. -Instead, Operator waits for the TLS certificate issued by cert-manager. - -There are several options for defining an environment variable. -The steps below define the variable with kustomize. - -1. Create a kustomization patch file called ``kustomization.yaml`` with the below contents: - - .. code-block:: yaml - :class: copyable - - # minio-operator/kustomization.yaml - apiVersion: kustomize.config.k8s.io/v1beta1 - kind: Kustomization - - resources: - - github.com/minio/operator/resources - - patches: - - patch: |- - apiVersion: apps/v1 - kind: Deployment - metadata: - name: minio-operator - namespace: minio-operator - spec: - template: - spec: - containers: - - name: minio-operator - env: - - name: OPERATOR_STS_AUTO_TLS_ENABLED - value: "off" - - name: OPERATOR_STS_ENABLED - value: "on" - -2. Apply the kustomization resource to the cluster: - - .. code-block:: shell - :class: copyable - - kubectl apply -k minio-operator - -Manage tenant TLS certificates with cert-manager ------------------------------------------------- - -The following procedures create and apply the resources necessary to use cert-manager for the TLS certificates within a tenant. - -The procedures use ``tenant-1`` as the name of the tenant. -Replace the string ``tenant-1`` throughout the procedures to reflect the name of your tenant. - -Create the tenant namespace CA Issuer -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Before deploying a new tenant, create a Certificate Authority and Issuer for the tenant's namespace. - -1. If necessary, create the tenant's namespace. - - .. code-block:: shell - :class: copyable - - kubectl create ns tenant-1 - -2. Request a Certificate for a new Certificate Authority. - - Create a file called ``tenant-1-ca-certificate.yaml`` with the following contents: - - .. code-block:: yaml - :class: copyable - :emphasize-lines: 7,8 - - # tenant-1-ca-certificate.yaml - apiVersion: cert-manager.io/v1 - kind: Certificate - metadata: - name: tenant-1-ca-certificate - namespace: tenant-1 - spec: - isCA: true - commonName: tenant-1-ca - secretName: tenant-1-ca-tls - duration: 70128h # 8y - privateKey: - algorithm: ECDSA - size: 256 - issuerRef: - name: selfsigned-root - kind: ClusterIssuer - group: cert-manager.io - - .. important:: - - The ``spec.isCA`` field must be set to ``true`` to create this certificate as a certificate authority. - See the emphasized lines above. - -3. Apply the request file to the cluster. - - .. code-block:: shell - :class: copyable - - kubectl apply -f tenant-1-ca-certificate.yaml - -4. Generate a resource definition for an ``Issuer``. - - Create a file called ``tenant-1-ca-issuer.yaml`` with the following contents: - - .. code-block:: yaml - :class: copyable - - # tenant-1-ca-issuer.yaml - apiVersion: cert-manager.io/v1 - kind: Issuer - metadata: - name: tenant-1-ca-issuer - namespace: tenant-1 - spec: - ca: - secretName: tenant-1-ca-tls - -5. Apply the ``Issuer`` resource definition to the cluster. - - .. code-block:: shell - :class: copyable - - kubectl apply -f tenant-1-ca-issuer.yaml - -Deploy the tenant -~~~~~~~~~~~~~~~~~ - -With the Certificate Authority and ``Issuer`` in place for the tenant's namespace, you can now deploy the object store tenant. - -Create a certificate for the tenant -+++++++++++++++++++++++++++++++++++ - -Request that cert-manager issue a new TLS server certificate for MinIO. -The certificate must be valid for the following DNS domains: - -- ``minio.`` -- ``minio..svc`` -- ``minio..svc.`` -- ``*.-hl..svc.`` -- ``*..svc.`` -- ``*..minio..svc.'`` - -.. important:: - - Replace the filler strings (````) with values for your tenant: - - - ```` is the internal root DNS domain assigned in your Kubernetes cluster. - Typically, this is ``cluster.local``, but confirm the value by checking your coredns configuration for the correct value for your Kubernetes cluster. - - For example: - - .. code-block:: shell - :class: copyable - - kubectl get configmap coredns -n kube-system -o jsonpath="{.data}" - - Different Kubernetes providers manage the root domain differently. - Check with your Kubernetes provider for more information. - - - ``tenant-name`` is the name provided to your tenant in the ``metadata.name`` of the Tenant YAML. - For this example it is ``myminio``. - - - ``namespace`` is the namespace where the tenant is created, the ``metadata.namespace`` notes that in the Tenant YAML. - For this example it is ``tenant-1``. - -1. Request a ``Certificate`` for the domains mentioned above - - Create a file called ``tenant-1-minio-certificate.yaml`` with the following contents: - - .. code-block:: yaml - :class: copyable - - # tenant-1-minio-certificate.yaml - apiVersion: cert-manager.io/v1 - kind: Certificate - metadata: - name: tenant-certmanager-cert - namespace: tenant-1 - spec: - dnsNames: - - "minio.tenant-1" - - "minio.tenant-1.svc" - - 'minio.tenant-1.svc.cluster.local' - - '*.minio.tenant-1.svc.cluster.local' - - '*.myminio-hl.tenant-1.svc.cluster.local' - - '*.myminio.minio.tenant-1.svc.cluster.local' - secretName: myminio-tls - issuerRef: - name: tenant-1-ca-issuer - - .. tip:: - - For this example, the Tenant name is ``myminio``. - We recommend naming the secret in the field ``spec.secretName`` as ``-tls``, following the naming convention the MinIO Operator uses when creating certificates without cert-manager. - -2. Apply the certificate resource to the cluster. - - .. code-block:: shell - :class: copyable - - kubectl apply -f tenant-1-minio-certificate.yaml - -Configure the tenant to use the certificate created by cert-manager -+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - -In the tenant spec, do the following: - -- Disable Autocert by setting the ``spec.requestAutoCert`` field to ``false``. - - This instructs the MinIO Operator to not attempt to issue certificates and instead rely on cert-manager to provide them in a secret. -- Reference the Secret containing the TLS certificate from the previous procedure in `spec.externalCertSecret`. - - -1. Modify the tenant YAML ``spec`` section to reflect the above - - .. code-block:: yaml - :emphasize-lines: 6,9,11 - - apiVersion: minio.min.io/v2 - kind: Tenant - metadata: - name: myminio - namespace: tenant-1 - spec: - ... - ## Disable default tls certificates. - requestAutoCert: false - ## Use certificates generated by cert-manager. - externalCertSecret: - - name: myminio-tls - type: cert-manager.io/v1 - ... - -Trust the tenant's CA in MinIO Operator -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -MinIO Operator can trust as many CA certificates as provided. - -To do this, create a secret with the prefix ``operator-ca-tls-`` followed by a unique identifier in the `minio-operator` namespace. - -MinIO Operator mounts and trusts **all** certificates issued by the provided Certificate Authorities. -This is required because the MinIO Operator performs health checks using the ``/minio/health/cluster`` endpoint. - -If Operator is not correctly configured to trust the MinIO tenant's Certificate (or its CA), you will see an error message like the following in the Operator Pod logs: - -.. code-block:: shell - - Failed to get cluster health: Get "https://minio.tenant-1.svc.cluster.local/minio/health/cluster": - x509: certificate signed by unknown authority - -For more details about health checks, refer to :ref:`Healthcheck API `. - -Create ``operator-ca-tls-tenant-1`` secret -++++++++++++++++++++++++++++++++++++++++++ - -Copy the tenant's cert-manager generated CA public key (``ca.crt``) into the `minio-operator` namespace. -This allows Operator to trust the cert-manager issued CA and all certificates derived from it. - -1. Create a `ca.crt` file containing the CA: - - .. code-block:: shell - :class: copyable - - kubectl get secrets -n tenant-1 tenant-1-ca-tls -o=jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt - -2. Create the secret: - - .. code-block:: shell - :class: copyable +Next steps +---------- - kubectl create secret generic operator-ca-tls-tenant-1 --from-file=ca.crt -n minio-operator +Set up :ref:`cert-manager for the MinIO Operator `. -.. tip:: +.. toctree:: + :titlesonly: + :hidden: - In this example we chose a secret name of ``operator-ca-tls-tenant-1``. - Note the tenant namespace ``tenant-1`` is used as suffix for easy identification of which namespace the CA is coming from. - Use the name of your tenant namespace for easier linking secrets to the related resources. \ No newline at end of file + /operations/cert-manager/cert-manager-operator + /operations/cert-manager/cert-manager-tenants \ No newline at end of file diff --git a/source/operations/cert-manager/cert-manager-operator.rst b/source/operations/cert-manager/cert-manager-operator.rst new file mode 100644 index 00000000..22de9edd --- /dev/null +++ b/source/operations/cert-manager/cert-manager-operator.rst @@ -0,0 +1,239 @@ +.. _minio-certmanager-operator: + +========================= +cert-manager for Operator +========================= + +.. default-domain:: minio + +.. contents:: Table of Contents + :local: + :depth: 1 + + +MinIO Operator manages the TLS certificate issuing for the services hosted in the ``minio-operator`` namespace. + +This page describes how to manage the Operator's TLS certificates with :ref:`cert-manager `. + +Prerequisites +------------- + +- A `supported version of Kubernetes `__. +- `kustomize `__ installed +- ``kubectl`` access to your ``k8s`` cluster +- Completed the steps to :ref:`set up cert-manager ` +- The MinIO Operator must not yet be installed. + + +1) Create a CA Issuer for the ``minio-operator`` namespace +---------------------------------------------------------- + +This guide **disables** the automatic generation of certificates in MinIO Operator and issues certificates using cert-manager instead. + +The ``minio-operator`` namespace needs to have its own certificate authority (CA), derived from the cluster's ``ClusterIssuer`` certificate created during :ref:`cert-manager setup `. +Create this CA certificate using cert-manager. + +.. important:: + + This CA certificate should exist *before* installing MinIO Operator. + +1. If it does not exist, create the ``minio-operator`` namespace + + .. code-block:: shell + :class: copyable + + kubectl create ns minio-operator + +2. Request a new Certificate with ``spec.isCA: true`` specified. + + This certificate serves as the :abbr:`CA (Certificate Authority)` for the `minio-operator` namespace. + + Create a file called ``operator-ca-tls-secret.yaml`` with the following contents: + + .. code-block:: yaml + :class: copyable + :emphasize-lines: 7,8 + + # operator-ca-tls-secret.yaml + apiVersion: cert-manager.io/v1 + kind: Certificate + metadata: + name: minio-operator-ca-certificate + namespace: minio-operator + spec: + isCA: true + commonName: operator + secretName: operator-ca-tls + duration: 70128h # 8y + privateKey: + algorithm: ECDSA + size: 256 + issuerRef: + name: selfsigned-root + kind: ClusterIssuer + group: cert-manager.io + +3. Apply the resource to the cluster + + .. code-block:: shell + :class: copyable + + kubectl apply -f operator-ca-tls-secret.yaml + +Kubernetes creates a new secret with the name ``operator-ca-tls`` in the ``minio-operator`` namespace. + +.. important:: + + Make sure to trust this certificate in any applications that need to interact with the MinIO Operator. + + +2) Use the secret to create the ``Issuer`` +------------------------------------------ + +Use the secret created above to add an ``Issuer`` resource for the ``minio-operator`` namespace. + +1. Create a file called ``operator-ca-issuer.yaml`` with the following contents: + + .. code-block:: yaml + + # operator-ca-issuer.yaml + apiVersion: cert-manager.io/v1 + kind: Issuer + metadata: + name: minio-operator-ca-issuer + namespace: minio-operator + spec: + ca: + secretName: operator-ca-tls + + +2. Apply the resource to the cluster + + .. code-block:: shell + + kubectl apply -f operator-ca-issuer.yaml + +3) Create TLS certificate +------------------------- + +Now that the ``Issuer`` exists in the ``minio-operator`` namespace, cert-manager can add a certificate. + +The certificate from cert-manager must be valid for the following DNS domains: + +- ``sts`` +- ``sts.minio-operator.svc.`` +- ``sts.minio-operator.svc.`` + + .. important:: + + Replace ```` with the actual value for your MinIO tenant. + ``cluster domain`` is the internal root DNS domain assigned in your Kubernetes cluster. + Typically, this is ``cluster.local``, but confirm the value by checking your coredns configuration for the correct value for your Kubernetes cluster. + + For example: + + .. code-block:: shell + :class: copyable + + kubectl get configmap coredns -n kube-system -o jsonpath="{.data}" + + Different Kubernetes providers manage the root domain differently. + Check with your Kubernetes provider for more information. + +1. Create a ``Certificate`` for the domains mentioned above: + + Create a file named ``sts-tls-certificate.yaml`` with the following contents: + + .. code-block:: yaml + :class: copyable + :emphasize-lines: 7,12 + + # sts-tls-certificate.yaml + apiVersion: cert-manager.io/v1 + kind: Certificate + metadata: + name: sts-certmanager-cert + namespace: minio-operator + spec: + dnsNames: + - sts + - sts.minio-operator.svc + - sts.minio-operator.svc.cluster.local # Replace cluster.local with the value for your domain. + secretName: sts-tls + issuerRef: + name: minio-operator-ca-issuer + + .. important:: + + The ``spec.secretName`` is not optional. + + The secret name **must** be ``sts-tls``. + Confirm this by setting ``spec.secretName: sts-tls`` as highlighted above. + +2. Apply the resource to the cluster: + + .. code-block:: shell + :class: copyable + + kubectl apply -f sts-tls-certificate.yaml + +This creates a secret called ``sts-tls`` in the ``minio-operator`` namespace. + +.. warning:: + + Failing to provide the ``sts-tls`` secret containing the TLS certificate or providing an invalid ``key-value`` pair in the secret will prevent the STS service from starting. + +4) Install Operator with Auto TLS disabled +------------------------------------------ + +You can now :ref:`install the MinIO Operator `. + +When installing the Operator deployment, set the ``OPERATOR_STS_AUTO_TLS_ENABLED`` environment variable to ``off`` in the ``minio-operator`` container. + +Disabling this environment variable prevents the MinIO Operator from issuing the certificates. +Instead, Operator waits for the TLS certificate issued by cert-manager. + +There are several for defining an environment variable depending on how you install the Operator. +The steps below define the variable with kustomize. + +1. Create a kustomization patch file called ``kustomization.yaml`` with the below contents: + + .. code-block:: yaml + :class: copyable + + # minio-operator/kustomization.yaml + apiVersion: kustomize.config.k8s.io/v1beta1 + kind: Kustomization + + resources: + - github.com/minio/operator/resources + + patches: + - patch: |- + apiVersion: apps/v1 + kind: Deployment + metadata: + name: minio-operator + namespace: minio-operator + spec: + template: + spec: + containers: + - name: minio-operator + env: + - name: OPERATOR_STS_AUTO_TLS_ENABLED + value: "off" + - name: OPERATOR_STS_ENABLED + value: "on" + +2. Apply the kustomization resource to the cluster: + + .. code-block:: shell + :class: copyable + + kubectl apply -k minio-operator + +Next steps +---------- + +Set up :ref:`cert-manager for a MinIO Tenant `. \ No newline at end of file diff --git a/source/operations/cert-manager/cert-manager-tenants.rst b/source/operations/cert-manager/cert-manager-tenants.rst new file mode 100644 index 00000000..35ba162f --- /dev/null +++ b/source/operations/cert-manager/cert-manager-tenants.rst @@ -0,0 +1,242 @@ +.. _minio-certmanager-tenants: + +======================== +cert-manager for Tenants +======================== + +.. default-domain:: minio + +.. contents:: Table of Contents + :local: + :depth: 1 + +The following procedures create and apply the resources necessary to use cert-manager for the TLS certificates within a tenant. + +.. note:: + + The procedures use ``tenant-1`` as the name of the tenant. + + Replace the string ``tenant-1`` throughout the procedures to reflect the name of your tenant. + +1) Create the tenant namespace CA Issuer +---------------------------------------- + +Before deploying a new tenant, create a Certificate Authority and Issuer for the tenant's namespace. + +1. If necessary, create the tenant's namespace. + + .. code-block:: shell + :class: copyable + + kubectl create ns tenant-1 + +2. Request a Certificate for a new Certificate Authority with ``spec.isCA`` set to ``true``. + + Create a file called ``tenant-1-ca-certificate.yaml`` with the following contents: + + .. code-block:: yaml + :class: copyable + :emphasize-lines: 7,8 + + # tenant-1-ca-certificate.yaml + apiVersion: cert-manager.io/v1 + kind: Certificate + metadata: + name: tenant-1-ca-certificate + namespace: tenant-1 + spec: + isCA: true + commonName: tenant-1-ca + secretName: tenant-1-ca-tls + duration: 70128h # 8y + privateKey: + algorithm: ECDSA + size: 256 + issuerRef: + name: selfsigned-root + kind: ClusterIssuer + group: cert-manager.io + +3. Apply the file to the cluster. + + .. code-block:: shell + :class: copyable + + kubectl apply -f tenant-1-ca-certificate.yaml + +2) Create the ``Issuer`` +------------------------ + +The ``Issuer`` issues the certificates within the tenant namespace. + +1. Generate a resource definition for an ``Issuer``. + + Create a file called ``tenant-1-ca-issuer.yaml`` with the following contents: + + .. code-block:: yaml + :class: copyable + + # tenant-1-ca-issuer.yaml + apiVersion: cert-manager.io/v1 + kind: Issuer + metadata: + name: tenant-1-ca-issuer + namespace: tenant-1 + spec: + ca: + secretName: tenant-1-ca-tls + +2. Apply the ``Issuer`` resource definition to the cluster. + + .. code-block:: shell + :class: copyable + + kubectl apply -f tenant-1-ca-issuer.yaml + +3) Create a certificate for the tenant +-------------------------------------- + +Request that cert-manager issue a new TLS server certificate for MinIO. +The certificate must be valid for the following DNS domains: + +- ``minio.`` +- ``minio..svc`` +- ``minio..svc.`` +- ``*.-hl..svc.`` +- ``*..svc.`` +- ``*..minio..svc.'`` + +.. important:: + + Replace the filler strings with values for your tenant: + + - ```` is the internal root DNS domain assigned in your Kubernetes cluster. + Typically, this is ``cluster.local``, but confirm the value by checking your coredns configuration for the correct value for your Kubernetes cluster. + + For example: + + .. code-block:: shell + :class: copyable + + kubectl get configmap coredns -n kube-system -o jsonpath="{.data}" + + Different Kubernetes providers manage the root domain differently. + Check with your Kubernetes provider for more information. + + - ``tenant-name`` is the name provided to your tenant in the ``metadata.name`` of the Tenant YAML. + For this example it is ``myminio``. + + - ``namespace`` is the namespace where the tenant is created, the ``metadata.namespace`` notes that in the Tenant YAML. + For this example it is ``tenant-1``. + +1. Request a ``Certificate`` for the domains mentioned above + + Create a file called ``tenant-1-minio-certificate.yaml`` with the following contents: + + .. code-block:: yaml + :class: copyable + + # tenant-1-minio-certificate.yaml + apiVersion: cert-manager.io/v1 + kind: Certificate + metadata: + name: tenant-certmanager-cert + namespace: tenant-1 + spec: + dnsNames: + - "minio.tenant-1" + - "minio.tenant-1.svc" + - 'minio.tenant-1.svc.cluster.local' + - '*.minio.tenant-1.svc.cluster.local' + - '*.myminio-hl.tenant-1.svc.cluster.local' + - '*.myminio.minio.tenant-1.svc.cluster.local' + secretName: myminio-tls + issuerRef: + name: tenant-1-ca-issuer + + .. tip:: + + For this example, the Tenant name is ``myminio``. + We recommend naming the secret in the field ``spec.secretName`` as ``-tls``, following the naming convention the MinIO Operator uses when creating certificates without cert-manager. + +2. Apply the certificate resource to the cluster. + + .. code-block:: shell + :class: copyable + + kubectl apply -f tenant-1-minio-certificate.yaml + +4) Configure the tenant to use the certificate created by cert-manager +---------------------------------------------------------------------- + +To use cert-manager, the tenant spec must: + +- Disable AutoCert by setting the ``spec.requestAutoCert`` field to ``false``. + + This instructs the MinIO Operator to not attempt to issue certificates and instead rely on cert-manager to provide them in a secret. +- Reference the Secret containing the TLS certificate from the previous procedure in `spec.externalCertSecret`. + + +1. Modify the tenant YAML ``spec`` section to reflect the above requirements. + + .. code-block:: yaml + :emphasize-lines: 6,9,11 + + apiVersion: minio.min.io/v2 + kind: Tenant + metadata: + name: myminio + namespace: tenant-1 + spec: + ... + ## Disable default tls certificates. + requestAutoCert: false + ## Use certificates generated by cert-manager. + externalCertSecret: + - name: myminio-tls + type: cert-manager.io/v1 + ... + +5) Trust the tenant's CA in MinIO Operator +------------------------------------------ + +MinIO Operator can trust as many CA certificates as provided. + +To do this, create a secret with the prefix ``operator-ca-tls-`` followed by a unique identifier in the `minio-operator` namespace. + +MinIO Operator mounts and trusts **all** certificates issued by the provided Certificate Authorities. +This is required because the MinIO Operator performs health checks using the ``/minio/health/cluster`` endpoint. + +Create ``operator-ca-tls-tenant-1`` secret +++++++++++++++++++++++++++++++++++++++++++ + +Copy the tenant's cert-manager generated CA public key (``ca.crt``) into the `minio-operator` namespace. +This allows Operator to trust the cert-manager issued CA and all certificates derived from it. + +1. Create a ``ca.crt`` file containing the CA: + + .. code-block:: shell + :class: copyable + + kubectl get secrets -n tenant-1 tenant-1-ca-tls -o=jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt + +2. Create the secret: + + .. code-block:: shell + :class: copyable + + kubectl create secret generic operator-ca-tls-tenant-1 --from-file=ca.crt -n minio-operator + +.. tip:: + + In this example we chose a secret name of ``operator-ca-tls-tenant-1``. + We used the tenant namespace ``tenant-1`` as a suffix for easy identification of which namespace the CA comes from. + Use the name of your tenant namespace for easier linking secrets to the related resources. + +Deploy the tenant +----------------- + +With the Certificate Authority and ``Issuer`` in place for the tenant's namespace, you can now :ref:`deploy the object store tenant `. + +Use the tenant YAML modified above to disable AutoCert and reference the secret you generated. + From 41b50c0883442af41ef1f063741232f871d354d8 Mon Sep 17 00:00:00 2001 From: Daryl White <53910321+djwfyi@users.noreply.github.com> Date: Tue, 17 Sep 2024 16:10:00 -0400 Subject: [PATCH 04/11] Build error and adding reference to deployment page --- source/includes/k8s/deploy-operator.rst | 6 +++ .../cert-manager/cert-manager-tenants.rst | 53 +++++++++++-------- 2 files changed, 36 insertions(+), 23 deletions(-) diff --git a/source/includes/k8s/deploy-operator.rst b/source/includes/k8s/deploy-operator.rst index 246d34fa..58227cc1 100644 --- a/source/includes/k8s/deploy-operator.rst +++ b/source/includes/k8s/deploy-operator.rst @@ -162,6 +162,12 @@ The output of the example command above may differ from the output in your termi Alternatively, you can generate x.509 TLS certificates signed by a known and trusted CA and pass those certificates to MinIO Tenants. See :ref:`minio-tls` for more complete documentation. +Certificate Management with cert-manager +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Rather than then MinIO Operator managing certificates, you can configure the deployment to use `cert-manager `__. +For instructions for deploying the MinIO Operator and tenants using cert-manager, refer to the :ref:`cert-manager page `. + Procedure --------- diff --git a/source/operations/cert-manager/cert-manager-tenants.rst b/source/operations/cert-manager/cert-manager-tenants.rst index 35ba162f..ce8d88d4 100644 --- a/source/operations/cert-manager/cert-manager-tenants.rst +++ b/source/operations/cert-manager/cert-manager-tenants.rst @@ -18,6 +18,14 @@ The following procedures create and apply the resources necessary to use cert-ma Replace the string ``tenant-1`` throughout the procedures to reflect the name of your tenant. +Prerequisites +------------- + +- `kustomize `__ installed +- ``kubectl`` access to your ``k8s`` cluster +- Completed the steps to :ref:`set up cert-manager ` +- The MinIO Operator installed and :ref:`set up for cert-manager `. + 1) Create the tenant namespace CA Issuer ---------------------------------------- @@ -166,8 +174,8 @@ The certificate must be valid for the following DNS domains: kubectl apply -f tenant-1-minio-certificate.yaml -4) Configure the tenant to use the certificate created by cert-manager ----------------------------------------------------------------------- +4) Configure the tenant to use the certificate +---------------------------------------------- To use cert-manager, the tenant spec must: @@ -176,26 +184,25 @@ To use cert-manager, the tenant spec must: This instructs the MinIO Operator to not attempt to issue certificates and instead rely on cert-manager to provide them in a secret. - Reference the Secret containing the TLS certificate from the previous procedure in `spec.externalCertSecret`. - -1. Modify the tenant YAML ``spec`` section to reflect the above requirements. +Modify the tenant YAML ``spec`` section to reflect the above requirements. - .. code-block:: yaml - :emphasize-lines: 6,9,11 - - apiVersion: minio.min.io/v2 - kind: Tenant - metadata: - name: myminio - namespace: tenant-1 - spec: - ... - ## Disable default tls certificates. - requestAutoCert: false - ## Use certificates generated by cert-manager. - externalCertSecret: - - name: myminio-tls - type: cert-manager.io/v1 - ... +.. code-block:: yaml + :emphasize-lines: 6,9,11 + + apiVersion: minio.min.io/v2 + kind: Tenant + metadata: + name: myminio + namespace: tenant-1 + spec: + ... + ## Disable default tls certificates. + requestAutoCert: false + ## Use certificates generated by cert-manager. + externalCertSecret: + - name: myminio-tls + type: cert-manager.io/v1 + ... 5) Trust the tenant's CA in MinIO Operator ------------------------------------------ @@ -233,8 +240,8 @@ This allows Operator to trust the cert-manager issued CA and all certificates de We used the tenant namespace ``tenant-1`` as a suffix for easy identification of which namespace the CA comes from. Use the name of your tenant namespace for easier linking secrets to the related resources. -Deploy the tenant ------------------ +6) Deploy the tenant +-------------------- With the Certificate Authority and ``Issuer`` in place for the tenant's namespace, you can now :ref:`deploy the object store tenant `. From 4e2506c424cdb4b25ea4954379098c1b563c0f26 Mon Sep 17 00:00:00 2001 From: Daryl White <53910321+djwfyi@users.noreply.github.com> Date: Tue, 17 Sep 2024 16:20:34 -0400 Subject: [PATCH 05/11] Some missed PR feedback newly applied --- source/operations/cert-manager.rst | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/source/operations/cert-manager.rst b/source/operations/cert-manager.rst index 3c2a9454..6184421f 100644 --- a/source/operations/cert-manager.rst +++ b/source/operations/cert-manager.rst @@ -13,10 +13,12 @@ cert-manager TLS certificate management with cert-manager -------------------------------------------- -`cert-manager `__ manages certificates within Kubernetes clusters. +This guide shows you how to install `cert-manager `__ and set up the global ``Cluster Issuer``. + +cert-manager manages certificates within Kubernetes clusters. The MinIO Operator supports using cert-manager for certificates as an alternative to the MinIO Operator managing certificates for itself and its tenants. -cert-manager obtains valid certificates from an ``Issuer`` or ``ClusterIssuer`` and can renew certificates prior to expiration. +cert-manager obtains valid certificates from an ``Issuer`` or ``ClusterIssuer`` and can automatically renew certificates prior to expiration. A ``ClusterIssuer`` issues certificates for multiple namespaces. An ``Issuer`` only mints certificates for its own namespace. @@ -33,7 +35,6 @@ The following graphic depicts how various namespaces make use of either an ``Iss :alt: A graphic depiction of a kubernetes cluster with five separate namespaces represented by different boxes. One is labeled minio-operator with a text box that says "minio-operator: issuer". A second is labeled tenant-1 with text box that says "tenant-1: issuer". A third is labeled default with a text box that says "root:ClusterIssuer". A fourth is labeled tenant-2 with a text box that says "tenant-2: issuer". The fifth is labeled cert-manager with a text box that says "minio-operator: issuer". :align: center -This guide shows you how to install cert-manager and set up the global ``Cluster Issuer``. .. note:: From 0134e9d6c04102960492d55f48385b96f26ba150 Mon Sep 17 00:00:00 2001 From: Daryl White <53910321+djwfyi@users.noreply.github.com> Date: Wed, 18 Sep 2024 10:49:48 -0400 Subject: [PATCH 06/11] Applying additional PR feedback --- source/includes/k8s/deploy-operator.rst | 2 +- source/operations/cert-manager.rst | 2 +- .../cert-manager/cert-manager-operator.rst | 14 +++++++------- .../cert-manager/cert-manager-tenants.rst | 9 ++++++--- source/operations/network-encryption.rst | 2 +- 5 files changed, 16 insertions(+), 13 deletions(-) diff --git a/source/includes/k8s/deploy-operator.rst b/source/includes/k8s/deploy-operator.rst index 58227cc1..a5991c72 100644 --- a/source/includes/k8s/deploy-operator.rst +++ b/source/includes/k8s/deploy-operator.rst @@ -165,7 +165,7 @@ The output of the example command above may differ from the output in your termi Certificate Management with cert-manager ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Rather than then MinIO Operator managing certificates, you can configure the deployment to use `cert-manager `__. +Rather than the MinIO Operator managing certificates, you can configure the deployment to use `cert-manager `__. For instructions for deploying the MinIO Operator and tenants using cert-manager, refer to the :ref:`cert-manager page `. Procedure diff --git a/source/operations/cert-manager.rst b/source/operations/cert-manager.rst index 6184421f..d759436c 100644 --- a/source/operations/cert-manager.rst +++ b/source/operations/cert-manager.rst @@ -32,7 +32,7 @@ The following graphic depicts how various namespaces make use of either an ``Iss .. image:: /images/k8s/cert-manager-cluster.svg :width: 600px - :alt: A graphic depiction of a kubernetes cluster with five separate namespaces represented by different boxes. One is labeled minio-operator with a text box that says "minio-operator: issuer". A second is labeled tenant-1 with text box that says "tenant-1: issuer". A third is labeled default with a text box that says "root:ClusterIssuer". A fourth is labeled tenant-2 with a text box that says "tenant-2: issuer". The fifth is labeled cert-manager with a text box that says "minio-operator: issuer". + :alt: A Kubernetes cluster with five namespaces, shown as a box for each namespace in the cluster. The minio-operator namespace contains a "minio-operator: issuer" local issuer. The default namespace contains a "root: ClusterIssuer" cluster issuer. The cert-manager namespace contains a "minio-operator: issuer" local issuer. The remaining two namespaces are individual tenants, "tenant-1" and "tenant-2", each with its own local issuer. :align: center diff --git a/source/operations/cert-manager/cert-manager-operator.rst b/source/operations/cert-manager/cert-manager-operator.rst index 22de9edd..a387f876 100644 --- a/source/operations/cert-manager/cert-manager-operator.rst +++ b/source/operations/cert-manager/cert-manager-operator.rst @@ -11,7 +11,7 @@ cert-manager for Operator :depth: 1 -MinIO Operator manages the TLS certificate issuing for the services hosted in the ``minio-operator`` namespace. +MinIO Operator manages TLS certificate issuing for the services hosted in the ``minio-operator`` namespace. This page describes how to manage the Operator's TLS certificates with :ref:`cert-manager `. @@ -30,12 +30,12 @@ Prerequisites This guide **disables** the automatic generation of certificates in MinIO Operator and issues certificates using cert-manager instead. -The ``minio-operator`` namespace needs to have its own certificate authority (CA), derived from the cluster's ``ClusterIssuer`` certificate created during :ref:`cert-manager setup `. +The ``minio-operator`` namespace must have its own certificate authority (CA), derived from the cluster's ``ClusterIssuer`` certificate created during :ref:`cert-manager setup `. Create this CA certificate using cert-manager. .. important:: - This CA certificate should exist *before* installing MinIO Operator. + This CA certificate **must** exist *before* installing MinIO Operator. 1. If it does not exist, create the ``minio-operator`` namespace @@ -128,7 +128,7 @@ The certificate from cert-manager must be valid for the following DNS domains: Replace ```` with the actual value for your MinIO tenant. ``cluster domain`` is the internal root DNS domain assigned in your Kubernetes cluster. - Typically, this is ``cluster.local``, but confirm the value by checking your coredns configuration for the correct value for your Kubernetes cluster. + Typically, this is ``cluster.local``, but confirm the value by checking your CoreDNS configuration for the correct value for your Kubernetes cluster. For example: @@ -181,7 +181,7 @@ This creates a secret called ``sts-tls`` in the ``minio-operator`` namespace. .. warning:: - Failing to provide the ``sts-tls`` secret containing the TLS certificate or providing an invalid ``key-value`` pair in the secret will prevent the STS service from starting. + The STS service will not start if the ``sts-tls`` secret, containing the TLS certificate, is missing or contains an invalid ``key-value`` pair. 4) Install Operator with Auto TLS disabled ------------------------------------------ @@ -191,9 +191,9 @@ You can now :ref:`install the MinIO Operator `. When installing the Operator deployment, set the ``OPERATOR_STS_AUTO_TLS_ENABLED`` environment variable to ``off`` in the ``minio-operator`` container. Disabling this environment variable prevents the MinIO Operator from issuing the certificates. -Instead, Operator waits for the TLS certificate issued by cert-manager. +Instead, Operator waits for cert-manager to issue the TLS certificate. -There are several for defining an environment variable depending on how you install the Operator. +There are various methods to define an environment variable depending on how you install the Operator. The steps below define the variable with kustomize. 1. Create a kustomization patch file called ``kustomization.yaml`` with the below contents: diff --git a/source/operations/cert-manager/cert-manager-tenants.rst b/source/operations/cert-manager/cert-manager-tenants.rst index ce8d88d4..c1d0556e 100644 --- a/source/operations/cert-manager/cert-manager-tenants.rst +++ b/source/operations/cert-manager/cert-manager-tenants.rst @@ -38,6 +38,8 @@ Before deploying a new tenant, create a Certificate Authority and Issuer for the kubectl create ns tenant-1 + This much match the value of the ``metadata.namespace`` field in the tenant's YAML. + 2. Request a Certificate for a new Certificate Authority with ``spec.isCA`` set to ``true``. Create a file called ``tenant-1-ca-certificate.yaml`` with the following contents: @@ -116,10 +118,10 @@ The certificate must be valid for the following DNS domains: .. important:: - Replace the filler strings with values for your tenant: + Replace the the placeholder text (marked with the ``<`` and ``>`` characters) with values for your tenant: - ```` is the internal root DNS domain assigned in your Kubernetes cluster. - Typically, this is ``cluster.local``, but confirm the value by checking your coredns configuration for the correct value for your Kubernetes cluster. + Typically, this is ``cluster.local``, but confirm the value by checking your CoreDNS configuration for the correct value for your Kubernetes cluster. For example: @@ -134,7 +136,8 @@ The certificate must be valid for the following DNS domains: - ``tenant-name`` is the name provided to your tenant in the ``metadata.name`` of the Tenant YAML. For this example it is ``myminio``. - - ``namespace`` is the namespace where the tenant is created, the ``metadata.namespace`` notes that in the Tenant YAML. + - ``namespace`` is the value created earlier where the tenant will be installed. + In the tenant YAML, it is defined in the the ``metadata.namespace`` field. For this example it is ``tenant-1``. 1. Request a ``Certificate`` for the domains mentioned above diff --git a/source/operations/network-encryption.rst b/source/operations/network-encryption.rst index d526b5be..1f2fa78f 100644 --- a/source/operations/network-encryption.rst +++ b/source/operations/network-encryption.rst @@ -67,7 +67,7 @@ Enabling TLS Certificate Management with cert-manager ---------------------------------------- - Rather than then MinIO Operator managing certificates, you can configure the deployment to use `cert-manager `__. + Rather than the MinIO Operator managing certificates, you can configure the deployment to use `cert-manager `__. For instructions for deploying the MinIO Operator and tenants using cert-manager, refer to the :ref:`cert-manager page `. From ff4e5f804aa8fbfa37a8edf7d3b57a67789f3bba Mon Sep 17 00:00:00 2001 From: Daryl White <53910321+djwfyi@users.noreply.github.com> Date: Fri, 20 Sep 2024 16:05:21 -0400 Subject: [PATCH 07/11] Applying another round of PR feedback --- source/operations/cert-manager.rst | 23 +++++---- .../cert-manager/cert-manager-operator.rst | 13 +++-- .../cert-manager/cert-manager-tenants.rst | 50 +++++++++++++------ source/operations/network-encryption.rst | 2 +- 4 files changed, 59 insertions(+), 29 deletions(-) diff --git a/source/operations/cert-manager.rst b/source/operations/cert-manager.rst index d759436c..7c12c03c 100644 --- a/source/operations/cert-manager.rst +++ b/source/operations/cert-manager.rst @@ -13,10 +13,20 @@ cert-manager TLS certificate management with cert-manager -------------------------------------------- -This guide shows you how to install `cert-manager `__ and set up the global ``Cluster Issuer``. +This guide shows you how to install cert-manager for TLS certificate management. +The guide assumes a new or fresh MinIO Operator installation. + +.. note:: + + This guide uses a self-signed ``Cluster Issuer``. + You can also use `other Issuers supported by cert-manager `__. + + The main difference is that you must provide that ``Issuer`` CA certificate to MinIO, instead of the CA's mentioned in this guide. + +Defer to the `cert-manager documentation `__ and your own organization's certificate requirements for more advanced configurations. cert-manager manages certificates within Kubernetes clusters. -The MinIO Operator supports using cert-manager for certificates as an alternative to the MinIO Operator managing certificates for itself and its tenants. +The MinIO Operator supports using cert-manager for managing and provisioning certificates as an alternative to the MinIO Operator managing certificates for itself and its tenants. cert-manager obtains valid certificates from an ``Issuer`` or ``ClusterIssuer`` and can automatically renew certificates prior to expiration. @@ -36,13 +46,6 @@ The following graphic depicts how various namespaces make use of either an ``Iss :align: center -.. note:: - - This guide uses a self-signed ``Cluster Issuer``. - You can also use `other Issuers supported by cert-manager `__. - The main difference is that you must provide the ``Issuer`` CA certificate to MinIO, instead of the CA's mentioned in this guide. - - Prerequisites ------------- @@ -68,6 +71,8 @@ The following command installs version 1.12.13 using ``kubectl``. `Release 1.12.X LTS `__ is preferred, but you may install the latest version. For more details on installing cert-manager, see their `installation instructions `__. +.. _minio-cert-manager-create-cluster-issuer: + Create a self-signed Cluster Issuer for the cluster ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/source/operations/cert-manager/cert-manager-operator.rst b/source/operations/cert-manager/cert-manager-operator.rst index a387f876..d11a58ef 100644 --- a/source/operations/cert-manager/cert-manager-operator.rst +++ b/source/operations/cert-manager/cert-manager-operator.rst @@ -73,7 +73,12 @@ Create this CA certificate using cert-manager. kind: ClusterIssuer group: cert-manager.io -3. Apply the resource to the cluster + .. important:: + + The ``spec.issueRef.name`` must match the name of the ``ClusterIssuer`` created when :ref:`setting up cert-manager `. + If you specified a different ``ClusterIssuer`` name or are using a different ``Issuer`` from the guide, modify the ``issuerRef`` to match your environment. + +3. Apply the resource: .. code-block:: shell :class: copyable @@ -107,7 +112,7 @@ Use the secret created above to add an ``Issuer`` resource for the ``minio-opera secretName: operator-ca-tls -2. Apply the resource to the cluster +2. Apply the resource: .. code-block:: shell @@ -170,7 +175,7 @@ The certificate from cert-manager must be valid for the following DNS domains: The secret name **must** be ``sts-tls``. Confirm this by setting ``spec.secretName: sts-tls`` as highlighted above. -2. Apply the resource to the cluster: +2. Apply the resource: .. code-block:: shell :class: copyable @@ -191,7 +196,7 @@ You can now :ref:`install the MinIO Operator `. When installing the Operator deployment, set the ``OPERATOR_STS_AUTO_TLS_ENABLED`` environment variable to ``off`` in the ``minio-operator`` container. Disabling this environment variable prevents the MinIO Operator from issuing the certificates. -Instead, Operator waits for cert-manager to issue the TLS certificate. +Instead, Operator relies on cert-manager to issue the TLS certificate. There are various methods to define an environment variable depending on how you install the Operator. The steps below define the variable with kustomize. diff --git a/source/operations/cert-manager/cert-manager-tenants.rst b/source/operations/cert-manager/cert-manager-tenants.rst index c1d0556e..c6a71b84 100644 --- a/source/operations/cert-manager/cert-manager-tenants.rst +++ b/source/operations/cert-manager/cert-manager-tenants.rst @@ -67,14 +67,20 @@ Before deploying a new tenant, create a Certificate Authority and Issuer for the kind: ClusterIssuer group: cert-manager.io -3. Apply the file to the cluster. + .. important:: + + The ``spec.issueRef.name`` must match the name of the ``ClusterIssuer`` created when :ref:`setting up cert-manager `. + If you specified a different ``ClusterIssuer`` name or are using a different ``Issuer`` from the guide, modify the ``issuerRef`` to match your environment. + + +3. Apply the resource: .. code-block:: shell :class: copyable kubectl apply -f tenant-1-ca-certificate.yaml -2) Create the ``Issuer`` +4) Create the ``Issuer`` ------------------------ The ``Issuer`` issues the certificates within the tenant namespace. @@ -96,7 +102,7 @@ The ``Issuer`` issues the certificates within the tenant namespace. ca: secretName: tenant-1-ca-tls -2. Apply the ``Issuer`` resource definition to the cluster. +2. Apply the ``Issuer`` resource definition: .. code-block:: shell :class: copyable @@ -142,7 +148,8 @@ The certificate must be valid for the following DNS domains: 1. Request a ``Certificate`` for the domains mentioned above - Create a file called ``tenant-1-minio-certificate.yaml`` with the following contents: + Create a file called ``tenant-1-minio-certificate.yaml``. + The contents of the file should resemble the following, modified to reflect your cluster and tenant configurations: .. code-block:: yaml :class: copyable @@ -168,27 +175,39 @@ The certificate must be valid for the following DNS domains: .. tip:: For this example, the Tenant name is ``myminio``. - We recommend naming the secret in the field ``spec.secretName`` as ``-tls``, following the naming convention the MinIO Operator uses when creating certificates without cert-manager. + We recommend naming the secret in the field ``spec.secretName`` as ``-tls`` as a naming convention. -2. Apply the certificate resource to the cluster. +2. Apply the certificate resource: .. code-block:: shell :class: copyable kubectl apply -f tenant-1-minio-certificate.yaml -4) Configure the tenant to use the certificate ----------------------------------------------- +3. Validate the changes took effect: -To use cert-manager, the tenant spec must: + .. code-block:: shell + :class: copyable -- Disable AutoCert by setting the ``spec.requestAutoCert`` field to ``false``. + kubectl describe secret/myminio-tls -n tenant-1 - This instructs the MinIO Operator to not attempt to issue certificates and instead rely on cert-manager to provide them in a secret. -- Reference the Secret containing the TLS certificate from the previous procedure in `spec.externalCertSecret`. + .. note:: + + - Replace ``tenant-1`` with the namespace for your tenant. + - Replace ``myminio-tls`` with the name of your secret, if different. + +4) Deploy the tenant using cert-manager for TLS certificate management +---------------------------------------------------------------------- + +When deploying a Tenant, you must set the TLS configuration such that: + +- The Tenant does not automatically generate its own certificates (``spec.requestAutoCert: false``) *and* +- The Tenant has a valid cert-manager reference (``spec.externalCertSecret``) + +This directs the Operator to deploy the Tenant using the cert-manager certificates exclusively. + +The following YAML ``spec`` provides a baseline configuration meeting these requirements: -Modify the tenant YAML ``spec`` section to reflect the above requirements. - .. code-block:: yaml :emphasize-lines: 6,9,11 @@ -210,7 +229,8 @@ Modify the tenant YAML ``spec`` section to reflect the above requirements. 5) Trust the tenant's CA in MinIO Operator ------------------------------------------ -MinIO Operator can trust as many CA certificates as provided. +The MinIO Operator does not trust the tenant's CA by default. +To trust the tenant's CA, you must pass the certificate to the Operator as a secret. To do this, create a secret with the prefix ``operator-ca-tls-`` followed by a unique identifier in the `minio-operator` namespace. diff --git a/source/operations/network-encryption.rst b/source/operations/network-encryption.rst index 1f2fa78f..101c5fee 100644 --- a/source/operations/network-encryption.rst +++ b/source/operations/network-encryption.rst @@ -67,7 +67,7 @@ Enabling TLS Certificate Management with cert-manager ---------------------------------------- - Rather than the MinIO Operator managing certificates, you can configure the deployment to use `cert-manager `__. + The MinIO Operator supports using `cert-manager `__ as a full replacement for its built-in automatic certificate management *or* user-driven manual certificate management. For instructions for deploying the MinIO Operator and tenants using cert-manager, refer to the :ref:`cert-manager page `. From c5611d2f9f3244b8be4f81bd7e569fb59896a6cf Mon Sep 17 00:00:00 2001 From: Daryl White <53910321+djwfyi@users.noreply.github.com> Date: Fri, 20 Sep 2024 16:21:29 -0400 Subject: [PATCH 08/11] Adding a section on migrating existing Operator to use cert-manager --- .../operations/cert-manager/cert-manager-operator.rst | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/source/operations/cert-manager/cert-manager-operator.rst b/source/operations/cert-manager/cert-manager-operator.rst index d11a58ef..70ed9a2f 100644 --- a/source/operations/cert-manager/cert-manager-operator.rst +++ b/source/operations/cert-manager/cert-manager-operator.rst @@ -238,6 +238,17 @@ The steps below define the variable with kustomize. kubectl apply -k minio-operator +Migrate an existing MinIO Operator deployment to cert-manager +------------------------------------------------------------- + +To transition an existing MinIO Operator deployment from using AutoCert to cert-manager, complete the following steps: + +1. Complete the steps for :ref:`installing cert-manager `, including disabling auto-cert. +2. Complete steps 1-3 on this page to generate the certificate authority for the Operator. +3. When you get to the install step on this page, instead replace the existing Operator TLS certificate with the cert-manager issued certificate. +4. Create new cert-manager certificates for each tenant, similar to the steps described on the :ref:`cert-manager for Tenants ` page. +5. Replace the secrets in the MinIO Operator namespace for the tenants with secrets related to each tenant's cert-manager issued certificate. + Next steps ---------- From 01c1958f503498d37ff81daa679cba65b88aa3ac Mon Sep 17 00:00:00 2001 From: Daryl White <53910321+djwfyi@users.noreply.github.com> Date: Mon, 30 Sep 2024 13:59:44 -0400 Subject: [PATCH 09/11] Updating graphic with new image --- source/images/k8s/cert-manager-graph.png | Bin 0 -> 54513 bytes source/images/k8s/cert-manager-graph.svg | 1 + source/operations/cert-manager.rst | 14 +++++++------- 3 files changed, 8 insertions(+), 7 deletions(-) create mode 100644 source/images/k8s/cert-manager-graph.png create mode 100644 source/images/k8s/cert-manager-graph.svg diff --git a/source/images/k8s/cert-manager-graph.png b/source/images/k8s/cert-manager-graph.png new file mode 100644 index 0000000000000000000000000000000000000000..1730e57d1aee6e45a0a409ad8935e834aa2a7470 GIT binary patch literal 54513 zcmeFZcT`i`*ESmUC`S=Fs1!j!u^>tj=`|LLiWIRRAVj4~M|w$uauk%NBA|3Znl$M> z;Gsze=>!N8dI%x31k%2>!S{QAcZ~b*9rwQ9J$noe`|Q=`n&p|#Tzlhd13hhy1N;YI zFc`;;>l%hI7~3)ow(pNW_kkx{HSbTuU?*TVG_KtBj$a%zbBLb(B)XhEyl8jh)nG#C zgRr}Cg@!|W9D3D;#HByJ)-Gyg6K0ES{lcdeX=YnmDwp-`XQRf#)pWy)kw0!<`bT5` zbwS;W68HTM-#RgJ@4ECy2fb3J@@KZ!#OwXZX-I_pV}0wPJx})GDZPBP3d^(`_ZnAy zYPwdbC$?}D4>W$H#C{hL3%c$O?&+7@xrM?0n%J{*d-?Dm)ZY)^!gi_uzCC?8fb;k5 z{{QZ}``@nr&uBm>$8#6DdO9C-jEjXC@%XKI7X*#r9Kj;8@~n5=<6Or3db4hqg#2!t zz-+i`6yGg^Nv!=g@QGc!{eL%J4}b~(zCC=Rj&0{Q;Ez9-|LyvJqy|(U5O)J3C&!}v z=+4eG{`w=euY-dNy3+AHf}hBK$=ST3Qj}E>W42jJ((nxE`^|eh zC|uZT`nGQ6WFwAs)oz5BZgEqm$9Y4RSa=C^$@M7EuWr&Od1NniQdEI_Z=Z*P=)$aG+kMT)O5$LTDmD}gR5SJL4Jy2N8h|#9| z72zmp6ulSGV3FWwL0^@+nYh|+k+2FAy40=Q3$E_gN5um~@LS>ZggWCcgZ9=cNxJ!ykBhd1K z&~sWD*lrzTXS?jDffU|0%eSh5+IsK{(Q1abf2j4Qu;x>WDETiQadhN9taV z)j_Zr32AmMD{c2Pgjv{~r0|BWPJcs}&F7|!4Jz*bU40BbNECGhjUjcs$%C+6j0=MJ zy4}j_^VeHowJ1$T|7CDzpPP#~=Ir4RF=7?*0m%aYZJ;O=Z~u@Z;ID;QYrO9mD|W-P zxUY{j9qCh>B(2`B+HLBhK%YY^4YAMA*V)_I7#yHsC$D%yGfa znLv**QD}%3n{+zV38&6--GmHxC|nAZHsDw zHv<#2UyV7+HE8plwWf`Whz*K4s+Q+8e2N@7TkEH77aZ(e(uYmXX$G{StNY_6YK#G&1L^XwfHn@h3Trc0F!j{SzKL8-dvqjG9aZu{y7Cy)lxwKWm&f9=641vNp=zB*xJe z>{P2rxk|H1!nJ%LVw)<+A!i12(T^>3s{K|{f6VN!61d+7S$Otd{g6#BdGb4D(#A$z z>=?$`=ZS8$Cw(gC`9Uy{M{V|gln}%Ty!LEc>F8~{F=|Qn{ie#Ts5612V{nz#8{@?3 z91a%Uo;1V3vZ2%4MM1EX&TXxK3W{#%jmQm8n$1d$U)lv%rJDEfcSayWSxm3m&{}4j zO^EsICl)mUa`yKWVA=rM>+KBo<{Y<%kqD)~--;ge?-wmq2>`SOnCXpImkROb>1 zI_)bF!PE(K)S=E(A^;@A?wq;0QL=tv*1`V4ZPP*qtE=|{{H~toF?T^f(edP9Wk33T9}2--wd#2jWLNLJi*96d&)zH<6yb5(%A0S zW948ZjB;cLgCCO;RgXvs`R7kC4P|;j_}=MZbn1UN)J0%+2zM|kh4Aw(Qyvxhc{4;I zgCD>=Y4+!SK7k-?Nt^S26*N`3kuIY0xYS0C(KC79`rTa``KN8A2B|iPyQt1tga)IG zy=zfQ&KBzWTj26N5P-0{E4;4cdE|w9u8FmdJ%9cO=-?uC1cHpqT+oirvVN_^6T9fm z{hBs0HbN$c0kVpfFCov|EE$3b#S+K%_Rm4y(t_R10{FKL_=&XC0S@-V6H@LWj7G!S zlw}yy`%xa-*KODy4TJ3d>oxAES~{=}*^uz+{owh25%`^72k9zYRE1pM;z_qloZR_- z^rHFAXdn-aa(Y``?Y$vvEXL%URt?WnJDJExR%!T zj`7UX8`CZ4pKqnxVzDBj3l@h_WOj`8YA`%W+^1b9w+qTf2bSP)RKb)uT{x=e4O}Dz zV8xlhjnyM=fvZ_MF%|&4Q)ZE18sh|*oshX<;VXdHAFB1T{J@NLH9JC;Jzn@Z4IKN^ z`(}o%4jG2In`~CK_|E&W1^E-5i_6)Aso7$I@;r+k47iGQ&ij>iY2V>$0A@ftoI~*P z4p=Sbg(nkC=u)=L(d!(2krS3J$JR^}AKY>0*!0@h^`W{8!I~~2KYTv0CdVoDx=guD z8w7iZIV9nWTml+3bG3yZQ~Nn>?(z%J7!!|7qAe^qcGXE`Xu`qNqmB?a-Cm;&>j`6{>75Y04W^+9-jO?!OnN(YIZ2eQ9j~2-(h{o41Ag=T$Uy_>;a=R>=P~e-sdDW?#EffXP{14J9Y8tqX}okcWo(rzMDpw!MR@zRmp ze^D63Z__GzRnNG&Q#4qrR0GoHD91Ku`gVEz82wzjl$oUOEk;@GJr_Y*FyXt0(iOim zO>mY-;uyucrF2~e@QrXAN=-@fNz_tyHhB?K_T+2B>L-w8k{ZC0cikg4-619x@L`+b zKiX9z?@~A=4YwOJaA$RO`5$J@*9x9f;!8fBp)Qzpt~yEj$fOZ}%B&2uUmN(sYrqsQ zD3gNNaM|8}JvWBx!O@N#0RvE|naKCwY3u9wkRYl>l5TSnqb$)IIT7g7@plr&U5UAB zPik7tuQ7G?hG)ML9y8~FFZxL_7q!)_x~jXpSRm9tAoW1RoX0?@`DIdIFwZR8ZoAwu z5t&T=$^-OKhs?hFmI;#P`9nz{A2oqw0W<>bweyh=OX=b2@_X;Cf_^e!Xci!;LMf8u zicdmw;ekMGEPG3H`yVl;*XrMlWJ>(?K_4LEtf-`VMVHOiGI4f;J8i&c^B0Coc(#83@4ts9|7zerH30b(aE;54@u{iO z(#rgIA|tuDq*Y~)u7`!a%gTCvOINo~L0;>Q+iG=wQk%7fzq@U2fx7tA)BN+M zo=LkbK5!VWRx(j=d^4|FlnG_3>q*C;e6XJn%$^q>{*h0hDwpt3(MvM%VXeCPa4p;f zJHnG$No&50Mjt1Ul1g$h1=1Tt93t%v81|dTC$~0fYo~hjrkB5OllZX?mg?yrKE`#j zA-}tCYAXH=be0O^34vI8@|#u;k}#soX&`oOM|CE}9VjCk8%sO8$*eE|-W+qN^U`sM zm0jlc@NhqRkQ8HmiQLL$ko%e--M09(17X%S){Y)r@$ztQc=_(#i&=>R} zWf>V0u@rr$p}%;wv{FiPb8V^PBLh73r%n}6`i)kXYTvgtV$}S2zU-_fzOx!+1p&50 zQ)7db9`?tTe2Y_zS5>fBi)K#LTQ6K)frYf1l2X1hq_+pE+nT=n{nxdi6(>Z0_g^(g z2`{L9lASK>&?SDuRu8m>&ELHnKrICFaRE_Z9A41ZHn!G@NFU@VkgEqo|1uzulw zdkx~+wRm+u&(4ispFa;3`mDwin{jA(Y;4@>cst!Gt!B25gfbe~tWi^w?#2oC z-nnxo!YsC>p>tznXm~tP;Rlg8FE+IQ^M_{(f*?A^6B7&V%z(Ph#>OCjuo|l!a%;CL z>cRFN(H=*skSaFqq}Ie+ky7>7u0aoZI}O* zF4RvPGi1UnTIlRa^$I393Tio!^Q?Y?p#z;I;NkC5OzMUXyEDXcP1u8;n4Bm0o}Nn z_N%p#&xRd}okotDB_vo4Doe}rsFXQJ2x3b+=Fb^qf#2K*Nu<9)6arPb4gW$mtfvV= zVYqLl{m(8N^)B}r^@x1)noJT}9Zb~No}O1n%pRNLiR^3bo&EzJF>a=51SJGMW?#-=2&IZ-~Dw@f>!_8!I7 z;g(2o06f(Bo+Q?i)+Q#ZaVJ5TuIE{r_unQ_c$|`k?8#GxD;uQQ! zjo(!EpELMA;q~;an?(&^)q9K%aEODUx7}Sd0rGvfky|a3S!FhBYp;&a-jw7u6BP7` zd7oJderwi#y2rtxkf{6F#v9lt0bANQUu1JZhzi{0P7fJKJ*u>g%9_zq0XnYtUCtla zTgb)bhd|1CmX>u5tEP*`IvPqhHhtRYudRaODtqBSx7?E~N#C$r%zzMqqQo}N7d zT(lr9*YSRNY@0QH`SUebYWHWQ5_15qhlG_sP$=APs~$p#GRee|a@PUMz05rg{>Q~G zK1lAaTm=^Lbim5bB7uG= zv=8CzN(=)6DAB^J8J+;NJF^o(fLnN5o(7WGAY}k%lSZZ$z?PQecPv@_wkV0jyBU2v zIQab)_d9NGi`4I&Pk@b}6ancAPpYkbv}+^xz)NCKgpkBB)PRC~(*+-ET|~?DjNC*r zW#aoBk&sQTzByGmT#ZGD{Wae8aXl$OpB=Re70Gx{bTX-eQs%barULa14n ztkDW`3e5may-X=$vsC~*&B=%V2oXbcs~0U^QCbiX$oDc)^mG{504anUNy+Bt|C<6- z^!g%nY98=~n}5_rg*eSSvfOPj{;ALO{K22Adr48Aj^`BUHFtP?U%$9ov^gu&rh2Br z!E<0_Rpz{)%UcpsUtjm0>~dP{9+0nLA9bN{zx?w+$0K1yPjTSFJQ+tG^r|9$n420I zkt=^~?z`}?W%~R93*PLwYo95XfLrQj8v$~`a`e#uczcclQ4hdRZcZ|~v_GfqIxiS- zdv5b+B}#qT^EFstqO%tQ;S;+6x{u89E`tnk+tf%Qqr_hW0<9+K=jW}nV6&-N6-vPK zQZWSubIObCxdsMiPh$eE4jvO0l%UZb)C(^J(kM61rht1P=bJ&one?^?tw5@Pwe7;3=iOTr z3OMkSwwaE-6yP{ce^-D!@E4csDMcqo`zpLl=xT>&Vq$FN?Y0^_9FqZsL>V_d^mRdj zF%2R)YfWL~zCqT7X5b^_sjYRN7_pDitIP=C<{A%40T)-9r1aXVms{59+vU60neDfO zhYjYHXDM+&az!SSRv;TGMvDhiDz0o`$!kgU8zAM}y{+Tg8VPvm^CO~@WDi2CIs}S) zn!_7AiP+Sq>kEY@6NA7bp{V`(>Qx~L0t~1*x+4xW+o3JMHb=R)R}B$I42eCI%moUT z3=M6_8>_l-VC7U^S0`#qpLu%U7L8=!LCOF0S1pK{H@i222`6dLBH5{_Qw2bMu3L-4 zL$rl#U^gQ&veaRGkpaltTh;BY@E9X4@bT-_`BTqaeOwHU%RhU6<}|g#Fgou{xP$aApn*IAVUHIhF_a_6dDoU1Y)3PV@n2P z*Hd@y6rDiZc(sCk@=hsx^%g?(j}A@^gOAMp`Cs1xfTIQxKRbMr0Kpx%2=JQyu=o-dBskxrLxo2?6q0jcr`{@E{6s~Rr2ZJQd zC&XXLRnB{0jZ;|pC;&4c!<_--g7dLp;4%D15a>OP<19d5-~YMH&gi6(p zuWr`W*F95Usrgy*FR?y=VVBppgW+qkf&Dt5Byav%HYJ|_7EQfc@=WT>`)*h1-L0TF z0;_XByny`7#&y)r2&M9Tt>!CW=@S!gvRYd9R&{~Q;SKP!nwEZ{T>nh-{S^%RUaFT} z0Q6?Yi*1LkFFH6!qg03!LSlkP**k~AEW|C z`kN2}8W94LXB2YjNy*K8XNPv0I#Bc79uGavLrp-blYqQ9a!frC2mPUp4VX`G#z4N5 z%3_}T<(76N1ZsfcAvSzoVrc$dr3T5U>UL*8;9Sm>kgJ#E|{T#e9#ph~nC`V8Ss%qhg2qMF>jOF5l@ z)PfG06n;Q^x417rr~(yp`0P(&E!tQu}HT~Ja<0?Er604xG>CU^tLj18F zNu2tL5Fix-fAU?C@OcYhQsvLo_%sO+9vAo@Tsnn=*u=dam84B9X<;!mo#ZbXdp|!s zZ+C(s*yZ)4?K1D?n-XdwpFjxITB)pjVBaR7G@uHaB=={tzk-s_%B#Q2QVpu8M&_uB zN5OCqqq%YNjZqJst}NZ_G~5H#18Q<|HZ%a|&H<)Bq8{eu?Al{GGP1I-4&u)oPJr!6 z&u-oV-luW|j=LNVi1xW-c@ZFb-<9Dyt4IUk`*k}ye3ItJ)%%$6)p*Acb9#6IVXNHlkt>;irNQ zb=bW_*<N-Afun;^5FSc`N9|&yA^>RlkZ7%KQqA+qAB4ax&=l`yZOnzfM@GZ}hv1k}=eF;<^8s9JbLO2XgEPOgQ z06#;x$UoCK>Nt#O1!SvD;DH`E_V9^F0!k4b%ugCTIBN=6Ekxzn_mu-@6GGHf=J;{2 zwg!D{R!Bpo63&0B3Ih^fG7%7Xm6X@*CZ1N7AxfGEd@|@~$Jp9Sb7DujMO8;a%aeZ% z7ip?ogc7IA3`z{5{^mZL(@I#Knn>UqM~@%9lLeKOW%*Jh%qFu2#Pz7QdLCm&!St(f z!1kfxz5BQ!K9Ek2I%FWSK)qr0l2T>?a=wpgX3Wn@4??{Bd>X_7^A~Xla*b07@SJ+> z2p-~f5O89C8_z!TEIF^OH$rZb-4s{iG*&s$CM;eI+;7Iw6_GU_2(rizP=@g-x=xmo zWMnea`d0RX5gi0`Ag;$a+6^)X5OUUIspHK{i0O?AZs+g)ANPy-aSF9po%K`#do8f1 zEU5^MYt!9oqW-3;pq2xDfPC zGhsi@`yMcvUYjm0()U2k`c6>=hhX3z!Zas09wgxQ5BZM026M)!{dhbxKHXl8F$Vbx zqTwmE_IA$-=j5Pzg*sRp4z$yDzzS6UPe54(mB5zC3Gd_Bl0FDbb@>-%bR|kudH8{p zXN<$NI28hGnDoSpbI(P-Sx$c8whg{RY>+-yiB zaO&L2cLVP1N_p}2fWi6TJG=DuN^?Il`YU6%2DoXsyF9u3Q7zp1LTJ-gp@b>tOy$56 zL8MbhOqViYRK_phk3atRhozV6pwBtL_3y)NHC+Gyt3r6rryc)b*p&96Cocwk)H&g| zT%QNM@PbX8j52H@{!$c?v|-uh#;6!4vl-mmH*^MwR56o1J+q-}FwT4#)!+nFTYMe_3ZfMYta=}ZQ+t&206%j{C zeilNd(&(I$L!uhRp=mBm?kSEY%SR|=blSF1oT`cYlB!lQOJQ>EHC?zh#fQZXN!HE3&JqR%>^)V*RS!8|;(@{1|AgLf)>*6; zQm7WW>uYBbPIvyPCDjzmekq-xIJ@_*!v(7u;v%Tb;Lw_5;T_2=jC53 zE@1j({yD10R}q|#^_Y9ALFchAwZ{$?x$+PR8C+y#k#tVvi+^IRuo0feOW%Uej`yKy z>=E^bt@CCv0*dvfdg5Xw=B;eSkt|94wTLD)hm?qA|OsVP3Xd! z4H448(E?4?!%-Axkx5Mrdj0_TE=X47iRY6mek6F!kwejm?iXbhZ+|{Gj*{kci?m|t zOU-TM>oHSvyB=e;SoZm5B^1i3$&tH6iMqsts7YM`LUNdq$RuT)Myn@{T4PI)j7rMS z=JIKb6G84cp}CZf-@vj2LQBYt9A!Ov8uLNKqF}y`=!^khzE!L}TqK75@w6 zNRXWmTNk@O+Vq%xt1Q0!O|T}`gWqf|(YRQ3u;~~o89?rLMbrMg=p>aZkG+qVTB*EV z`ttS@V{W6q+qFIE;Bq0d3_f6$>v4$>0GqtX4ge2a{ONeDsMqou(>5)t3>*mU)CC($>^`_z=R&Q?Wxvpa^6(8hzk8CK>v_UFT zGnZkmd4@zazgVu39q;Nh5iJ?GP3xL}-}Nv@nJamC=7vB$Qq64r*9Hm>i(nHzhC1NFpkvyX`Bb>1FatYHWk;D2npHo~(v&U$! zqbl&{>TOaK5I3(~t&1Nmrg#_66vZe!h|bXpob9D;eKu^pB4+jQrVmI0!z<2Veh~^| zgN&}QeI&C6x-1RA0YFE~d&?M``t{goL%o|~c$IKB*EZMEr;_DTacYkrRXzy?{?q>o zzaPNU=nu}S?N1Q`XofeH%7d~$(u}4s0Bgoqs1aA-zNpa|bz7rg{SoKmO?1W&Qf~sq zBX|9d*z`G(@eOm-I8#$>+QNO85C4Vq&$x6G!qX~!`eNx{G%m8K2L1lc*K^iKKWa6& z_d6UZ8HoJOm%Ua)tBxW_<)^ljJ+8(QUp*R9G~0GWG<>Fu~=!ci;nXiL?*dRDMrH z6Jz`i(_&QB0PSovC6vSZVMqK*>-|`pF9)xn3pGc%%zB; zm*8A~vZjq0xlxAXW#Av9>n(Ka*kBV{+fgua`>Bx}g(SuF?R6Q&&q5GLw6)$v@awJLm)h0~!ut+cP9`FY@ zkCh8Z2H9B7v_o{asZ)AON1>Fufr1 z9s+0QBHQFBW_>2MF`Z-ynJ zbWZ#5%;2U{MDT1)OomPq!PNVi0WA!!@Bom|kE{Px8mRY~M3@{z_WNzl&gK1MMp@@9 zn7sK5z(j~=Q2|KdIK$SUxfEO`LttMM-jkb@&~GAjQ%o37;KtOJobc!3Vk;R)&>KeB zCzw5F8u>kii!kllRKuK1sJ9>Dv}z^GYZ*7(X?gI@pt*tdOO;4D_Ix90^rgBKE37xR z-$dWUJCR7Rait-V$Hy3pHf@yB1l4CkV>k8&Iht*?u|5y@R@xz*vWcmYRQ`S&!iGAg ziQw9uGTjkl?$J5<+e6Ato^kA8MA3-PcWps~o~{ zG^yC(k#C$>f=zT!)&qMr^NZu|-~5Xr1Ff6!R|M=!Z`nw*a96-vgZDc;&IGV|v1yS? zm&!{E8`TlPF@&pr0xD0`JJmv_jrJz>n>asXk#vtY4qg`O$;Gb*FTI&`Ob~Z5Eg4uf zY|Zc4OC8=o^S1@g-;kF2VVu@QYk98Q_#mHY2(dvdc_aq$5SE$c?3CyM^>Ob<7tV<=gEzB^ zX?ZDCW$Pg(1n=HuGy4RMhLE`w)+G|QJd%Ug$ATzCY(t@oiMXmTo+-}Ms>42?ih`-G z{ZTWHX3Yiu!Wd@@2iY5TmqGbh3jakk41 zs#1Tw4;J98pH;HtdkjEnfa!=|p+eYZwjsiQlWtWRm$oTie8Gm++lC&|I&F5*Nu#0h z+P-S}T-NjpvAJHpQ7QU;EM^RW?DL!NB=|=i%Sry_ zW7|mhZ${Wr!@u+Z(!;;>VA~u1b%q^hfG>gyWMJ5@5WKZuqg-6GrTUXv#>r}QJy0#j zbRjxPPaR;e_gGL_hX<3@INJ zjR6XlX|*tpxA%8y^ZTGM_u}K~4IaXgQ$`1Zg89&!^y%I_-alXgW40{O1zC|qqW5hl z!ud*;u3rIR?=IM*8Zc{Quy++-s;4{YHaWO-u_eKXT|ro$Z&U0)8>j{Q;h=}I_tXq zeC5NlB0^u%?n*Dlo_Jz^vhH!Qc=cE!yUfy%%trgr>Z~Ab^@ zC!iKCXE?qmE_S!D?Ezhr$qS@ium^o#&R|xAV97V+p1#hnW|LRjtKKaeo6}%4z6WNI z+;Y<&6o;21?F$bhd^rC);Gf(gdjskRd!(srxIKe>@L4f(6b06GZuxn7t|* z?&@<+*^<9MP{CGU1L7xl8Y8b?hZ-L|2P1tsB)3QAmZBKDN88m+3kJ@frNY~nFOtHHQjXV#H%sIa^av`sE zr}5ss3P{8pu$Q6kuiG7Awja;$OwnqXw!IfENL*8!9dUmR7;lT4`9m4nah);{SNCz) z&U6hAEVg?+(j9mJ^%}VGoYR4yJ9=B9!x{f<+o!Jb?sntX%b@YR)$hja_U*>Y_daaT z|8)vZCpx3;*Z5!yOBK&J0CF)IpJ+so z#mO#a*MlErteVe(yJ3$`(JqRp`)v@if~mFqiaY@vEr3G3|1RN$vMsLHJfiFg3%Z7VP341+f9A z$FP3RUC9X&KLG14w!964@uz(@E}q$o((qRiVa6y|gO2^{&03tv2QY$#=YXZQEK=DF z_Tmm1AB80!V-&n|UiuUIk_Ni|Pt)1|EDQgCKw5BSUQz9b$*gFHW z?bc4sbi61R*E>Oh{NW*h4MW?WW+h?l?)2<@qMn}S-{J?>Q|f{ao0K|-ls?B?rwE4z z2UoO6ol=n(k{NW8)Jt~ooM#5UTOht3Sjr0y{t=Gk=afX2rxSWd#<;6Gg-i#%my#-u z7kzsD2A*$4E478A6b);nG&NRH`CbRtG_iG@gX{cW<~~JVNfiU67T*#F+{R+p4CeYe zruZ)diu>K^l3ANV>9`6(FvrJ^TMe~vnjGwIq`1k5R%?pDFF&&|ag%vD@pWXR=B`J? zkqF_ZTmFVRZAEhq)*?a=D+?cZ-WPdlyoQr0!}t8Ky@Sk;{HQ4$Mn5SO9-3!=%l=)G zrfF5x6Z_dt+p*gG^Z61XSBYZFkGUnVWsT$6voXwi&rFq(@Qub*n<8nXlOmY%XZbGq zq44LJIYoxa_tSS547`ek;fKKTN z1j{ka65pEm8TCSM48tc;>xrvd+s{x_n*BU@=kMCX34Wc z+oGvR6Ubg>HWAz%DUN%Px>1^CzyWHE&}OGYwA|x`9KMn~4j0*aF7|qaWgMd~_4 zSsbFq@)W?(8FfMT+%c?;H>~cGFLw$EM)}PO5yG$M-D;hGoUhcnvA)reSJZr;W0%DI zU4H_j3}2L()aPnxPx+_2C2bm+80~&uS&u`DMm|Lbf_uweIv;TQmc5se|0?wtRg3kV zyEbCI`@Np#*7TltsLjiZ7(VG$atC!`Busam#RDTbx(M2ey;hD zyfKf2{AOHal4fnC=c)8Aj<%Hjr$Tmr<02~4D#cA2wAxduf>8G_g?PsEhiVIlwx;aw z8yQ<)oO+IFu+F~$F34MA$ zYdmqtZ+)5#wDb#=KSf_~--Cnv*U?3eq} zlDld`{F~wShDk2HqEA@O2)9<&mDBw+!(s>hT%QzqR&WK)MzXoRM8KW~=*!-$Yh#dU z1>1W1}7H=id zYQ#iLlCXfFj%B>@r{9zzXa`(SQ7NTYmRK z?3Y6x+6x9{btYC)2NQDUP6uy(a`5*cYPCPCGA?R7x`W)}s0avYLA+>wE+N_?iaos= zr?N@1{Q%))BT7?|Ph74Q=e^U2G|AqJ3 zcU#?}x!#s6Z{ybu;IZ~VFrAQ*kn2%(B)Xq+X zKerUpuC+*(+76HP>_Od+_peBPLwV7?*e`gFVQo6ji;-rR^qOlFBpZ_aQ2dC#&2#C) z?HZ$SHzjq#=8ap2n3oC%Wvr0q1SYa)y~O}*yx;q8n8W=|<6;K&ofc;DN8aH>OhU+RnUwq7Z+ zZJq}bCN06sKWau%2*49q%Q398A$h&**F@VSNl^M^EVZc=cG$q)5<*0x@2^P# zk>olXrGN*5ys4<_J(GlcuLtEy3y?&WLk|UXd<8ydR_m+gYU~b>r!7UU&`Q7g{eCy& z;!9E1Jzce4B3xzqZQ)^3);+_fWjgBc7){;y-3B@pBJkJ7#covILR$`TMD@v#`|0m> zbt^Q#6rOW9e!$;0VDx+ggnj^^Uxl9JHK3r})lDcQ`yYJrkpRaCi_)5o$CULrqqz-> z24$ql4u!H$3oY>5TKhiDsiR&71p@`QQGXC72lOO`y43egd}L&-z4Lk^5fBt)A2%mU#5Mf||!j_IVMeVG87-7>^Y zv;j17vhcO7DOPrH)R#r^txXDU=r7|n(;3aj`}eBL8UUKG$e8nAtl1(|=3ayV%$!eI zELr|K0Dofhp=K|2IiJc{fRe_|*VMKi$u3o|SWl9K(ZEg1DNQdwCbi^&kHHzV!+!C{ zVFLQVmqLt-%_B2*Fu65_14^gZ9(Cj5zi$@K&EizvQN@)3VL8i&Hong@rsiI#m%95= z5lCmLkB!$!(!dhRyY zzk6w5c0Q+6SY@$m56mOV^sWg3q;+2d9^JPOd5Hcj$6}W722NC&(!_O+j_3R%~X=-72P5L8`nT z)B~$C+zes2f2Y|t+;fYOhEP8dGBYgQZFDK5)8D`7Y0+HGrTKI1sQ>yFph+TRclV$3 zF`RvtHl8RR#~T@6H!AOHGxwJi&Ryk^@TkH9iqUJ`(_(}Z+f+3^%5!e{g72fu5VJ z@2z|L#(4D)Wa^&t$W-w}s8)#Wv9Dx3Y7x{2DHtc)d@a`!_{e^WNsAR)aY z1tht+vmu>h2|ygjGxFZwtwHwu+x|ud&o!Mav%h{*vE2*%-U&azg5^eTm)?k8f`40R zh9I~8U5|2Dy622A{54xlUd~qhT3a@P9hMmc=%*)RVZHEdH^`6>pv>)io1{5dwI|Q` zM+!OywON-=P2BU=p+cngy?7;H15k$@;4FPwXqyGc3!Dkj5_jk}(Ood0fWGy1N2%SX zOBd&ZK{7pBBWz&rHTdMRuHb@G_n%o&08O6^=9}lY0_kEFkWc#~Sf;OEH(9S_hSm}J zkg)e63eWG0hd2>O#X_7O0kw+{?<)ILGzTo_!GD7o6$xq(qgvK^Sf7q0WNd?;ZE^*K zDxu513QN#5N@5$mZtOlY((%4aTB_BWQliBP%e=LOx;#wy1QlIS%EtC|c*J*GN3rJ+ znEGjiD=FNFzR?T^~pb@vbR8QSzkn0i=SPkOgw0g|qdYqG zag%BdC0pqwIyDz!9jQS@Cmo5S!*eJ5#ZN_l{E7AB-;3}}SwY_oGCI4?R}etsD$3DNY^P>O4O z&A!{7!=d3uuVM51PV2J^vWH>vDWe*(;+16GB!TeFX8kAfGABa}OAXmAt!~2xPQ_@# zSxf=lPnsDxL8mpB*e2kNWcTf|U!o;tRc2U=GxuqGl47@msheYf-#_6=sJZ96l)*IvLVs#PD#o&B-t7Rr#H2k%4b)Su6&|+00hL3) zpd1`t^ouQj4!6!53-0no9j1(zzx~+K`(#(tf{(HiH>K+#50H;Vz0p$3EAaHEJ0LFa zGYnsDUY*(9B=hrEK5r5M=l~n^uDG)vjk#$VH z-3GdVzEkObqK7|e!U5w0w6|yOmxJPJM3Q(*%8gBo=)J7Hk^*8Or4BeAP<~{qXT*1y ze|rZ++yyvQ6G+xQ-8W=+m0(u(*uO^^+=yKq%E(61}A4a{ykiw-!dkQ(@=&cqg=%RM;8N?&_%2==%HI;@c@ z;QNlW2j>a&v5V^>A@b%Z9^D1$(lC(_<`^g_1V3|L)>SKaPg~0Pqod2xs%KwYR|)_m zEl$zfHyNKCZe-lFF1PLmgrP;(r10Fi<i&t*4SfX*(=X7UW@^PQaC3w zZP_#Xzu0^4s3xLie(rTPy`%N5eOhi?=u53f`Fp*rgW*&YY6J7C;}?ID+oyM zEmRdm2oM6&n*yN|qy|VxzWc%Xy))lh??2}|?|05RXT7X7t|2^m%HDUm?(4eueh6Il zTL`Ksuw=zdz_A_>$d8T6-iStTO(lU79ay?&iJ+-;bexUEr@ub_oRa1wuoZdi z;jj0PUPY|FR?6iU1@H%*!-Bi*mfMW0ekqvrky?K8*MM-z7-SG{jRNWON1gxe?kUTB zo72bkDVdOprV(##E1F)d+*(-8@CQR^M9Zu!&B}I2J_NSJY;7DE5@1q1k^C;Ze=7yh ziB9rmFRuZ|9@M*R8rrZ;u1GF;fEX{gY4GK41czd9I`wE&q{kw;QSe~G9F2%Oe@kPMd z89yN^E%SZ<<&CSAQyS$e+x2)ede%6UpFIl-x36lqzr8a^tW7_dO^TvGrE5!(M;!)8 z9saN`=VI`cKhXmd;Ls28+)kZn-!~I(kcJIHFAa-yu=(XD3H!Ly!F(VLof5+8d8Qex z5}8tqY>$EUK)HiY5wL}#KXI{cSj<2S+-)8`d+wq>f(!7&yxVlbsK7~mE@S$2rC}x3 zJ@$l2<{TLHn1J$CktzT`V*rs-N~^Muq#DM9DVg{7TUbR>Ag;=qvIi>1R(kRvMcI(w z_;1bV<>`mSM^7aT+<*7}5f(-w4BRCyvrw>f)8?sCPLqp`@#ll1;w~lDc*v3?MeZCz zt-a#pFl$XodqxBA8*A8S;Jyd>E)@Ebj;uA5D>K)kfX^B$wpB(5vJ}7`fX#-6?;0Fm z2K&D%SK)r1KD5$R!C0Y0o&H7KZ*KiJ7e=SnI6W2*C`XL3(fq33aHeJ4S{&n4k;79A zc{iAi;VuGY%>}3Knb7->_U%t2FB~tt=lcn!ZbSBv(pmY_Ck10w>cE6yLdEYzeEBVa zwFWLAp}~p)2+(Ci1ZeiG=-r1sfp1$zzeXgh0C*DSBSX;ORSeek)poq>CnailgLT+r zU4N{|7HWT_*exoaHfs$AF$-$pz-5RNKOYq`L4cN+??o{sp#n|FD`0;_m#xx1s49pb zvoeQX-he4kQz#$*avMm|<@v?WPYxgUj|{+0+7v<2SWbI=Kq&=sxe=XW#wr+5f*z3g z{t@g`Xl}C-EfXL6Zs{}XHU^N0J$)(MTT1j4Kw>N6S%43rQ1(AkdB>A_IML!7@S5x6 zs!+l5?o&`Y1{Fx;wRZ8;M~O=PMY{F~w*=fF@HC$g)&p2y%cf6rqW8f8*5;{1z2!gN z`6GE2hVSCyuE6w{_>@RZ`1+b9F&`2}ti9x1oJ!X=lR#W+63jbQ?m%o1kP zXV5yeEVhQfK~W^(`-cuPh<_o4f(mo7W-!+s(xh@K16Y6kKlY&{N_NsMV?3^C+F{Z& z^)j0gG2Rtbsgx%<~2nsBstE07kH>{;0Zj zDFk1MwzsDA+&v(309B0o8R&|#!>f%z&GW@}gK_Cps`|@+PAMql`@T!!zVYouFa#v& z0F9=y#{^(5IoT(F7msoB4nGBfj$mwxUb$H#PsCx(OqxWzykv?jk`d_vjKU$V*#;G8 z$S)b(K7t#L*s*yawR}X745f@?yasXd1qw!8t}3J>@|A)6a<|v1OCQ$wqEv3y2zPIL z(|*u^kI@Ga9DAZwu|1Fauei;oHjIN0Pu_cLAHtQE1w*a1bvf!{T1-4m6zsv?lto%n zx^z}NXKJlEa#Ol8-wn`NU+1QO_lG%v&CFQOPYNOZtqVZ-iM1UxLB)NvZsBcoh%&wy zoTnHZg?qe)Sp8GbN=~zs;&*&=W+DRNYjPx$&uz0WyE;5J)(@dLq%V5#6xqt>YY7Ek z0m7jS=*ZqGN;evDG8In;W7bFf)T3DH;Hcf!{&G*+bTZG_LR{?u)UAu4o~`yaf_x^N zcdCsc?y!4t9=jpT82TaDD*a5FA^@vm+h4ovm}hxILIe!N?shIBs;S)|Sa)`y?U)o$ z@!Fma)zk1UrPn<>s+$VyqY1w};$PkUDAeEL_rLzH6~F&0rO~hUAzZ+SBcwG%X`~*8 z5`Ap^2luQD7QK@j|AelJV!-Ye>Qn7mb!2elR>c7AwchiUh{V}L#_S8TY5K%*=@jZr zCY&n8$}ej+U%<^l2s8WxRZ%OXJ)Rg8q4w`!`D(lyi3VL9lDeFiF9TkP^68jXc-@>mu5BOwSH|4dE{ZkiGrkAV~$3{uD3I*?gnJKyVtY?}fLyaeRa;?i8<)lJ;9k zkx>+Bgy1h9!OAKyMh>`5DqV%%o2SMoKHU)l!E!v?UNUNa-f4mSY!%ORFli8i)4a|Y z^h%}A_jzAynq~gU>k#b{urd0y?V;fr?;W6S)1Il!F#r?cO`k7(kZcAa+}`xXA(X?B zuq*VRAd~4~+Lw*wtnzKt0*-JPQI|gMBW*)M;tF|=1U!U90=$SA56B$qqBB5#7S7Gj zA*J;imqdr<@s0YQ0hF4!r`SW@0B+@Dij_Z112ctc;Aw1yko%zbA;`ISX*a4s1VD#6 zS9uc9L09iH++ABTe~%vmNrRco4>$oR5l<$5tg}U?-5QcW$%Na%gS0{|5VQ8#Qvs$B|J)%1J4R14`7$cINpgLI?|aPccPYw*N~T8rWRS%0rTJ(Utfvjd+z?# zNe&$LQ&jtu1>AV!+_tTydv|0EoX;HmG$^QVYaYu`_!JF!Y!q8Z8;H6b^@u}jvCYJmzn%J0F`{(fB|)f#y>?Ezf~KLb~;?zx;{W_s~a^pg(c`7GXnkL@4|h-Ec; z1hygdi1JcaiKR_B!3jG$-HWWJiJ1-Al z)uK<~WJPue83GlPEQK4VI5YP1{0L@dV$-FEX$8~0;ncWg#i*(>fdF{{<9MkroQ|qI z2;=(Ml+XZ}P+ilSjZ091i{LoQinJ;`$nTZ^hzj(zhZ*ur#K;4!eInqy0T+MdM^_jc z^k)Sqpb-72Pky!dgMVanE-=(W0tBJcT|U4aa`RPp>x|4PVkJ1tOsZ~(;)Mt=xVGd)mczGXm^Y&J=TA)tL>dF@`=_O5+NhNXVTRmjcVs3&9FxXNbdCJ$7kgxCjO=`<)j!=Rb$3>K=&av zg|$NMNy@W?oD$lp2}DV;RmhJ-BrvuskVbxnIv}`x&z_tHmMxgDoOWaKDAEhl>kCEL zdkYd5a|z6`zYVQ?9|=?YqeJ4;NOqm+P;*TjQpf`jEf6)l34tlczwbeRacu`i2fKgI$jik z@KhiNyxPoylC1jDEpNuUf90cGm{Mju6`~D7yN-U4m%xAiyb1BT((VBF(!8&+d~(I0 zg~k!OHgo}=>w&7%Uza>&1ZvDnvZ6lG@2#&NG!|qNYLMm->R9uu72n7SlCeQ;m2$eE zH8i}OpVICU=GMPimYW9A%Ei7ZjFmynnc}s=B>CKUt$h%Rp_JQaoj_qL3TroWf)WCN zxR6iI&~9H!2SH6!Q>bth_9YzIcQ}l>Q>Pd#6|^B z!o83Qb&}C0_|m6AMIQocpGaF|}50Ofx3qs&bi5S=|gIl@`L{z0?8iC&DB@#L;YEJ67_f5+P6Ai%8W z^JA;>Z&q3$UIH%3x)U2AsNMec#)xN$Fko)9j7u+s>XO$<0bUhfP1W&cRiIf8|E?Ge zC7uDX!p3>n)HSRlY5{w+dZ*y?toXXZZA`xeJfe_Lj4h;0XD-ER%d@5MHMYl^2)9_c z(FZQcW`+eSME${H>d`CFr=N;5yk{Ofi_B|-U&j$Mm5PGQkCZ3(hLaWja^}ai+yt)u zXuYcO=I#bSD=z%=F4VhQh31&a@p!0;yhXw%LbJ_kIWUyQa4VZ+V~JK%0(vqj=+VBt z2+?%uF<=;#Vh1GC9*bYO%&eL`8Fn7P^NW2(z-p|IKf%igiBWtY%|U|@ZeyU*Z%~+U z9|%(Dc`%^itf6HKG-gBZ+#QKBjoHDC8z0*?BqV(Sl4dUyqQt}*{M323lP*4!TRl|g z=+Y`##XTV1A$;_{72n3?DK=iA8uSTzs*@$6|azdu^9sguMTJqI6gGTM;QRkx#Zj#q2vT~ z+84~V;Kbi90g?WokYIfQ$;e}s3xiRAQb6WS0Tj?!5N8%3EIgg8shKN7cFost68Qj{ zJ~Qj_oa5`kv5@fri5ZGogDCMX8Un~8X5D-nbEv|i?x?E3!jMorRRTPj+X5&rXwVwxOm``2Spp;5$NY1PfhF_Te_>wR77o2jBr<2q zj2bPC17xxQTUF<>N8D0B6M5(dBOo(mKOR9uTB)tNp>gd+2Ce4RHvzA(x9}!G`(Fy` zWUKAoA;j5xi`?Nz&NqhF3hggLcbZOq8<>;wy03M!1+ZK#3UMBre_lc<$wNxK*SrYE zA#`zZ1>wWbxJCK3`on>ZDOdnB8%3W(Ukj>)iXg(tj_!jf`ZmnF18!a)nW`f%8yht5 z9_TI_fYUbso;@F?L+#N7*sK9|s2;!omQu&-@u;lJFV@b2@|m`q_BocDR}(e?Vl7QS zOydT*^x)(wLgw1YD?U}MTkYEE7(faFqw`o$gAPs}#J2@iHl{Y3`XR{BYfN%q7#kjc zH5#J%y`o;It5v4_cSYU&y65S#GbpL-huof>AmQ15@we!&c?3jHG`bHa_k^6yW_w1Y zzs~y_NyJp3H{sQ|L2Y)eVU$^X(6Vj-D|2)sD;6DHlXL@?HXQB(AN}^By_!c#Pdmp} zn3oFucOOBxuJkPaS{Qjj7IDTWJj_*_>I>%#=MESGh>C8t1E}7+I)FmO-vAtXo3d@k z5p|Xx@bj%&*TrLIcBc9Im35sL-T=Bnpw0y|OonAGIx<6V83Bz1jEuBSLCl*W4dtr{ z=Do-K2uO&2Uf*5F5{?MhP^mbuW!g2*t3PS3t)uy3qP6m(WM?Y;mqi)uosq2|L{kKr zmPzFEI%NJqHNiRwNa}e0KAXJZixuYxc3l7N1t8}RQ`NXatUj=zY~PxPhm%e(#Kub* zy9TF;I^~x{J#7ae@hkyN_&0296Sr=1UG@?$cB(}>Fo{=pflvbIK3@DJGfJuSc&oZ8 zAwtKK4R6)`Ik~$BPz!|j^Qv^clmws8B#!tjkKJGBPZ-+774>1!(io&`!*9+1{n=1x zQ|p`98=-OkXaiKU?=28i>$r}MKBeySi?sy22-$LUStkaMO`gT@t($IpH1`Th_!#nE z%ojiByfB2YhJBrtmyw=a{Q9ka0sCHy&C*TE!3X2ZbEeHTI=fJ(-t<|TNmpJvYoO{e zc|i^7Aw{k}P+6SQn4Xs7F?AfadKRpU_6PB;XvO9gU<1~bDZuinP!ImfSHrqydhfcH zKorH<_@X_Fh*Ico?u#;a%SW19NXOFfVe^Q9nqcw?)oh3D^n0GY9N5d9mTrw1czW-@e_+M+7Ak}+iS!H47v?wFkyh# z?Q7ZmDu2z5KyoFX3dRa`BUI)rZeZW;u69elI=*J?ODus)#UdHSfy;5kF- z*fK+1ex&nE6*_nuTLGwYKz#vJa>Gbid#|H^_dRgWD@TD~nnak|k1rK-7To6ChiGd1 z+#5c8%7;=KY9T{WY(HpcGY4r1z*y^b0E8|ximkHsVL;)lgH^ok2G0R+!-MqkYvqtc zM|t{*2hd+|Ty9+_ma@U;MN~SU=GO4xB{jr!p@N1(1_qi#K<7yNmB;XUvw0Ay7vr>y zw>t_;QnIWsKPq>%a~DEGgJQvLQKZ8PDN{|@U|;_0m7sd6YhLk{k&fJ;8O(pC!F&wK zF2x_l5VqC~XTG3G@&%?y=dO|{l+QQr9o-7O`S8S|I}nJ79IwuW2N&X8jpD8`YeiBE z1!YoO6|lL;=5(M#s!{X&#&Ro32TxVY$5cYNf*jjo_|r~|7-rOb{!itN$3zehRq@^S9Ss7M;hSt61#G{hpdXPkL)RhRY$BR z1j5aL@$js702BsoWI{?heyi!3?%kNQ}w_RD3OT)zU?wY$JfFMWWKOe&MmyXERdr?BE!Olkb5dRWM|T-F*bZLr~-`siX1e~Awicp?AZ`@;BuQDSDMk<9SGTIg-dAOx9n z4Z{;;kLNYqiDBDE8RvHxo<4!n0ez_MTaxsJ=mP@DQr8`RuKIG9xiQc0Jg)%oO`Xer zq0rrRU3AI=I&NXo5J0{0gt25cUNv54-zSs`lc8&(TmV47kiuPB080@ZAf2h~q}r;0 zHQ(ko6=>})vou`>WMU`XsuLNn;E%Us=*1$zK;oSfsjY;3aNsL6;5U z2u)C6g+fFCUZ;4&4CzN7AMsMCY23V$c;H^}F=;_XcZTAwysHFk86D zeHn(7(RY2g&3L-E4(j#_pM-N6)K@7So>1o*#}vzbshL>vwIaZx0vvYqS$qyM0|ms4 z8&RWqXUy-kmnIB8%4<~nH?buYD=ECjU2j+0GrbGdEDEFqs4PM<2V!KtnEZ53PgF5H z1l(T7WtRdl|44TeaD+(zTw`|@1F1Yb)28su7NmA%*Oq6vaF(#eQeq365iX%FZ9*tk>kH-pJ77u2t#$EsNs#*APk9_+m@oC> z!<^$lkp<9V+jIc^nn#0G$`sQDL$ ze8DG-PzC~s_zSgT^Bws3Y&W&@Lug*1Rw|4)tbRbpS#X^9>byyhfp(vga3}F?n^M}Kn8BSjH;Y3*7m_dcgu-kR=XO6^%M@C$Tfb9^mRcW zs&I2*q-B%gR=K*K?smoS6;!=70`rhQruXBUZq5Rc6pE(tySZEesBxCQx_-No4$gZP zSR>%K*mk{Xu0dG}hKtg+`yMR?>HuP436 zBdX|}xN2a<)oN`(pt?}&m16lK5t(mjfK@S5C6{K`T%HCE4H>luR07*&Na!lr4s5mU zk@33r8nFOx#?q7gU_)h%xo@9shc;b&e9-3Kl+&Bvn|=EDvYU6G{Ws)u zMEXA@iwyaDvkQ~%Z;{qPnON!?`NLH^RA9&~Qima327acZG79Zh?KC>_GCC}>p6NoGh+|~H|w4*%3*fX#cqA+b`voY~1RJ91D_G7V>Y3b88 zgu3=$REDVk9Qyf#WrHCrBdHryZo{R$RzP2JOb0_ee_*kKbT5OQf#sbKhFxoc@*Y)R zG^ZFEHUqH=jQLYA?Vz&z+(P{%aWD|?1-FmE7W$@n`Rk^YDr`VsBL%c}9~6~{fMHvZ z1Y%g2YdOcXV#P!sZHVFxe9I#fy4Btll7(iAr6{U{#sy2FK>8D+*S8Z)0F{G$u>ae{=44PuH`c#d?O z0q7}WwHB&-OtLIhTUi7aVjLy_7O@x=jU&n?&g!H2DD07}rel#b2aV}Vww7Xl5xC=X zaM8*02;Y$qjb7WryBSFHH|JEIrnHSeiEhPDev)=u_%xQD3eH+2I%c@+dcIPz{3;*N z_djUk6?FyOHVs4jN;=AZiJQJ&Ue~>P7~GVosM;_njyd+4@2_N}&gHhq9jNSM!}y5Z zoVs$WD+?Mh`y-c;f7ewdVE52f)ryy2cIr^8x_cQb!iM8&mN09EG z2GR-nQ928C%8&p3KX_5=f2FSXnrk2Z=N(VBz%ZEqdSdu3PW9IY8}4^-P3?a4DUj6(Zv5Z4G+gXGw-( z&B(iPMTnpb_X<;ICRF_?4aU?_i!DWL-}IeZk;t;!(>t>4c|L;@F4=#tNe=v(j$J47 zO0--i9NWn6tkm`v=E=SEAX3+`viTt8&6{rzj6zB$erj}7|5fb>AGd9qa9{2jSKBzL zKNa?}c)U+D`u26~#UIKa7A`(1ctDI{KD284$@ce{m%67^*>lCBMIKE`nfTpfbJVM` z(u-Wk%9tli9j~LWC^(arNt+6U75;zU$eCS#i^Bg{2K|qL|HnG&|337;N8o?q2z<9I zXF{#!J!;Ij5m#9_JJ9hhRm6S6Hu}n5RK4aeNLUqw<$^}AXF`eNxBjk;>Di2I;iMzL zVg?HG{>Pu2S+@R)gD?KEL}KI z=xdJp#L6P#v*h23r7Ym{nJsuzFL{cdJ?{?>lF1*bE|hQE71}t~#n9Rljc(kkW(uRL7;ERzXU)NN znF0%XzN^uJUYJR|6g;aMz#)#=s20LFe9@!Vu<-l*-QiJNmCJ;(g$(6l(PujPSEGLW&f|vF#d9u!IEVtLgD}=%GfT9Aj|4m1*d$ z$)#gAaPC%J#nTi5>-bh{v;)uEcpr_@Q4yS2Z~E7NGvPs7p__mwNpv6uLu6GJ{bPDK&7byqz$9hMeon!+r((56GiIa=%!FTo7aRuc=O zil?t6Yq0J^`3}RiA3C3QmWEXkQ@V$0mx}fE{Rh4f%EMLaux0Fu7#s8^TE=6|Rj+7z ze!-~dVuuJQ0;yTY(o0oO=`0L(aU#*}3FQs5O zCn-sF7ggbLy6#tW9kk&!G^Jf%qp3)LQjBpB$Pq#-gfs_F4dR4RO8gRWDo;jEw$Vn^ z<`Tt9by-6;@Glb&f9iAOXii}vda|1EM?+KX>#5vG6|cpXeCP>cNx-0|!VJ#-HcK|yj5)1%;t?@C zdqfa1EpVIJ(DOLz_kake5%0~BH-^l?T9+RV#W!g+d})ZqrM{Zv8VZJc)eFfs9LrcA zNf}!$ZGChA83OPD_LBkIq#MY0ka%Q_S-I53cZ!xCIvST4$T;WWJM7s7%Etd4zO12-&vIhedSI`}1fcUBKRrQ2_} z11s#fbE8z?Gz@mr5ZmS0a4*cJ|M5?nSIl5}XtlM+47lTAW}5kx7u4Z=VX*{DP8ol( zNrmL6+f4z+<=1aR6pFgQTCSan8kz22r!1P7vWs|%x_LNfhkp=IjGxhZc32UM3cOYN zN`>d_K@0q`OfWfz?ltq7;>Z1>0>PTcHC2{|9rzoj!y z*W(axZT&FTg*y@Us4#qvb%N(3@7OT0+%%xZPfuY#ld1g%ow&JJYqGXhE5Ci1S3i4e zc{67F<7$>Y*vM5oHmx4bEqG=P`N5&7%N|Y6I=qjJ3gN5t4eH%v=_nVUVC*r<;7l1 zuoDPA!l^KD3ECOy*4@1jNhQ&X*-=v}>e+c|Q7XY;_C>a-%K>Ru3|6T&!6JTC-G`%g z*I4(-G#>GPZNZw<&4jo1DrX~LPCwj}-`JrQnn#r_U9=sjvvGcyZ0tDwoft!Z%Z}L` zZq5Cr7w3J~W$yQ7t4~7jr15Op%k1S*d}+7o;0GyNFdtBA)VW zN=VsEq0Zlu%4$V?5ATz08WnbSQWJ#5YsoSNo#0MAf{S|mCb*v!eaAOu?cwrbgv}~5 zvllA&LxG!^&dNzmEm|e5h)CDcv%Unq^E3N0{Dcx8z(icGlI;?BXRk-NqI^w5j9Z&0 z>Z8|!U6-zI9C;*?O?y{hQlz*~?3!=N{HUWsaE~uU(&61*SF#MmN)L2ZGErtd<(eVz zx8VnF->On_6}*{2C`nX_mtV8B+)#dHAU0Qu$_2B%zsCgvp-!)z*-}Z8HCz5}hyCDH zA_i6Vh6i4@wH$;uAI;65FHJ^{bbPcv(0G7g77=ec)X~0;LTSH@w5r9H^>sGfy)4Rh z(D<&?8>K9};%Tri!G(l;{^xl@Uz6)DO4d~Tio|n+g~zEuB(z~wvA+PBCs>H%WIzqs zu-KrajKG| z@y(A!hQSbstcLHPDL+|1h%D-smr~u_4vBvF~3B+ZUsCNvp?8h zjxt>}{eubBhrW5tMl`HtUYT{9-fs>m(I8rdiRp+XIYRB~f_W!Qn)&nsF^Zw7%bLg_ zy9_cz!E3^$2((Oi*=#VhvFED4kI-1LZe^W&)E>(k-e4fuVt%#Kb8}sQzBJSt9 ztQ)L?&l9gLzuOqNI@#8mqAon|yBp>C7ztM3W7kaO%%bzNBzS_`>2FGMi)j`WKG5}K@|54NUe7Wp$AMDil? z8nGNQ$9Wb*B)3hyVO-Sj%v;jBdcEQT!mMNsO2Eb8z6{$_z|p+HVqn!I%HbviLJ>;i zHPxbNkR{cwOex+dHfi~?I!u zG_GL7T}RV+PUqUA;$tGzCo}0};>F;0xWI7L)og@MS;)U>;120GMh0_*b>H+ao?rJg zvqOS~0RlXDo|R1JqkE9=(8#@PqIz#k`AqERq%vy#6|OwiQic>4!`RZiMPFnDE7AjG zR{FB0gj;T)F+DxS4rK#``Wv8>#M@&TBRI4S*xcnExMo!KO4piLDQxX~0UCSz;u!6r zXlPit6|chB=4M#6p=BUX;wiS`!4iI!SYz)DXY!&E)-%vFwVvr!Dbxgj zGM;VbGbAqOL|HpIv|!+t8iv^V7cVB(V6tcGy6mnyHqzz&^e_wT0;s@TokV5F2o)Ou z^?Z~}YvNI@Ix|X5XWM-BVDzYom!klicaIIxxqRLfg+fO!bfe=uXcz&vD0>5e;Ida< zMr}`+32}rxDzzuS1ZwWUC89hA9`dL0eh-G#>a-4lM+)oVUcvlR6}_fAg0 zZ^bw6*A#4sK^3J8umap`bX6X3OPic%7Lm|nm-fO8@bNttr6`CIP&!C|%F2?$2wo6Q z8ZFVmp-?rJfOw(}+~xjsol%t5UvvPA9N!r_h`~UCE%E>0xSa1#r2UwTi7(8B`5E z3DD*q0W3WRNIN@EgH_Xtm26<=CA2US>AX=4Auosj8@IgAX)t}O$1OQ7WhuCd+a%?= zwOC;bfS;`gy3qhs$Mxz=j*#`#de53~a%#8H6;Y^ZD~#e6_Y$p9)V5Fm-@O24yhyne z-+n`#H09B|AG={VTnaeYY%(Xh*YI;nX{=W++A#hwa>3$qXYsXbxz{GISlSTt$wP)o zr`z>y2890d1QlN4q-d4UBGerU9wmkKOtTa&uqgx z8~WH#;18Q&dze)5k7Outz+(j$ytClt#frGnd!8wo;fKKS**8hv!)w-i$htxAl>q6HFGrH$Qyd!C^PwYW-(EP?oMtsZ!;+HmOn!l2iK_V8 zt&@2PC80sn_(dn5(_jMcJVTd=W}f5u8DMxVPYvACaxIAiUBC6paPrUH3WRZ!jZuN;bgKFJurEe`qu8bDab7 zhBeOCtZgwt5V}K)CfSjTh%AqIf7}UB>IUwX8x85!#8L?fD~z*1@$}dF`#AKeLb>wl zp2u0Um}1t$pHq}tcEC4e6^|wX6*Tu$K$^SYkt3&rYEOJ~1fP2>z!N;B&|We=uTp8UGUUbX!o}BerQ3TM_*_bhnt<`PZ&UC{_tP$N($RROG>GMV=mek@QApi#KPhz2a=lH=x#MI zL3 zGL;Va|M0^ZHKmd^b8UAp$a%7g-rUF z(K5ge-6mUub=57N{_OXt^$`+a<9&6$D1VCxe~k*{%piVB z2%0ROhC#d2W{v@547bv1{I37US>P58DN~SXpH|k@^)6@smZ8X3R+DK-IB^XZ&fDP#f z*JoF9)->NmW)TW{-a22VBtVY!_ukr$!g>Mx+8AHVcytSOTQ$qNJZD_8_h#FTHcKK@ zF@Wj{Nie$AmofA?SqtQ-Cv~u-F%$Nq2Vaa*Fp><`T?8D58f>(-B%03{wn8=823%6;GyRdL zb@u=vSrHMJf;oTZ})$RN+oHT2ohA8~O;X$PnY-1zvIjR20 zrQWvqDgH<`$wg#$WryoR(SDm@RJF8MbCUpm`$jCa0&~~YK<)YK=Wopsw*LO>vp@c8 z<;mMkek7*+RS&;IKh6(|r)NoB1jQ|%_BB$wya^Z!_^EGSM?|O&!R}>+vq!Epyw?l( zu14EbWEfA+zTP4*9 zb%kZ6S4o)2pMMDdfr4rEOl4^rLW|iE8$a&V;Usb*hZ{a-X$?OH`?fk1XpUCFcq;#S zg&zmSYr;n09Gv|B_!>WaP0+pKA6L5z$`D~r`yVH|2`5VGAuZRcJw+xN#_lm6d=mM` zKaTnRuc5LvFeh!R^)hJ5SRr@6pHQfIbfqzwq(~NsRLhIEfF1n(!sU0BMRCGKu2xis z6&(BPktC2)zViP2l&`G-f!VD7<07xm!za9}`u{lQE-VxBkvIH%KyB6Xv@_jd>g|2O z2#)iZ7YvT~z3xnAtPs+ij_wAQfiD_gLu&O2d zA-_I*82T*0C`Brb^R%}Y|KFZ&1*_J7{G9toa=-T{c!u=Yft|ku-s<~%Bldrc%-;y% zKNo2KXAccDPSc)QXaDQ=Za4ksBUwFw#J!tH6ytjN?`K24)7L-$_onfG2eJQ^Jr+A= z0k`6CO(;717Nbf#a{@v#3RNcAf9$qBK;*xk?Z7{V3TnQ2+feTxA@9ryKyW_}Qri9h z_jv#3v-nwuKhi1eM|?z~JmvnQZRVfdDmyl&BuR3uj9z5Pyv1MX6bdh%ho8X2w*S;& z{s&xPXAoaJGZxZo#+hc`ay44=iZ6Xh^;#JjuC23KY_F-pFiZp1P3c3DzF5LYqMsL) zUPGI&p;bDs(vxarDdhESzs-KXO}EkMZFxU0at)D6S)QsTQvKG+Lsf&6MUu@TiGWF^ zQ09j#y~-Esh<?G z)dSqITd}XndtENPyZme90WQA>{Kw?h_j)V-l=Vwo#6^RmlT*0%Z-w;hXZ{r!c+~m9 z^OzEa+IHzt1s`ivpemKCX+5Izu5yBE!&PPDi3B#I9(X-6`s|1<_ozyP*p$CXPi&N! zvwB?blXgjXd%FK#kO0_2B08^%@xHZTS;ELwF-pR@?WMM2jaf|oD{24QUyOd7Ti%qTdY9SaMR(&B5|>&%re z&a>Qetu{JiufL|=;!&3&|NMnXcB+Z>bW%h)K4E3K@a~77nw){pR`nUM9aC~+7 z_m!u?OPGDJbYnLV^XyIR-9h>s@?qpb0=8BbW7rC$B~&fQ&n-&NkNn*4b0qMq7Mb+w6fAH5giX=3Vp zf-Rx~$%$_xgl?*p^Q9RJ2Ad>T8Fk8N@tJtN^^l%mt=FNm@}+gXnyikjYka&wz&p>a zIp0*%<4Y?}8cobAJ2^*?S8ya{l*_5zuF5_gOLZk^MTHr(q`7Jc|`e&=28CVVH?1=$J5?)=1vQK7Dn zTZ==Zzn9h|m1Gnulwtc^qBg460?MrXFbi1m{aX7cD(w%|{=sQ+Psb^K>x;@<71LVJ z(&m73iqD_*=?Wgbh1MR%B@8Y7h(*sC-t_aic?uaT*uv$iM0{T9Na`F;&0hRSNmzV0 zLsNKc=`afA|IxLbHMr5dP#?{T4;LmT)VFnu_Va|?6$mkCl3AIh`iQ-MW;N%vK$1DD zx!;N5scu*~Z}04z>ybemihZ5<2h-95r&RCKYAd{v8!X*&R4TMkQxYmdH* z$BnSDxRj;kOp?5QA82~$sq5$^;yzlnUT2forTt{E#g}-4@cXRoy-o6<(|YFAFIHXA z`G@D)%1R#A(iTfcjh3s~VRYuk243kI#uA*gmYP8R++G? zW8Ce9zP2%`wdxz0#bJd3t^Og@qMYp{U$WgLQN6+u30(D4K|+^MlP!!To}?qTz;L}y zPg3?ssCLAxiF2*1J42w`W7Dpca`*Bet+SUm!r*-`aoDDe`X(sd-)2u)<-pWBdq-q> zhJO^T#1|9kHnOSvU3#iG!HpmP);>}D3bwI8WP`%4u%Ey)mXRnbr)`CeUiw}!y`}g~ zN10aoE$?ubEq&^4#)+i@&bnvm)kj5xYbG*>y;SFsK`yOz`5NYIc^ZZKy6}Hy?`y9*C8^cBQ z6H-^Dr+n;3rwX|)RX+{#$}OTv_h?NsL@%8w9mp8LjCRf0YSOU$cQw_nPn+Z<)662) z$SM6#xYZ2k>@3)_PXDaHKDnNUtjQ^rMvWzCUgDSSmN{HbwoHH???Bf0Oh@bdpp#*! z4q>olZrph#fB{rw_4o!mHX>{8vSJcHr(Q7hnidAFR?m`TfdwQ$m#Us&JC88 z*c)S0IFkIOX|=>o(Zjc9*PKmf?Ta@GJzA-mgQEj=Cx7ZNiNtFRQ4VCjl`PS!c)n(_ zap7!$>As9?g<_%6uszG`7s%A?EveaoUbDHuVD5JdYnj4<15*^ENK7Wa%K4B}*v!hV z=KRo1EH%TnWmV4nUUJ;ydDV8G+HaDJ%Sjk@5`WL&$dv6s^DN^}?vxqX*)lJplOJi@ zN??KDGCFl_>`s;6iCpF<>wHz(y&m;7`FV599unOiRm1Kfdprzl!uCQMvDlf>W{-qJ znAV(cR%ui|w}?KB~TWTOsy$T6-yJz3Z8+ zO4Y{0v1jrXf8P)zYgOd0Po*Ty#JHJGCJ+nY>~PZ;g&?2)jWEBw9770ClQQ?(SBOcaa@jTe+w_u=Uxd9dTkU&Kb{!pRvWZd zzY?=M8v_%S*mA8yreNbu>hQ=Fx7MEbIWvXJ?n;a5XTFav3v3fhN=F`g-Uw~>RLY)< zF?D}Si73YTr9B}SFUs^d<6X4P{FnG2|E8mP4>s((cUAeKih z@cZP#H;NmjYCi~FsGrn1^e+DhBh;qbrDUMj^QXE*`g@O|?%O_ZHQFpNhmzKwzqct= zII=BMFiO8p(Mr2k_oynKmTer+~@{Tgqh{ zc@OyzK@XDwn_(v*-%t08E4Z1Q#s}GfD)hdC07LI1OqIEIg>zTef(3cUY;q~~yhZ9t z=}Ea;Z!n@hX}o7D1aG^CyE#q0NhDXd%21`_>KoIyGxw;32~s>q2m|jY_`|X>Wfy8!hsNwRd~2<&0^R|C-#m{k*M^>3f&~0W-iB<{o8r zw>*Q`ik6>|EmO6;$#COryS5s6waL!Lg}cLW-$QEX%$$_}J@QYp0W%jQRtwMT8<@Qn z=Xep|9NX|Qz$#Zx?fQyX+{q6B2owq;oH+v%4t*XpFvIFR$gE2Dy1wvr`j?2;4)XgM zJ+)_21_ET+ISMhSw~D45D21^)`I76EI%Y0xZH>2k7Z&koDtGFP+ayPsU%zi1WqwV& zcWwO82&FVLE6;v#7MDMa9uGJjvswz+R_X!*?`l0T_ahZJY(k=rm3ws94;ZEeV9OVj zvC<;7>OWz=KOGb0aUa~m&#sl(oOiyQYl_XXS@z)L9q_8$Q(vh03OoF&7k5~2OLyR> z2HA8R%^Kq+a`FC$#xBblViFH~XI_=YuOlQ$vB7m$JO$43cAUBuQ$0{Ndj6tZne6c| zyJ|NB$4j{Olnq}U$EUBhc&&*H#OmI?$+l;?Or6_J@@q`~oEz@<0SgJZw4h^=UK2ab zc)TN=Z(LcP8t{2ov>mUaZo?rIR1q*1%rT4G!Yw?=f`=ZqO*iVq@{L0cFZ==pq~0jb z#M*24(4h}PS-sANquskU-tcVhc&s5N$PHzO=h)Gw3vTmM1lB9ww~8_v`ve zuz}WHvR&mz9D(KS=v$#WJr=-PGThCrQy9S{--n#V@z|?;g4$}uyuQaj+;FH0y3m8) z@@S%X$eub?kwdMivFYXdjVi@1db7+TvH5X#^q!(ZujH57aS`+_xit3^N1Ng`9zRzU z7C{;GZkm5}o^_3xFCdp^$txU^7UJp()zVw={#@pHuLCEMZd zb;;K%_s_>Bs`iRLNqTMSwBe*(6A-W6BJ05T+H%>1kiz$*%bSL#)-GH#V)-LL6oSyEM2R+|xud;RHbTz?lngD_&6l$``gFMoD>un7KSpA$`@ zbh6G_Eayjb=G7&02?_WaFylDiD4KHdzxnx$oB;luCvy)rTwAS+XXxZxt`!)P?anIe za>wxw^#V`h)2ea2J|Uy| zOu<@W>8CvNIr%ivamSM^Sk;DnVnQJK;K-N_EnMhYqKqLa`KSa-;`Z zdO`}ToII3SEW_9qofD;|b4m}?R7|H?q%kr!PGvF1Ha3iCj>hID=HA$_@2&6o1J3#W ze6C;a`}Ml7`@XN&`}4Wp*XQ%T?(4eSTD)j3Ds33O?Hww`vK3ni{bE#q%+EaXrv&?< zID36|F=1|%Gd#K9j9%dBM^k5FnOD~N7b}PIK3t9k*|o%vU>l;&1Us^vC_YK=~P4RqQM`i6?xwHb$tyTDBE+V znkQ4=EE$5~I0KQfK&;d^G>m+T<3Qy_XJnvvlz&7y+sIs)UXkZV*^PuVN$Xk>me|ug z+hYAokLY|Sb63Y_tg_h^NXspSF+7RDaAp~BCNV5yI=FM^)?1n8wzW61`j7iXxks5d zE{0J5N}o=Wcm%BqZ+fU?3ueMEZ|`krghVml@pjt7uK~p zs+9vH`IFtV*BP3-V=^|-lIiI~Y@xjH8Vht{(ynSyg${Ba1uWPQyuT_-M?m;HJRZfH z9eOx2cRWS6UGDjU9p0#2lg1R zH3-w9yO9JgG(TdIeR>%3txMR9FB{D(3PPg)bXqfxC8wmx?>k0Izr(?Jhy$*vxU*`08{Hz!^_I`@R;O ze+{jA_y&VyO+D>LzT!+pfbTPX$k`!&BOcVt<*aIp|@S)&^e>kUj`C*O|ni#gg*aR z4blOCuaCddbPZk5h+jm=LY~Kl)DgoJXYb%1r!8AN>kDd;D5-tJRUXnfcY_Z(i(@N0 z=g)e~{D`T2FtntB-jptaWTHK-x*$b0GDYGMY&OBkn%?YY$e>B>%X+8iH7?zga|p_$ zEL-~^?K`t)*`MtuJcncsK1~^dB5e<-llzXh%f%?AOC2&iLTKQ2=t9Ud&}6*8m$80ZX9>!|90#he z?kDWS?e6{#VO;zdgpDjVQ_f`sdq@^?7+Na(!&eDB1X;0cxBiR9`j{+=#zx|HKp86$;2hk z@Sw6Cec*_Vm@CH-7cUwZ{%d^I-NUlC*xgn>nUzLI-t>j1KfE)%49D*O6hbel__){g z>Xbj+F49iW!lVV9>Gy3jo_VCQp-x5h8X-}>u7gV0Silse@vRg?!jEDju>7i*XwLO4 zWy0CFn*c_&*L9+~)Zl$^dH8$@;mxfqUw%qBVMjLXQn{Z>qd1p-9dxj zgQZ%k;n`B2M)bW-lJK1w>BP96rdD~#XS=S5ec=GQ^B(a=Y(T7dc9{y`Q;)6c!oA88=n*? zY;HhiwG)Kx3)4RjZN!I2MtbjE=F8X!PNTM5OtrF0<&Bvo&OIG`)}VecS+1~iyJZkj zy$ezS``3OY)G&7e$+GpJOl>^JzEh7Xe3-)FHBF!IZoJSqCLJa=f}4`t^Ktx{TEp)Z zPd?=jFJq@&H)w6&EJa)gSN4SI63knh6dYzzQlE#nRk{1`p0{Ab3|hWK;FUTBvnk%y z`~9blJF_!g%~0I8PwL@Jqi-SwhKrQhzHY4@bFT8<`=3*Kjj%y6b+@7;&yL|Ue-^Ip zNmq3et_4h75+W7|QOyqZ!` z2q1@Bk>up?!0s(HfOBI-EHKlZ$qm1G`2clUs`;|00k!4k67B8_oDR{Lw()vE`KJtu zZOu*F$~tK4m(V3bion8hJUH%L{VxU^fh^EtY`w@fZ(A~ZBah_~U4V}sI`>usIcWZT zzzmdsf<+<(TPsYlMyB?xuJ_6Tv*Yl*xSX1K3(JU$ZF=YY=D~wnXr_Evwi;U8eDg>C z%%;4M6V~Tc*yosM4->t*qhMkD8J7jodq=&%9g^Fb%vq39_8iJ;dr4ZG;%XlxZ4|kN~|T0P2-=tLA{#qUe<#yV68dg zb)Mt(?(N*zxAthBl-8Q2n8w!xJ~{NJh5xX;a*?YeJvClFEpg{53Ql>)l3}Itw6e)DAxP;Fp`V6RmYOb&+0@zaTih4 z*?>Y8F*sbvc2wdD09+32hXlgOH5& zcbCig&?JE8k;n&C1z3EY(iqz3>r*H2iPKRP2_)k{{z!04;!x7eD%UT~#7FyFl_?Wl zz`dv@%HGR}5++7hdD^aS95_Ys5z$7mp&dZtIIiChy8`)SOrU#ncD-nFQos$8FqE1m z%{DcwS<0~R2N8j$!o!s2rwYw&TEgRf*!WHmD}MoWq036pmd{KmSd+1?`RJTnb$oeL zK^30f7z0nbbrMl*n&imx5rb \ No newline at end of file diff --git a/source/operations/cert-manager.rst b/source/operations/cert-manager.rst index 7c12c03c..477857c1 100644 --- a/source/operations/cert-manager.rst +++ b/source/operations/cert-manager.rst @@ -33,16 +33,16 @@ cert-manager obtains valid certificates from an ``Issuer`` or ``ClusterIssuer`` A ``ClusterIssuer`` issues certificates for multiple namespaces. An ``Issuer`` only mints certificates for its own namespace. -The following graphic depicts how various namespaces make use of either an ``Issuer`` or ``ClusterIssuer`` type. +The following graphic depicts how cert-manager provides certificates in namespaces across a Kubernetes cluster. -- cert-manager is installed in the ``cert-manager`` namespace, which does not have either an ``issuer`` or a ``ClusterIssuer``. -- The ``default`` namespace receives the global ``Cluster Issuer``. -- Each tenant's namespace receives a local ``Issuer``. -- The ``minio-operator`` namespace receives a local ``Issuer``. +- A ``ClusterIssuer`` exists at the root level of the Kubernetes cluster, typically the ``default`` namespace, to provide certificates to all other namespaces. +- The ``minio-operator`` namespace receives its own, local ``Issuer``. +- Each tenant's namespace receives its own, local ``Issuer``. +- The certificates issued by each tenant namespace must be made known to and trusted by the MinIO Operator. -.. image:: /images/k8s/cert-manager-cluster.svg +.. image:: /images/k8s/cert-manager-graph.png :width: 600px - :alt: A Kubernetes cluster with five namespaces, shown as a box for each namespace in the cluster. The minio-operator namespace contains a "minio-operator: issuer" local issuer. The default namespace contains a "root: ClusterIssuer" cluster issuer. The cert-manager namespace contains a "minio-operator: issuer" local issuer. The remaining two namespaces are individual tenants, "tenant-1" and "tenant-2", each with its own local issuer. + :alt: A graph of the namespaces in a Kubernetes cluster. At the root level is a ClusterIssuer that issues certificates to each of the other namespaces. Three other namespaces exist under root. The Operator namespace has its own Issuer, as do namespaces for Tenant-1 and Tenant-2. A dotted line going from each Tenant namespace back ot the Operator has text that the Operator must trust the Tenant certificates. :align: center From ee2d2cca50feef4e1b1726225ba5b7a101ee18f5 Mon Sep 17 00:00:00 2001 From: Daryl White <53910321+djwfyi@users.noreply.github.com> Date: Mon, 30 Sep 2024 16:42:25 -0400 Subject: [PATCH 10/11] applying PR feedback --- source/operations/cert-manager.rst | 4 ++-- .../operations/cert-manager/cert-manager-operator.rst | 10 +++++----- .../operations/cert-manager/cert-manager-tenants.rst | 4 ++-- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/source/operations/cert-manager.rst b/source/operations/cert-manager.rst index 477857c1..3315cf19 100644 --- a/source/operations/cert-manager.rst +++ b/source/operations/cert-manager.rst @@ -23,7 +23,7 @@ The guide assumes a new or fresh MinIO Operator installation. The main difference is that you must provide that ``Issuer`` CA certificate to MinIO, instead of the CA's mentioned in this guide. -Defer to the `cert-manager documentation `__ and your own organization's certificate requirements for more advanced configurations. +Refer to the `cert-manager documentation `__ and your own organization's certificate requirements for more advanced configurations. cert-manager manages certificates within Kubernetes clusters. The MinIO Operator supports using cert-manager for managing and provisioning certificates as an alternative to the MinIO Operator managing certificates for itself and its tenants. @@ -42,7 +42,7 @@ The following graphic depicts how cert-manager provides certificates in namespac .. image:: /images/k8s/cert-manager-graph.png :width: 600px - :alt: A graph of the namespaces in a Kubernetes cluster. At the root level is a ClusterIssuer that issues certificates to each of the other namespaces. Three other namespaces exist under root. The Operator namespace has its own Issuer, as do namespaces for Tenant-1 and Tenant-2. A dotted line going from each Tenant namespace back ot the Operator has text that the Operator must trust the Tenant certificates. + :alt: A graph of the namespaces in a Kubernetes cluster showing the relationship between the root level ClusterIssuer and three other namespaces with their own Issuer. :align: center diff --git a/source/operations/cert-manager/cert-manager-operator.rst b/source/operations/cert-manager/cert-manager-operator.rst index 70ed9a2f..a9ddc944 100644 --- a/source/operations/cert-manager/cert-manager-operator.rst +++ b/source/operations/cert-manager/cert-manager-operator.rst @@ -95,7 +95,7 @@ Kubernetes creates a new secret with the name ``operator-ca-tls`` in the ``minio 2) Use the secret to create the ``Issuer`` ------------------------------------------ -Use the secret created above to add an ``Issuer`` resource for the ``minio-operator`` namespace. +Use the ``operator-ca-tls`` secret to add an ``Issuer`` resource for the ``minio-operator`` namespace. 1. Create a file called ``operator-ca-issuer.yaml`` with the following contents: @@ -145,7 +145,7 @@ The certificate from cert-manager must be valid for the following DNS domains: Different Kubernetes providers manage the root domain differently. Check with your Kubernetes provider for more information. -1. Create a ``Certificate`` for the domains mentioned above: +1. Create a ``Certificate`` for the specified domains: Create a file named ``sts-tls-certificate.yaml`` with the following contents: @@ -173,7 +173,7 @@ The certificate from cert-manager must be valid for the following DNS domains: The ``spec.secretName`` is not optional. The secret name **must** be ``sts-tls``. - Confirm this by setting ``spec.secretName: sts-tls`` as highlighted above. + Confirm this by setting ``spec.secretName: sts-tls`` as highlighted in the certificate YAML. 2. Apply the resource: @@ -199,9 +199,9 @@ Disabling this environment variable prevents the MinIO Operator from issuing the Instead, Operator relies on cert-manager to issue the TLS certificate. There are various methods to define an environment variable depending on how you install the Operator. -The steps below define the variable with kustomize. +The following steps define the variable with kustomize. -1. Create a kustomization patch file called ``kustomization.yaml`` with the below contents: +1. Create a kustomization patch file called ``kustomization.yaml`` with the following contents: .. code-block:: yaml :class: copyable diff --git a/source/operations/cert-manager/cert-manager-tenants.rst b/source/operations/cert-manager/cert-manager-tenants.rst index c6a71b84..507e7048 100644 --- a/source/operations/cert-manager/cert-manager-tenants.rst +++ b/source/operations/cert-manager/cert-manager-tenants.rst @@ -146,7 +146,7 @@ The certificate must be valid for the following DNS domains: In the tenant YAML, it is defined in the the ``metadata.namespace`` field. For this example it is ``tenant-1``. -1. Request a ``Certificate`` for the domains mentioned above +1. Request a ``Certificate`` for the specified domains Create a file called ``tenant-1-minio-certificate.yaml``. The contents of the file should resemble the following, modified to reflect your cluster and tenant configurations: @@ -268,5 +268,5 @@ This allows Operator to trust the cert-manager issued CA and all certificates de With the Certificate Authority and ``Issuer`` in place for the tenant's namespace, you can now :ref:`deploy the object store tenant `. -Use the tenant YAML modified above to disable AutoCert and reference the secret you generated. +Use the modified baseline tenant YAML to disable AutoCert and reference the secret you generated. From fb4c6d8085fe72e94f9febfe874ca679ce2d5085 Mon Sep 17 00:00:00 2001 From: Daryl White <53910321+djwfyi@users.noreply.github.com> Date: Tue, 1 Oct 2024 15:16:54 -0400 Subject: [PATCH 11/11] Numbering issue noted in PR feedback --- source/operations/cert-manager/cert-manager-tenants.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/operations/cert-manager/cert-manager-tenants.rst b/source/operations/cert-manager/cert-manager-tenants.rst index 507e7048..f5ef6d1b 100644 --- a/source/operations/cert-manager/cert-manager-tenants.rst +++ b/source/operations/cert-manager/cert-manager-tenants.rst @@ -80,7 +80,7 @@ Before deploying a new tenant, create a Certificate Authority and Issuer for the kubectl apply -f tenant-1-ca-certificate.yaml -4) Create the ``Issuer`` +2) Create the ``Issuer`` ------------------------ The ``Issuer`` issues the certificates within the tenant namespace.