Skip to content

Latest commit

 

History

History
1297 lines (898 loc) · 68.2 KB

README.md

File metadata and controls

1297 lines (898 loc) · 68.2 KB
Raw

Automate Everything using Terraform and Flux CD

Introduction

Terraform helps you create and automate the running infrastructure, like your Kubernetes cluster. Then, Flux CD helps you synchronize the state of your infrastructure using Git as the source of truth, and follow GitOps principles.

In this tutorial, you will learn to:

  • Use Terraform modules, to automate all the steps required to provision your infrastructure.
  • Create Flux CD resources, to keep your Kubernetes cluster applications state synchronized with a Git repository (use GitOps principles).

After finishing all the steps from this tutorial, you should have a fully functional DOKS cluster with Flux CD deployed, that will:

DOKS and Flux CD Automation Overview

DOKS-FluxCD-Automation-Overview

Table of Contents

Prerequisites

To complete this tutorial, you will need:

  1. A GitHub repository and branch, for Flux CD to store your cluster and sample application manifests.

    Important note:

    The GitHub repository (and branch) must be created beforehand - the DigitalOcean Terraform module used in this tutorial doesn't provision one for you automatically.

  2. A GitHub personal access token that has the repo permissions set. The Terraform module used in this tutorial, needs it in order to create the SSH deploy key, as well as to commit the Flux CD manifests in your Git repository.

  3. A DigitalOcean access token, for creating/managing the DOKS cluster. Please follow the official DigitalOcean tutorial on how to create a personal access token. Copy the token value and save it somewhere safe.

  4. A dedicated DO Spaces bucket for the following:

    • Terraform state file (e.g.: terraform).
    • Loki data storage (e.g.: loki-storage).
    • Velero backups storage (e.g.: velero-backups-storage).

    Please follow the official DigitalOcean tutorial on how to create each. Make sure that it is set to restrict file listing for security reasons.

  5. Access keys, for DigitalOcean Spaces (S3-compatible object storage). Please follow the official DigitalOcean tutorial on how to manage access keys. Copy the key and secret value, and save each in a local environment variable for later use (make sure to replace the <> placeholders accordingly):

    export DO_SPACES_ACCESS_KEY="<YOUR_DO_SPACES_ACCESS_KEY>"
export DO_SPACES_SECRET_KEY="<YOUR_DO_SPACES_SECRET_KEY>"

  6. A Git client, for cloning the Starter Kit repository.

  7. HashiCorp Terraform CLI, for provisioning the infrastructure.

  8. Doctl CLI, for DigitalOcean API interaction.

  9. Kubectl CLI, for Kubernetes interaction.

  10. Flux CLI, for Flux CD interaction.

  11. Kubeseal, for encrypting secrets and Sealed Secrets Controller interaction.

Understanding Flux CD Concepts for Automated Helm Releases

The way Flux CD knows how to handle Helm releases is via a dedicated custom resource definition, the HelmRelease. Next, each HelmRelease makes use of a HelmRepository CRD, to fetch the required chart to install.

You have some familiarity now with Helm installations performed via the command line interface (each section of the Starter Kit makes use of it). Flux CD is achieving the same thing via the Helm Controller. You define manifests that make use of specific CRDs (HelmRepository, HelmRelease), which in turn tell Flux CD how to perform the same steps as you did when using the Helm CLI.

Using HelmRepository CRD to Define Helm Repositories

The HelmRepository CRD is used by Flux CD to handle Helm repositories, and to fetch the available charts (the CLI equivalent of helm repo add <name> <url> and helm repo update).

Typical structure of a HelmRepository manifest, looks like below:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
  name: ambassador
  namespace: flux-system
spec:
  interval: 10m0s
  url: https://app.getambassador.io

Explanations for the above configuration:

  • <kind>: Tells Flux CD what type of CRD the manifest is using (HelmRepository, in this example).
  • <metadata.name>: Name of the HelmRepository CRD (ambassador, in this example).
  • <metadata.namespace>: Namespace to use for the HelmRepository resource (flux-system, in this example).
  • <spec.interval>: Time interval to use for synchronizing with the remote Helm repository (to fetch new chart releases metadata - CLI equivalent of: helm repo update).
  • <spec.url>: Helm charts repository URL.

Using HelmRelease CRD to Install Helm Charts

The HelmRelease CRD is used by Flux CD to handle Helm releases in your DOKS cluster (the CLI equivalent of helm install <name> <chart> -f <values.yaml>). It can also take care of upgrades, as well as how to recover in case of a failure, like: how many retries to perform for a failed install operation, rollbacks, etc.

Each HelmRelease makes use of a source type, so that it knows where to pull the Helm chart from:

Typical structure of a HelmRelease manifest, looks like below:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: ambassador-stack
  namespace: flux-system
spec:
  chart:
    spec:
      chart: edge-stack
      sourceRef:
        kind: HelmRepository
        name: ambassador
      version: 7.2.2
  install:
    createNamespace: true
  interval: 1m0s
  targetNamespace: ambassador

Explanations for the above configuration:

  • <kind>: Tells Flux CD what type of CRD the manifest is using (HelmRelease, in this example).
  • <metadata.name>: Name of the HelmRelease CRD (ambassador-stack, in this example).
  • <metadata.namespace>: Namespace to use for the HelmRelease resource (flux-system, in this example).
  • <chart.spec.chart>: Specifies the chart name to fetch and install from the HelmRepository (ambassador, in this example).
  • <chart.spec.sourceRef.kind>: Tells Flux CD to use a HelmRepository resource for fetching Helm charts (this is a good example that illustrates CRDs dependency).
  • <chart.spec.sourceRef.name>: HelmRepository CRD name to reference.
  • <chart.spec.version>: version of the Helm chart to install.
  • <spec.install.createNamespace>: Tells Flux CD if a namespace needs to be created before installing the chart (Helm CLI equivalent of: helm install --create-namespace).
  • <spec.interval>: Time interval at which to reconcile the Helm release.
  • <spec.targetNamespace>: Namespace to place the Helm release (Helm CLI equivalent of: helm install --namespace <name>).

Using HelmRelease CRD to Override Helm Values

Each Flux CD HelmRelease can override values via a values file - the equivalent of helm install <name> <chart> -f values.yaml, or by setting each value individually - the equivalent of helm instal <name> <chart> --set <key> <value>.

Flux CD lets you override Helm values via two spec types:

  • <spec.values>: Allows you to override values inline as seen in a standard values.yaml file). This is the equivalent of: helm install -f <values_file>.
  • <spec.valuesFrom>: Allows you to override values individually, by using each key fully qualified path from the values file (e.g.: aws.access_key_id). This is the equivalent of helm install --set <key> <value>.

Typical usage of spec.values:

...
spec:
  values:
    loki:
      enabled: true
      config:
        schema_config:
          configs:
            - from: "2020-10-24"
              store: boltdb-shipper
              object_store: aws
              schema: v11
              index:
                prefix: index_
                period: 24h
...

Explanations for the above configuration:

<spec.values>: This key holds the original Helm values file content, just as when using the CLI variant: helm install -f <values.yaml>.

When using HelmReleases, the <spec.values> field copies the values file content exactly as seen in the YAML file. It means that sensitive data, like DO Spaces credentials or DO API token will be exposed, and that is not a good thing. What Flux CD allows you to do in this case, is to override values from Kubernetes Secrets.

Typical usage of spec.valuesFrom in combination with a Kubernetes Secret (the below example expects that the do-credentials Kubernetes secret to be created beforehand):

spec:
  valuesFrom:
    - kind: Secret
      name: do-credentials
      valuesKey: spaces_access_key
      targetPath: aws.access_key_id

Explanations for the above configuration:

  • <spec.valuesFrom.kind>: Kind of the values referent (Kubernetes Secret, in this example - can be a ConfigMap as well).
  • <spec.valuesFrom.name>: Name of the values referent (e.g. Kubernetes Secret), in the same namespace as the HelmRelease.
  • <spec.valuesFrom.valuesKey>: The data key where a specific value can be found (defaults to values.yaml when omitted).
  • <spec.valuesFrom.targetPath>: The YAML dot notation path at which the value should be merged. When set, the valuesKey is expected to be a single flat value (defaults to None when omitted, which results in the values getting merged at the root).

You can have a combination of spec.values for storing non-sensitive data, and spec.valuesFrom that reference Kubernetes secrets for sensitive data. In the end, each Flux CD HelmRelease will merge all the values together, so you can benefit from both worlds. In fact, this is how Helm works, so the same principles apply.

In a GitOps flow, Kubernetes Secrets cannot be stored directly using Git, so you're going to use Sealed Secrets to encrypt sensitive data.

Using Sealed Secrets Controller to Encrypt Kubernetes Secrets

Sealed Secrets allows you to encrypt generic Kubernetes secrets and store them safely in Git (even in public repositories). Then, Flux CD will create a corresponding Sealed Secret Object in your cluster when syncing the Git repository. Sealed Secrets Controller notices the sealed objects, and decrypts each to a classic Kubernetes secret. Applications can consume the secrets as usual.

Flux CD Sealed Secrets GitOps Flow:

Flux CD Sealed Secrets GitOps Flow

For more details, please refer to Section 08 - Encrypt Kubernetes Secrets Using Sealed Secrets.

Next, you will learn how to provision the DOKS cluster and Flux CD, using Terraform.

Step 1 - Initializing Terraform Backend

In this step, you're going to initialize the Terraform backend. A DO Spaces bucket for storing the Terraform state file is highly recommended because you do not have to worry about exposing sensitive data, as long as the space is private of course. Another advantage is that the state of your infrastructure is backed up, so you can re-use it when the workspace is lost. Having a shared space for team members is desired as well, in order to perform collaborative work via Terraform.

First, clone the Starter Kit Git repository on your local machine, and navigate to the terraform directory:

git clone https://github.com/digitalocean/Kubernetes-Starter-Kit-Developers.git

cd Kubernetes-Starter-Kit-Developers/15-automate-with-terraform-flux/assets/terraform

Next, rename the backend.tf.sample file provided by the Starter Kit repository to backend.tf:

cp backend.tf.sample backend.tf

Then, open backend.tf using a text editor of your choice (preferably with Terraform lint support), and replace the <> placeholders accordingly (explanations for each can be found inside). For example, you can use VS Code:

code backend.tf

Finally, initialize the Terraform backend. You're going to use the DO Spaces access and secret key, created at Step 5. from Prerequisites:

terraform init  --backend-config="access_key=$DO_SPACES_ACCESS_KEY" --backend-config="secret_key=$DO_SPACES_SECRET_KEY"

The output looks similar to (check for the Successfully configured the backend "s3"! message):

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Finding hashicorp/kubernetes versions matching "2.3.2"...
- Finding gavinbunney/kubectl versions matching "1.11.2"...
...

In the next step, you will create the DOKS cluster and provision Flux CD, using Terraform.

Step 2 - Bootstrapping DOKS and Flux CD

In this step, you're going to use a custom Terraform module provided by DigitalOcean - DOKS-FluxCD, to provision your Kubernetes cluster and deploy Flux CD.

First, change directory to 15-automate-with-terraform-flux/assets/terraform, from where you cloned the Starter Kit Git repository:

cd Kubernetes-Starter-Kit-Developers/15-automate-with-terraform-flux/assets/terraform

Next, rename the main.tf.sample file provided by the Starter Kit repository to main.tf:

cp main.tf.sample main.tf

Then, open main.tf using an editor of your choice (preferably with Terraform lint support), and replace the <> placeholders accordingly (explanations for each can be found inside). For example, you can use VS Code:

code main.tf

Next, run Terraform and inspect infrastructure changes:

terraform plan -out starter_kit_flux_cluster.out

Finally, if everything looks good, apply changes using Terraform:

terraform apply "starter_kit_flux_cluster.out"

After running above steps, you should have a fully functional Kubernetes cluster and Flux CD deployed. In the next step, you're going to inspect the state of your DOKS cluster, as well as Flux CD.

Step 3 - Inspecting DOKS Cluster and Flux CD State

First, please navigate to DOKS panel from your DigitalOcean account, to visualize Kubernetes cluster state:

DOKS state

Next, check that the Terraform state file is saved in your DO Spaces bucket. Bucket listing looks similar to:

DO Spaces Terraform state file

Finally, the Flux CD manifests for your DOKS cluster should be present in your Git repository as well:

GIT repo state

DOKS Cluster

First, you have to set the kubectl context to point to your DOKS cluster. List the available Kubernetes clusters first:

doctl k8s cluster list

Point kubectl to your cluster (make sure to replace the <> placeholders accordingly):

doctl k8s cluster kubeconfig save <your_doks_cluster_name>

Please check that the context was set, and that it's pointing to your Kubernetes cluster:

kubectl config get-contexts

List cluster nodes, and make sure that they're in a healthy state (STATUS column says Ready):

kubectl get nodes

The output looks similar to:

NAME                             STATUS   ROLES    AGE    VERSION
test-fluxcd-cluster-pool-8z9df   Ready    <none>   3d2h   v1.21.3
test-fluxcd-cluster-pool-8z9dq   Ready    <none>   3d2h   v1.21.3
test-fluxcd-cluster-pool-8z9dy   Ready    <none>   3d2h   v1.21.3

Flux CD

Flux provides a CLI binary, which you can use for provisioning Flux CD itself, as well as for main system interaction. You can also use it to export manifests and create Flux CD resources for your DOKS cluster.

First, you can perform some sanity checks via:

flux check

The output looks similar to the following:

► checking prerequisites
✔ kubectl 1.21.3 >=1.18.0-0
✔ Kubernetes 1.21.2 >=1.16.0-0
► checking controllers
✗ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.11.1
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.13.1
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.15.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.15.3
✔ all checks passed

Then, inspect all Flux CD resources via:

flux get all

The output looks similar to the following (long commit hashes were abbreviated in the output, for simplicity). Notice the gitrepository/flux-system component fetching the latest revision from your main branch, as well as the kustomization/flux-system component:

NAME                      READY MESSAGE                        REVISION      SUSPENDED 
gitrepository/flux-system True  Fetched revision: main/1d69... main/1d69...  False     

NAME                      READY MESSAGE                        REVISION      SUSPENDED 
kustomization/flux-system True  Applied revision: main/1d69... main/1d69c... False

In case you need to perform some troubleshooting, and see what Flux CD is doing, you can access the logs via:

flux logs

The output looks similar to the following:

...
2021-07-20T12:31:36.696Z info GitRepository/flux-system.flux-system - Reconciliation finished in 1.193290329s, next run in 1m0s 
2021-07-20T12:32:37.873Z info GitRepository/flux-system.flux-system - Reconciliation finished in 1.176637507s, next run in 1m0s 
...

Finally, check that Flux CD points to your Git repository:

kubectl get gitrepositories.source.toolkit.fluxcd.io -n flux-system

The output looks similar to (notice the URL column value - should point to your Git repository, and the READY state set to True):

NAME         URL                                                       READY  STATUS                          AGE
flux-system  ssh://[email protected]/test-starterkit/starterkit_fra1.git  True   Fetched revision: main/1d69...  21h

In the next step, you will prepare the Git repository layout for use in this tutorial. Although Flux CD doesn't care about the directory structure of your Git repository (it performs a recursive search for manifests), it's best practice to keep things organized, and follow naming conventions as much as possible, to avoid frustration in the future.

Step 4 - Cloning the Flux CD Git Repository and Preparing the Layout

In this step, you will learn how to organize your Git repository used by Flux CD to sync your DOKS cluster state. For simplicity, this tutorial is based on a monorepo structure, and is using a single environment to hold all your manifests.

Please make sure that the following steps are performed in order, before anything else:

  1. First, clone your Flux CD Git repository. This is the main repository used for your DOKS cluster reconciliation (please replace the <> placeholders accordingly):

    git clone https://github.com/<YOUR_GITHUB_USER>/<YOUR_GITHUB_REPOSITORY_NAME>.git

    Explanations for the above command:

    • <YOUR_GITHUB_USER> - your GitHub username as defined in main.tf file (<github_user> variable).
    • <YOUR_GITHUB_REPOSITORY_NAME> - your Flux CD GitHub repository used for your DOKS cluster reconciliation as defined in main.tf file (<git_repository_name> variable).

  2. Next, change directory where your Flux CD Git repository was cloned.

  3. Then, checkout the correct branch as defined in main.tf file for Flux CD to use (<git_repository_branch> variable).

  4. Now, create the directory structure to store Flux CD HelmRepository, HelmRelease and SealedSecret manifests for each component of the Starter Kit. Please replace <YOUR_GIT_REPOSITORY_SYNC_PATH> with your Flux CD cluster sync directory path, as defined by the <git_repository_sync_path> variable from main.tf file:

    FLUXCD_HELM_MANIFESTS_PATH="<YOUR_GIT_REPOSITORY_SYNC_PATH>/helm"

mkdir -p "${FLUXCD_HELM_MANIFESTS_PATH}/repositories" 

mkdir -p "${FLUXCD_HELM_MANIFESTS_PATH}/releases" 

mkdir -p "${FLUXCD_HELM_MANIFESTS_PATH}/secrets"

  5. Finally, add the .gitignore file to avoid committing unencrypted Helm value files in your repository, that may contain sensitive data. Using your favorite text editor, paste the following (below example is using the Starter Kit naming convention):

    # Ignore all YAML files containing the `-values-` string.
*-values-*.yaml

# Do not ignore sealed YAML files.
!*-sealed.yaml

After finishing all the steps from this tutorial, you should have a Git repository structure similar to:

├── README.md
├── clusters
│   └── dev
│       ├── flux-system
│       │   ├── gotk-components.yaml
│       │   ├── gotk-sync.yaml
│       │   └── kustomization.yaml
│       └── helm
│           ├── releases
│           │   ├── ambassador-stack-v7.2.2.yaml
│           │   ├── loki-stack-v2.5.1.yaml
│           │   ├── prometheus-stack-v30.0.1.yaml
│           │   ├── sealed-secrets-v2.0.2.yaml
│           │   └── velero-v2.27.3.yaml
│           ├── repositories
│           │   ├── ambassador.yaml
│           │   ├── grafana.yaml
│           │   ├── prometheus-community.yaml
│           │   ├── sealed-secrets.yaml
│           │   └── vmware-tanzu.yaml
│           └── secrets
│               ├── do-api-credentials-sealed.yaml
│               ├── do-spaces-credentials-sealed.yaml
│               └── prometheus-stack-credentials-sealed.yaml
└── pub-sealed-secrets-dev-cluster.pem

Next, you're going to provision the required Flux CD manifests for each component of the Starter Kit. Then, you will inspect and commit each manifest to your Git repository used by Flux CD to reconcile your DOKS cluster. For sensitive data, a Kubernetes Secrets will be created and encrypted using Sealed Secrets, and then stored in your Git repository as well.

First example will make use of the Flux CLI for you to accommodate and get familiarized with creating manifests via the CLI. Then, you will use the already prepared manifests provided by the Starter Kit repository, to speed up the steps from this tutorial.

You're going to start with the Sealed Secrets Helm release first, because it's a prerequisite for the rest of the Starter Kit components.

Step 5 - Creating the Sealed Secrets Helm Release

In this step, you will learn how to create manifests using the Flux CLI, to define the Sealed Secrets Helm release. Then, Flux will trigger the Sealed Secrets Controller installation process for your DOKS cluster.

Please use the following steps, to create required manifests for the Sealed Secrets Helm release:

  1. First, change directory where your Flux CD Git repository was cloned. Also, please check that the required directory structure for this tutorial is created, and that the FLUXCD_HELM_MANIFESTS_PATH environment variable is set (please refer to Step 4 - Cloning the Flux CD Git Repository and Preparing the Layout, for details).

  2. Then, create the Sealed Secrets HelmRepository manifest for Flux:

    flux create source helm sealed-secrets \
  --url="https://bitnami-labs.github.io/sealed-secrets" \
  --interval="10m" \
  --export > "${FLUXCD_HELM_MANIFESTS_PATH}/repositories/sealed-secrets.yaml"

    Explanations for the above command:

    • --url: Helm repository address.
    • --interval: Source sync interval (default 1m0s).
    • --export: Export in YAML format to stdout.

    The output looks similar to (you can notice that it has a similar structure as explained in Using HelmRepository CRD to Define Helm Repositories):

    ---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
  name: sealed-secrets
  namespace: flux-system
spec:
  interval: 10m0s
  url: https://bitnami-labs.github.io/sealed-secrets

  3. Next, fetch the Starter Kit values file for Sealed Secrets. Please make sure to inspect the values file first, and replace the <> placeholders where needed:

    SEALED_SECRETS_CHART_VERSION="2.0.2"

curl "https://raw.githubusercontent.com/digitalocean/Kubernetes-Starter-Kit-Developers/main/08-kubernetes-sealed-secrets/assets/manifests/sealed-secrets-values-v${SEALED_SECRETS_CHART_VERSION}.yaml" > "sealed-secrets-values-v${SEALED_SECRETS_CHART_VERSION}.yaml"

  4. Now, create the Sealed Secrets HelmRelease manifest for Flux CD. Kubeseal CLI expects by default to find the controller in the kube-system namespace and to be named sealed-secrets-controller, hence we override the release name via the --release-name and --target-namespace flags. This is not mandatory, but kube-system is usually accessible only to power users (administrators):

    SEALED_SECRETS_CHART_VERSION="2.0.2"

flux create helmrelease "sealed-secrets-controller" \
  --release-name="sealed-secrets-controller" \
  --source="HelmRepository/sealed-secrets" \
  --chart="sealed-secrets" \
  --chart-version "$SEALED_SECRETS_CHART_VERSION" \
  --values="sealed-secrets-values-v${SEALED_SECRETS_CHART_VERSION}.yaml" \
  --target-namespace="flux-system" \
  --crds=CreateReplace \
  --export > "${FLUXCD_HELM_MANIFESTS_PATH}/releases/sealed-secrets-v${SEALED_SECRETS_CHART_VERSION}.yaml"

    Explanations for the above command:

    • --release-name: What name to use for the Helm release (defaults to <target-namespace>-<HelmRelease-name> otherwise).
    • --source: Source that contains the chart in the format <kind>/<name>.<namespace>, where kind must be one of: (HelmRepository, GitRepository, Bucket).
    • --chart: Helm chart name.
    • --chart-version: Helm chart version.
    • --values: Local path to values file.
    • --target-namespace: Namespace to install this release.
    • --crds: Upgrade CRDs policy, available options are: (Skip, Create, CreateReplace).
    • --export: Export in YAML format to stdout.

    The output looks similar to (you can observe that it has a similar structure as explained in Using HelmRelease CRD to Install Helm Charts):

    ---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: sealed-secrets-controller
  namespace: flux-system
spec:
  chart:
    spec:
      chart: sealed-secrets
      sourceRef:
        kind: HelmRepository
        name: sealed-secrets
      version: 2.0.2
  interval: 1m0s
  releaseName: sealed-secrets-controller
  targetNamespace: flux-system
  install:
    crds: Create
  upgrade:
    crds: CreateReplace
  values:
    ingress:
      enabled: false

  5. Finally, commit Git changes to remote branch:

    SEALED_SECRETS_CHART_VERSION="2.0.2"

git add "${FLUXCD_HELM_MANIFESTS_PATH}/repositories/sealed-secrets.yaml"

git add "${FLUXCD_HELM_MANIFESTS_PATH}/releases/sealed-secrets-v${SEALED_SECRETS_CHART_VERSION}.yaml"

git commit -am "Adding Sealed Secrets manifests for Flux CD"

git push origin

After completing the above steps, Flux CD will start your DOKS cluster reconciliation (in about one minute or so, if using the default interval). If you don't want to wait, you can always force reconciliation via:

flux reconcile source git flux-system

After a few moments, please inspect the Flux CD Sealed Secrets Helm release:

flux get helmrelease sealed-secrets-controller

The output looks similar to:

NAME                        READY   MESSAGE                                 REVISION        SUSPENDED 
sealed-secrets-controller   True    Release reconciliation succeeded        2.0.2          False

Look for the READY column value - it should say True. Reconciliation status is displayed in the MESSAGE column, along with the REVISION number, which represents the Helm chart version. Please bear in mind that some releases take longer to complete (like Prometheus stack, for example), so please be patient.

Hints:

  • The MESSAGE column will display Reconciliation in progress, as long as the HelmController is performing the installation for the specified Helm chart. If something goes wrong, you'll get another message stating the reason, so please make sure to check Helm release state.

  • You can use the --watch flag for example: flux get helmrelease <name> --wait, to wait until the command finishes. Please bear in mind that in this mode, Flux will block your terminal prompt until the default timeout of 5 minutes occurs (can be overridden via the --timeout flag).

  • In case something goes wrong, you can search the Flux logs, and filter HelmRelease messages only:

    flux logs --kind=HelmRelease

Exporting the Sealed Secrets Controller Public Key

To be able to encrypt secrets, you need the public key that was generated by the Sealed Secrets Controller when it was deployed by Flux CD in your DOKS cluster.

First, change directory where you cloned your Flux CD Git repository, and do the following (please replace the <> placeholders accordingly):

kubeseal --controller-namespace=flux-system --fetch-cert > pub-sealed-secrets-<YOUR_DOKS_CLUSTER_NAME_HERE>.pem

Note:

If for some reason the kubeseal certificate fetch command hangs, you can use the following steps to work around this issue:

  • First, open another terminal window, and expose the Sealed Secrets Controller service on your localhost (you can use CTRL - C to terminate, after fetching the public key):

    kubectl port-forward service/sealed-secrets-controller 8080:8080 -n flux-system &

  • Then, you can go back to your working terminal and fetch the public key (please replace the <> placeholders accordingly):

    curl --retry 5 --retry-connrefused localhost:8080/v1/cert.pem > pub-sealed-secrets-<YOUR_DOKS_CLUSTER_NAME_HERE>.pem

Finally, commit the public key file to remote Git repository for later use (it's safe to do this, because the public key is useless without the private key which is stored in your DOKS cluster only). Please run bellow commands, and make sure to replace the <> placeholders accordingly:

git add pub-sealed-secrets-<YOUR_DOKS_CLUSTER_NAME_HERE>.pem

git commit -m "Adding Sealed Secrets public key for cluster <YOUR_DOKS_CLUSTER_NAME_HERE>"

git push origin

Important note:

In this tutorial the flux-system namespace is used to hold Kubernetes Secrets, so please make sure that it is restricted to regular users/ applications via RBAC.

Next, you're going to perform similar steps to define Helm releases for the remaining components of the Starter Kit.

Step 6 [OPTIONAL] - Creating the Cert-Manager Helm Release

If you want to have wildcard certificates support for your cluster, you need to provision Cert-Manager as well. This step is also required if proper TLS termination is needed for the Nginx Ingress Controller.

Steps to follow:

  1. First, change directory where your Flux CD Git repository was cloned. Also, please check that the required directory structure for this tutorial is created, and that the FLUXCD_HELM_MANIFESTS_PATH environment variable is set (please refer to Step 4 - Cloning the Flux CD Git Repository and Preparing the Layout, for details).

  2. Then, fetch the Jetstack HelmRepository manifest file provided by the Starter Kit Git repository:

    curl "https://raw.githubusercontent.com/digitalocean/Kubernetes-Starter-Kit-Developers/main/15-automate-with-terraform-flux/assets/manifests/fluxcd/helm/repositories/jetstack.yaml" > "${FLUXCD_HELM_MANIFESTS_PATH}/repositories/jetstack.yaml"

  3. Now, fetch the Cert-Manager HelmRelease manifest file provided by the Starter Kit Git repository:

    CERT_MANAGER_CHART_VERSION="1.6.1"

curl "https://raw.githubusercontent.com/digitalocean/Kubernetes-Starter-Kit-Developers/main/15-automate-with-terraform-flux/assets/manifests/fluxcd/helm/releases/cert-manager-v${CERT_MANAGER_CHART_VERSION}.yaml" > "${FLUXCD_HELM_MANIFESTS_PATH}/releases/cert-manager-v${CERT_MANAGER_CHART_VERSION}.yaml"

  4. Next, inspect the downloaded HelmRelease manifest file using an editor of your choice (preferably with YAML lint support), and adjust to your needs. For example, you can use VS Code (make sure to replace the <> placeholders accordingly, if present):

    CERT_MANAGER_CHART_VERSION="1.6.1"

code "${FLUXCD_HELM_MANIFESTS_PATH}/releases/cert-manager-v${CERT_MANAGER_CHART_VERSION}.yaml"

  5. Finally, commit Git changes to remote branch:

    CERT_MANAGER_CHART_VERSION="1.6.1"

git add "${FLUXCD_HELM_MANIFESTS_PATH}/repositories/jetstack.yaml"

git add "${FLUXCD_HELM_MANIFESTS_PATH}/releases/cert-manager-v${CERT_MANAGER_CHART_VERSION}.yaml"

git commit -am "Adding Cert-Manager manifests for Flux CD"

git push origin

After completing the above steps, Flux CD will start your DOKS cluster reconciliation (in about one minute or so, if using the default interval). If you don't want to wait, you can always force reconciliation via:

flux reconcile source git flux-system

After a few moments, please inspect the HelmRelease status:

flux get helmrelease cert-manager

The output looks similar to:

NAME                READY   MESSAGE                                 REVISION       SUSPENDED 
cert-manager        True    Release reconciliation succeeded        v1.6.1         False

Look for the READY column value - it should say True. Reconciliation status is displayed in the MESSAGE column, along with the REVISION number, which represents the Helm chart version. Please bear in mind that some releases take longer to complete (like Prometheus stack, for example), so please be patient.

Hints:

  • The MESSAGE column will display Reconciliation in progress, as long as the HelmController is performing the installation for the specified Helm chart. If something goes wrong, you'll get another message stating the reason, so please make sure to check Helm release state.

  • You can use the --watch flag for example: flux get helmrelease <name> --wait, to wait until the command finishes. Please bear in mind that in this mode, Flux will block your terminal prompt until the default timeout of 5 minutes occurs (can be overridden via the --timeout flag).

  • In case something goes wrong, you can search the Flux logs, and filter HelmRelease messages only:

    flux logs --kind=HelmRelease

Next, you're going to create Flux CD manifests for the Ambassador (or Nginx) ingress.

Step 7 - Creating the Ingress Controller Helm Release

In this step, you will use pre-made manifests to create your preferred Ingress Controller Helm release for Flux CD. Then, Flux will trigger the Ingress Controller installation process for your DOKS cluster.

There are two options available, depending on what Ingress Controller you feel most comfortable:

  • Ambassador Ingress Helm Release.
  • Nginx Ingress Helm Release.

Steps to follow:

  1. First, change directory where your Flux CD Git repository was cloned. Also, please check that the required directory structure for this tutorial is created, and that the FLUXCD_HELM_MANIFESTS_PATH environment variable is set (please refer to Step 4 - Cloning the Flux CD Git Repository and Preparing the Layout, for details).

  2. Then, fetch the HelmRepository manifest file provided by the Starter Kit Git repository (please pick only one option, depending on what Ingress Controller you want to install and configure):

    Ambassador Ingress:

    curl "https://raw.githubusercontent.com/digitalocean/Kubernetes-Starter-Kit-Developers/main/15-automate-with-terraform-flux/assets/manifests/fluxcd/helm/repositories/ambassador.yaml" > "${FLUXCD_HELM_MANIFESTS_PATH}/repositories/ambassador.yaml"

    Nginx Ingress:

    curl "https://raw.githubusercontent.com/digitalocean/Kubernetes-Starter-Kit-Developers/main/15-automate-with-terraform-flux/assets/manifests/fluxcd/helm/repositories/kubernetes-community-nginx.yaml" > "${FLUXCD_HELM_MANIFESTS_PATH}/repositories/kubernetes-community-nginx.yaml"

  3. Now, fetch the HelmRelease manifest file provided by the Starter Kit Git repository (please pick only one option, depending on what Ingress Controller you want to install and configure):

    Ambassador Ingress:

    AMBASSADOR_CHART_VERSION="7.2.2"

curl "https://raw.githubusercontent.com/digitalocean/Kubernetes-Starter-Kit-Developers/main/15-automate-with-terraform-flux/assets/manifests/fluxcd/helm/releases/ambassador-stack-v${AMBASSADOR_CHART_VERSION}.yaml" > "${FLUXCD_HELM_MANIFESTS_PATH}/releases/ambassador-stack-v${AMBASSADOR_CHART_VERSION}.yaml"

    Nginx Ingress:

    NGINX_CHART_VERSION="4.0.13"

curl "https://raw.githubusercontent.com/digitalocean/Kubernetes-Starter-Kit-Developers/main/15-automate-with-terraform-flux/assets/manifests/fluxcd/helm/releases/nginx-v${NGINX_CHART_VERSION}.yaml" > "${FLUXCD_HELM_MANIFESTS_PATH}/releases/nginx-v${NGINX_CHART_VERSION}.yaml"

  4. Next, inspect the downloaded HelmRelease manifest file using an editor of your choice (preferably with YAML lint support), and adjust to your needs. For example, you can use VS Code. Make sure to replace the <> placeholders accordingly, if present ((please pick only one option, depending on what Ingress Controller you want to install and configure):

    Ambassador Ingress:

    AMBASSADOR_CHART_VERSION="7.2.2"

code "${FLUXCD_HELM_MANIFESTS_PATH}/releases/ambassador-stack-v${AMBASSADOR_CHART_VERSION}.yaml"

    Nginx Ingress:

    NGINX_CHART_VERSION="4.0.13"

code "${FLUXCD_HELM_MANIFESTS_PATH}/releases/nginx-v${NGINX_CHART_VERSION}.yaml"

    Hint:

    Notice how the HelmRelease manifest configures remediation actions (you can visit Configuring Failure Remediation, for more details about Helm install/upgrade failure remediation options available in Flux CD):

    ...
install:
  createNamespace: true
  remediation:
    retries: 3
upgrade:
  remediation:
    retries: 3
...

  5. Finally, commit Git changes to remote branch (please pick only one option, depending on what Ingress Controller you want to install and configure):

    Ambassador Ingress:

    AMBASSADOR_CHART_VERSION="7.2.2"

git add "${FLUXCD_HELM_MANIFESTS_PATH}/repositories/ambassador.yaml"

git add "${FLUXCD_HELM_MANIFESTS_PATH}/releases/ambassador-stack-v${AMBASSADOR_CHART_VERSION}.yaml"

git commit -am "Adding Ambassador manifests for Flux CD"

git push origin

    Nginx Ingress:

    NGINX_CHART_VERSION="4.0.13"

git add "${FLUXCD_HELM_MANIFESTS_PATH}/repositories/kubernetes-community-nginx.yaml"

git add "${FLUXCD_HELM_MANIFESTS_PATH}/releases/nginx-v${NGINX_CHART_VERSION}.yaml"

git commit -am "Adding Nginx manifests for Flux CD"

git push origin

After completing the above steps, Flux CD will start your DOKS cluster reconciliation (in about one minute or so, if using the default interval). If you don't want to wait, you can always force reconciliation via:

flux reconcile source git flux-system

After a few moments, please inspect the HelmRelease status (please pick only one option, depending on the installed Ingress Controller):

Ambassador Ingress:

flux get helmrelease ambassador-stack

The output looks similar to:

NAME                    READY   MESSAGE                                 REVISION        SUSPENDED
ambassador-stack        True    Release reconciliation succeeded        7.2.2          False

Nginx Ingress:

flux get helmrelease ingress-nginx

The output looks similar to:

NAME                 READY   MESSAGE                                 REVISION       SUSPENDED
ingress-nginx        True    Release reconciliation succeeded        4.0.13          False

Look for the READY column value - it should say True. Reconciliation status is displayed in the MESSAGE column, along with the REVISION number, which represents the Helm chart version. Please bear in mind that some releases take longer to complete (like Prometheus stack, for example), so please be patient.

Hints:

  • The MESSAGE column will display Reconciliation in progress, as long as the HelmController is performing the installation for the specified Helm chart. If something goes wrong, you'll get another message stating the reason, so please make sure to check Helm release state.

  • You can use the --watch flag for example: flux get helmrelease <name> --wait, to wait until the command finishes. Please bear in mind that in this mode, Flux will block your terminal prompt until the default timeout of 5 minutes occurs (can be overridden via the --timeout flag).

  • In case something goes wrong, you can search the Flux logs, and filter HelmRelease messages only:

    flux logs --kind=HelmRelease

Please refer to the Ambassador Ingress or Nginx Ingress tutorial, for more details about checking Ingress Controller deployment status and functionality.

Next, you're going to create Flux CD manifests for the Prometheus stack.

Step 8 - Creating the Prometheus Stack Helm Release

In this step, you will use pre-made manifests to create the Prometheus Helm release for Flux CD. Then, Flux will trigger the Prometheus installation process for your DOKS cluster. The Prometheus stack deploys Grafana as well, so you need to set the administrator credentials for accessing the dashboards. You will learn how to use kubeseal CLI with Sealed Secrets Controller to encrypt sensitive data stored in Kubernetes Secrets. Then, you will see how the Flux CD HelmRelease manifest is used to reference Grafana credentials stored in the Kubernetes Secret.

Steps to follow:

  1. First, change directory where your Flux CD Git repository was cloned. Also, please check that the required directory structure for this tutorial is created, and that the FLUXCD_HELM_MANIFESTS_PATH environment variable is set (please refer to Step 4 - Cloning the Flux CD Git Repository and Preparing the Layout, for details).

  2. Then, fetch the Prometheus HelmRepository manifest provided by the Starter Kit Git repository:

    curl "https://raw.githubusercontent.com/digitalocean/Kubernetes-Starter-Kit-Developers/main/15-automate-with-terraform-flux/assets/manifests/fluxcd/helm/repositories/prometheus-community.yaml" > "${FLUXCD_HELM_MANIFESTS_PATH}/repositories/prometheus-community.yaml"

  3. Next, create and encrypt the Kubernetes Secret holding your Grafana dashboard admin password. Make sure that you have the Sealed Secrets public key exported as mentioned in Exporting the Sealed Secrets Controller Public Key (please replace the <> placeholders accordingly):

    SEALED_SECRETS_PUB_KEY="<YOUR_SEALED_SECRETS_PUB_KEY_NAME_HERE>"
GRAFANA_ADMIN_PASSWORD="<YOUR_GRAFANA_ADMIN_PASSWORD_HERE>"

kubectl create secret generic "prometheus-stack-credentials" \
    --namespace flux-system \
    --from-literal=grafana_admin_password="${GRAFANA_ADMIN_PASSWORD}" \
    --dry-run=client -o yaml | kubeseal --cert="${SEALED_SECRETS_PUB_KEY}" \
    --format=yaml > "${FLUXCD_HELM_MANIFESTS_PATH}/secrets/prometheus-stack-credentials-sealed.yaml"

    Explanations for the above command:

    • --namespace: Namespace where the Kubernetes secret should be created.
    • --from-literal: Create a Kubernetes secret from a literal value containing the grafana_admin_password. The prometheus-stack-credentials secret and grafana_admin_password value is used by the prometheus-stack-v30.0.1.yaml manifest (spec.valuesFrom key).
    • --dry-run=client: Exports the Kubernetes secret on your local machine using standard output (and afterwards, piped to kubeseal to encrypt the final result).

  4. Now, fetch the Prometheus HelmRelease manifest provided by the Starter Kit Git repository:

    PROMETHEUS_CHART_VERSION="30.0.1"

curl "https://raw.githubusercontent.com/digitalocean/Kubernetes-Starter-Kit-Developers/main/15-automate-with-terraform-flux/assets/manifests/fluxcd/helm/releases/prometheus-stack-v${PROMETHEUS_CHART_VERSION}.yaml" > "${FLUXCD_HELM_MANIFESTS_PATH}/releases/prometheus-stack-v${PROMETHEUS_CHART_VERSION}.yaml"

  5. Then, inspect the downloaded Prometheus HelmRelease manifest using an editor of your choice (preferably with YAML lint support), and adjust to your needs. For example, you can use VS Code (please make sure to replace the <> placeholders accordingly, if present):

    PROMETHEUS_CHART_VERSION="30.0.1"

code "${FLUXCD_HELM_MANIFESTS_PATH}/releases/prometheus-stack-v${PROMETHEUS_CHART_VERSION}.yaml"

    Note: You can see how the prometheus-stack-credentials secret is referenced inside the Prometheus HelmRelease manifest, by looking at below snippet:

    valuesFrom:
  - kind: Secret
    name: prometheus-stack-credentials # Kubernetes secret holding Prometheus stack credentials
    valuesKey: grafana_admin_password  # Key from the `prometheus-stack-credentials` secret, holding the Grafana admin password
    targetPath: grafana.adminPassword  # Helm value to override
    optional: false  # Helm release will fail if value is not found

  6. Finally, commit Git changes to remote branch:

    PROMETHEUS_CHART_VERSION="30.0.1"

git add "${FLUXCD_HELM_MANIFESTS_PATH}/repositories/prometheus-community.yaml"

git add "${FLUXCD_HELM_MANIFESTS_PATH}/releases/prometheus-stack-v${PROMETHEUS_CHART_VERSION}.yaml"

git add "${FLUXCD_HELM_MANIFESTS_PATH}/secrets/prometheus-stack-credentials-sealed.yaml"

git commit -am "Adding Prometheus manifests for Flux CD"

git push origin

After a few moments, please inspect the Prometheus HelmRelease:

flux get helmrelease kube-prometheus-stack

The output looks similar to:

NAME                    READY   MESSAGE                                 REVISION        SUSPENDED 
kube-prometheus-stack   True    Release reconciliation succeeded        30.0.1          False

Look for the READY column value - it should say True. Reconciliation status is displayed in the MESSAGE column, along with the REVISION number, which represents the Helm chart version. Please bear in mind that some releases take longer to complete (like Prometheus stack, for example), so please be patient.

Hints:

  • The MESSAGE column will display Reconciliation in progress, as long as the HelmController is performing the installation for the specified Helm chart. If something goes wrong, you'll get another message stating the reason, so please make sure to check Helm release state.

  • You can use the --watch flag for example: flux get helmrelease <name> --wait, to wait until the command finishes. Please bear in mind that in this mode, Flux will block your terminal prompt until the default timeout of 5 minutes occurs (can be overridden via the --timeout flag).

  • In case something goes wrong, you can search the Flux logs, and filter HelmRelease messages only:

    flux logs --kind=HelmRelease

Now, check if the prometheus-stack-credentials Kubernetes secret was created as well (then, you can use kubectl get secret prometheus-stack-credentials -n flux-system -o yaml for secret contents inspection):

kubectl get secret prometheus-stack-credentials -n flux-system

Finally, perform a quick check of Prometheus stack main services, and PVC:

  • Prometheus dashboard: kubectl port-forward svc/kube-prom-stack-kube-prome-prometheus 9090:9090 -n monitoring.
  • Grafana dashboards: kubectl --namespace monitoring port-forward svc/kube-prom-stack-grafana 3000:80.
  • Prometheus PVC: kubectl get pvc -n monitoring.

Please refer to the 04-setup-prometheus-stack tutorial, for more details about checking Loki stack deployment status and functionality.

Next, you're going to create the manifests for Loki stack, and let Flux CD handle the Helm release automatically.

Step 9 - Creating the Loki Stack Helm Release

In this step, you will use pre-made manifests to create the Loki Helm release for Flux CD. Then, Flux will trigger the Loki installation process for your DOKS cluster. Loki needs a DO Spaces bucket for storing backups, hence you need to use DO Spaces credentials. You will learn how to use kubeseal CLI with Sealed Secrets Controller to encrypt sensitive data stored in Kubernetes Secrets. Then, you will see how the Flux CD HelmRelease manifest is used to reference DO Spaces credentials stored in the Kubernetes Secret.

Steps to follow:

  1. First, change directory where your Flux CD Git repository was cloned. Also, please check that the required directory structure for this tutorial is created, and that the FLUXCD_HELM_MANIFESTS_PATH environment variable is set (please refer to Step 4 - Cloning the Flux CD Git Repository and Preparing the Layout, for details).

  2. Then, fetch the Loki HelmRepository manifest provided by the Starter Kit Git repository:

    curl "https://raw.githubusercontent.com/digitalocean/Kubernetes-Starter-Kit-Developers/main/15-automate-with-terraform-flux/assets/manifests/fluxcd/helm/repositories/grafana.yaml" > "${FLUXCD_HELM_MANIFESTS_PATH}/repositories/grafana.yaml"

  3. Next, create and encrypt the Kubernetes secret for your DO Spaces credentials. Make sure that you have the Sealed Secrets public key exported as mentioned in Exporting the Sealed Secrets Controller Public Key (please replace the <> placeholders accordingly):

    SEALED_SECRETS_PUB_KEY="<YOUR_SEALED_SECRETS_PUB_KEY_NAME_HERE>"
DO_SPACES_ACCESS_KEY="<YOUR_DO_SPACES_ACCESS_KEY_HERE>"
DO_SPACES_SECRET_KEY="<YOUR_DO_SPACES_SECRET_KEY_HERE>"

kubectl create secret generic "do-spaces-credentials" \
   --namespace flux-system \
   --from-literal=access_key_id="${DO_SPACES_ACCESS_KEY}" \
   --from-literal=secret_access_key="${DO_SPACES_SECRET_KEY}" \
   --dry-run=client -o yaml | kubeseal --cert="${SEALED_SECRETS_PUB_KEY}" \
   --format=yaml > "${FLUXCD_HELM_MANIFESTS_PATH}/secrets/do-spaces-credentials-sealed.yaml"

    Explanations for the above command:

    • --namespace: Namespace where the Kubernetes secret should be created.
    • --from-literal: Create a Kubernetes secret from a literal value containing the DO Spaces access_key_id and secret_access_key. The do-spaces-credentials secret and access_key_id/secret_access_key value is used by the loki-stack-v2.5.1.yaml manifest (spec.valuesFrom key).
    • --dry-run=client: Exports the Kubernetes secret on your local machine using standard output (and afterwards, piped to kubeseal to encrypt the final result).

  4. Now, fetch the Loki stack HelmRelease manifest provided by the Starter Kit Git repository:

    LOKI_CHART_VERSION="2.5.1"

curl "https://raw.githubusercontent.com/digitalocean/Kubernetes-Starter-Kit-Developers/main/15-automate-with-terraform-flux/assets/manifests/fluxcd/helm/releases/loki-stack-v${LOKI_CHART_VERSION}.yaml" > "${FLUXCD_HELM_MANIFESTS_PATH}/releases/loki-stack-v${LOKI_CHART_VERSION}.yaml"

  5. Then, inspect the downloaded Loki HelmRelease manifest using an editor of your choice (preferably with YAML lint support), and adjust to your needs. For example, you can use VS Code (please make sure to replace the <> placeholders accordingly, if present):

    LOKI_CHART_VERSION="2.5.1"

code "${FLUXCD_HELM_MANIFESTS_PATH}/releases/loki-stack-v${LOKI_CHART_VERSION}.yaml"

    Note: You can see how the do-spaces-credentials secret is referenced inside the Loki HelmRelease manifest, by looking at below snippet:

    valuesFrom:
  - kind: Secret
    name: do-spaces-credentials  # Kubernetes secret holding DO Spaces credentials
    valuesKey: access_key_id     # Key from the `do-spaces-credentials` secret, holding the DO Spaces Access Key ID
    targetPath: loki.config.storage_config.aws.access_key_id  # Helm value to override
    optional: false  # Helm release will fail if value is not found
  - kind: Secret
    name: do-spaces-credentials
    valuesKey: secret_access_key  # Key from the `do-spaces-credentials` secret, holding the DO Spaces Secret Key
    targetPath: loki.config.storage_config.aws.secret_access_key
    optional: false

  6. Finally, commit Git changes to remote branch:

    LOKI_CHART_VERSION="2.5.1"

git add "${FLUXCD_HELM_MANIFESTS_PATH}/repositories/grafana.yaml"

git add "${FLUXCD_HELM_MANIFESTS_PATH}/releases/loki-stack-v${LOKI_CHART_VERSION}.yaml"

git add "${FLUXCD_HELM_MANIFESTS_PATH}/secrets/do-spaces-credentials-sealed.yaml"

git commit -am "Adding Loki stack manifests for Flux CD"

git push origin

After a few moments, please inspect the Loki HelmRelease:

flux get helmrelease loki-stack

The output looks similar to:

NAME         READY   MESSAGE                                 REVISION       SUSPENDED 
loki-stack   True    Release reconciliation succeeded        2.5.1          False

Look for the READY column value - it should say True. Reconciliation status is displayed in the MESSAGE column, along with the REVISION number, which represents the Helm chart version. Please bear in mind that some releases take longer to complete (like Prometheus stack, for example), so please be patient.

Hints:

  • The MESSAGE column will display Reconciliation in progress, as long as the HelmController is performing the installation for the specified Helm chart. If something goes wrong, you'll get another message stating the reason, so please make sure to check Helm release state.

  • You can use the --watch flag for example: flux get helmrelease <name> --wait, to wait until the command finishes. Please bear in mind that in this mode, Flux will block your terminal prompt until the default timeout of 5 minutes occurs (can be overridden via the --timeout flag).

  • In case something goes wrong, you can search the Flux logs, and filter HelmRelease messages only:

    flux logs --kind=HelmRelease

Finally, check if the do-spaces-credentials Kubernetes secret was created as well (then, you can use kubectl get secret do-spaces-credentials -n flux-system -o yaml for secret contents inspection):

kubectl get secret do-spaces-credentials -n flux-system

Please refer to the 05-setup-loki-stack tutorial, for more details about checking Loki Stack deployment status and functionality.

Next, you're going to create the manifests and let Flux CD handle the Velero Helm release automatically.

Step 10 - Creating the Velero Helm Release

In this step, you will use pre-made manifests to create the Velero Helm release for Flux CD. Then, Flux will trigger the Velero installation process for your DOKS cluster. Velero needs a DO Spaces bucket for storing backups, hence you need to use DO Spaces credentials. You will learn how to use kubeseal CLI with Sealed Secrets Controller to encrypt sensitive data stored in Kubernetes Secrets. Then, you will see how the Flux CD HelmRelease manifest is used to reference DO API credentials stored in the Kubernetes Secret.

Important note:

Before following the steps, make sure that the do-spaces-credentials sealed secret was provisioned to your DOKS cluster as detailed in Step 8 - Creating the Loki Stack Helm Release.

Steps to follow:

  1. First, change directory where your Flux CD Git repository was cloned. Also, please check that the required directory structure for this tutorial is created, and that the FLUXCD_HELM_MANIFESTS_PATH environment variable is set (please refer to Step 4 - Cloning the Flux CD Git Repository and Preparing the Layout, for details).

  2. Then, fetch the Velero HelmRepository manifest provided by the Starter Kit Git repository:

    curl "https://raw.githubusercontent.com/digitalocean/Kubernetes-Starter-Kit-Developers/main/15-automate-with-terraform-flux/assets/manifests/fluxcd/helm/repositories/vmware-tanzu.yaml" > "${FLUXCD_HELM_MANIFESTS_PATH}/repositories/vmware-tanzu.yaml"

  3. Next, create and encrypt the Kubernetes secret for Velero HelmRelease to consume (DO API TOKEN). Make sure that you have the Sealed Secrets public key exported as mentioned in Exporting the Sealed Secrets Controller Public Key (please replace the <> placeholders accordingly):

    SEALED_SECRETS_PUB_KEY="<YOUR_SEALED_SECRETS_PUB_KEY_NAME_HERE>"
DO_API_TOKEN="<YOUR_DO_API_TOKEN_HERE>"

kubectl create secret generic "do-api-credentials" \
   --namespace flux-system \
   --from-literal=do_api_token="${DO_API_TOKEN}" \
   --dry-run=client -o yaml | kubeseal --cert="${SEALED_SECRETS_PUB_KEY}" \
   --format=yaml > "${FLUXCD_HELM_MANIFESTS_PATH}/secrets/do-api-credentials-sealed.yaml"

    Explanations for the above command:

    • --namespace: Namespace where the Kubernetes secret should be created.
    • --from-literal: Create a Kubernetes secret from a literal value containing the DO API token. The do-credentials secret and do_api_token value is used by the velero-v2.27.3.yaml manifest (spec.valuesFrom key).
    • --dry-run=client: Exports the Kubernetes secret on your local machine using standard output (and afterwards, piped to kubeseal to encrypt the final result).

  4. Now, fetch the Loki stack HelmRelease manifest provided by the Starter Kit Git repository:

    VELERO_CHART_VERSION="2.27.3"

curl "https://raw.githubusercontent.com/digitalocean/Kubernetes-Starter-Kit-Developers/main/15-automate-with-terraform-flux/assets/manifests/fluxcd/helm/releases/velero-v${VELERO_CHART_VERSION}.yaml" > "${FLUXCD_HELM_MANIFESTS_PATH}/releases/velero-v${VELERO_CHART_VERSION}.yaml"

  5. Then, inspect the downloaded Velero HelmRelease manifest using an editor of your choice (preferably with YAML lint support), and adjust to your needs. For example, you can use VS Code (please make sure to replace the <> placeholders accordingly, if present):

    VELERO_CHART_VERSION="2.27.3"

code "${FLUXCD_HELM_MANIFESTS_PATH}/releases/velero-v${VELERO_CHART_VERSION}.yaml"

    Notes:

    • You can see how do-api-credentials and do-spaces-credentials secrets are referenced inside the Velero HelmRelease manifest, by looking at below snippet:

      valuesFrom:
  - kind: Secret
    name: do-api-credentials  # Kubernetes secret holding DO API token
    valuesKey: do_api_token   # Key from the `do-api-credentials` secret, holding the DO API token
    targetPath: configuration.extraEnvVars.DIGITALOCEAN_TOKEN  # Helm value to override
    optional: false  # Helm release will fail if value is not found
  - kind: Secret
    name: do-spaces-credentials  # Kubernetes secret holding DO Spaces credentials
    valuesKey: access_key_id     # Key from the `do-spaces-credentials` secret, holding the DO Spaces Access Key ID
    targetPath: configuration.extraEnvVars.AWS_ACCESS_KEY_ID
    optional: false
  - kind: Secret
    name: do-spaces-credentials
    valuesKey: secret_access_key  # Key from the `do-spaces-credentials` secret, holding the DO Spaces Secret Key
    targetPath: configuration.extraEnvVars.AWS_SECRET_ACCESS_KEY
    optional: false

    • Please bear in mind that the above secret data is placed inside environment variables (configuration.extraEnvVars), so values will be visible in clear text if you describe the Velero deployment (on production environments you can restrict velero namespace access via RBAC, for example).

      kubectl describe deployment velero -n velero | grep AWS_ACCESS_KEY_ID

  6. Finally, commit Git changes to remote branch:

    VELERO_CHART_VERSION="2.27.3"

git add "${FLUXCD_HELM_MANIFESTS_PATH}/repositories/vmware-tanzu.yaml"

git add "${FLUXCD_HELM_MANIFESTS_PATH}/releases/velero-v${VELERO_CHART_VERSION}.yaml"

git add "${FLUXCD_HELM_MANIFESTS_PATH}/secrets/do-api-credentials-sealed.yaml"

git commit -am "Adding Velero manifests for Flux CD"

git push origin

After a few moments, please inspect the Velero HelmRelease:

flux get helmrelease velero-stack

The output looks similar to:

NAME                 READY  MESSAGE                             REVISION    SUSPENDED 
velero-stack         True   Release reconciliation succeeded    2.27.3      False

Look for the READY column value - it should say True. Reconciliation status is displayed in the MESSAGE column, along with the REVISION number, which represents the Helm chart version. Please bear in mind that some releases take longer to complete (like Prometheus stack, for example), so please be patient.

Hints:

  • The MESSAGE column will display Reconciliation in progress, as long as the HelmController is performing the installation for the specified Helm chart. If something goes wrong, you'll get another message stating the reason, so please make sure to check Helm release state.

  • You can use the --watch flag for example: flux get helmrelease <name> --wait, to wait until the command finishes. Please bear in mind that in this mode, Flux will block your terminal prompt until the default timeout of 5 minutes occurs (can be overridden via the --timeout flag).

  • In case something goes wrong, you can search the Flux logs, and filter HelmRelease messages only:

    flux logs --kind=HelmRelease

Finally, check if the do-api-credentials Kubernetes secret was created as well (then, you can use kubectl get secret do-api-credentials -n flux-system -o yaml for secret contents inspection):

kubectl get secret do-api-credentials -n flux-system

Please refer to the Velero tutorial, for more details about checking Velero deployment status and functionality.

Conclusion

In this tutorial, you learned the automation basics for a GitOps based setup. You learned about Terraform modules, and how to re-use configuration to provision the required infrastructure, targeting DOKS cluster and Flux CD.

Then, you configured Flux CD to perform Helm releases for you automatically by creating specific manifests, and deploy all the Starter Kit components in a GitOps fashion. Finally, you applied security best practices as well, by making use of Sealed Secrets to encrypt sensitive data for your applications to use (Helm releases).

Going further, Flux CD supports other interesting Controllers as well, which can be configured and enabled, like:

You can visit the official Flux CD Guides page for more interesting stuff and ideas, like how to structure your Git repositories, as well as application manifests for multi-cluster and multi-environment setups.