Can't install victoria-metrics-k8s-stack using Terraform helm provider

[air3ijai]

Moved from #341

I've an issue to install victoria-metrics-k8s-stack using Terraform helm provider, because of the missed CRDs.

# Victoria Metrics kubernetes monitoring stack
resource "helm_release" "vm-stack" {
  name       = "vm-stack"
  repository = "https://victoriametrics.github.io/helm-charts"
  chart      = "victoria-metrics-k8s-stack"
  version    = "0.19.4"
  namespace  = "monitoring"
  timeout    = 600
  values     = [file("${path.module}/values/vm-stack.yaml")]
}

Should we first install victoria-metrics-operator?

Values, like following does not help

victoria-metrics-operator:
  createCRD: true
  operator:
    disable_prometheus_converter: false

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: unable to build kubernetes objects from release manifest: [resource mapping not found for name: "vm-stack-victoria-metrics-k8s-stack" namespace: "monitoring" from "": no matches for kind "VMAgent" in version "operator.victoriametrics.com/v1beta1"
│ ensure CRDs are installed first, resource mapping not found for name: "vm-stack-victoria-metrics-k8s-stack" namespace: "monitoring" from "": no matches for kind "VMAlert" in version "operator.victoriametrics.com/v1beta1"
│ ensure CRDs are installed first, resource mapping not found for name: "vm-stack-victoria-metrics-k8s-stack" namespace: "monitoring" from "": no matches for kind "VMAlertmanager" in version "operator.victoriametrics.com/v1beta1"
│ ensure CRDs are installed first, resource mapping not found for name: "vm-stack-victoria-metrics-k8s-stack-cadvisor" namespace: "monitoring" from "": no matches for kind "VMNodeScrape" in version "operator.victoriametrics.com/v1beta1"

[air3ijai]

And now it is slightly better, when I've installed previously victoria-metrics-operator chart

vm-operator.tf

resource "helm_release" "vm-operator" {
  name       = "vm-operator"
  repository = "https://victoriametrics.github.io/helm-charts"
  chart      = "victoria-metrics-operator"
  version    = "0.29.2"
  namespace  = "monitoring"
  timeout    = 600
  values     = [file("${path.module}/values/vm-operator.yaml")]
}

But get a very, very long output with the error

│ Error: Provider produced inconsistent final plan
...
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

[air3ijai]

Both issues were related to the enabled hashicorp/helm experimental option

• Error: Provider produced inconsistent final plan when install Datadog from Helm provider 2.7.0 hashicorp/terraform-provider-helm#979
• Experiments Feature Breaking the helm_release asking for CRD to install First hashicorp/terraform-provider-helm#1217

  experiments {
    manifest = true
  }

And a working config may looks like following
main.tf

# Terraform
terraform {
  required_version = "~> 1.7"
  required_providers {
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.1"
    }
  }
}

# Providers
provider "helm" {
  kubernetes {
    config_path = "/etc/rancher/k3s/k3s.yaml"
  }
  # experiments {
  #   manifest = true
  # }
}

# Victoria Metrics kubernetes monitoring stack
resource "helm_release" "vm-stack" {
  name             = "vm-stack"
  repository       = "https://victoriametrics.github.io/helm-charts"
  chart            = "victoria-metrics-k8s-stack"
  version          = "0.19.4"
  namespace        = "monitoring"
  create_namespace = true
  timeout          = 600
  values           = [file("${path.module}/vm-stack.yaml")]
}

vm-stack.yaml

# https://github.com/VictoriaMetrics/helm-charts/tree/master/charts/victoria-metrics-k8s-stack

# VM Single
vmsingle:
  # https://docs.victoriametrics.com/operator/api.html#vmsinglespec
  spec:
    retentionPeriod: "90"
    storage:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 100Gi