Feature Request: Custom Resource

[timthesinner]

Enabling the K8S provider to apply and manage the lifecycle for custom resources has a number of advantages:

• Allow terraform to bring an entire K8S cluster under management, without waiting for formal resource models.
• Allow terraform to manage resources for custom APIs.

Terraform Configuration Files

Inline Configuration

resource "kubernetes_custom" "cluster-issuer" {
  apiVersion = "certmanager.k8s.io/v1alpha1"
  kind       = "ClusterIssuer"

  metadata {
    name = "lets-encrypt-prod-issuer"
  }

  set {
    name = "spec"

    value = {
      acme = {
        email  = "[email protected]"
        server = "https://acme-v02.api.letsencrypt.org/directory"

        privateKeySecretRef {
          name = "lets-encrypt"
        }

        http01 {}
      }
    }
  }
}

Inline configuration

locals {
  wildcard-cert = <<WILDCARD_CERT
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
spec:
 secretName: ingress-wildcard-tls
 issuerRef:
   name: '${kubernetes_custom.cluster-issuer.metadata.name}'
   kind: ClusterIssuer
 dnsNames:
   - '*.example.com'
 acme:
   config:
     - http01:
       domains:
         - '*.example.com'
WILDCARD_CERT
}

resource "kubernetes_custom" "wildcard-ingress-certificate" {
  metadata {
    name      = "ingress-tls-wildcard"
    namespace = "kube-system"
  }

  values = "${local.wildcard-cert}"
}

File based configuration

resource "kubernetes_custom" "default-ingress-certificate" {
  metadata {
    name      = "default-ingress-tls-wildcard"
    namespace = "kube-system"
  }

  values = [
    "${file("default-certificate.yaml")}",
  ]
}

Expected Behavior

Define and manage the lifecycle of any K8S resources using the kubernetes provider.

Actual Behavior

This resource type is not supported

References

Open feature requests for formal models

• Feature request: Ingress resource #14
• Feature Request: CronJob resource. #212

[alexsomesan]

Thanks for documenting this request.

We do want to support custom resources (and generic resources in general). We've been discussing various ways to approach this for a while. The main challenge with the current state of things in Terraform is achieving a diff-ing behaviour that is consistent with current Terraform resource diffs. The plan is to wait for Terraform 0.12 to land first and try to make use of some of its upcoming enhancements in implementing such a resource.

[timthesinner]

Sounds great, thanks for the quick response.

[bryanlarsen]

The most obvious workaround is https://github.com/ericchiang/terraform-provider-k8s

But what I'm doing is creating a simple helm chart with my custom resources and using the helm provider, providing a local path as the chart name. That makes sense for us because we're using the helm provider to install a bunch of other stuff too.

[nfisher]

Hi @alexsomesan given HCL is JSON compatible what are the big obstacles you see to adopting it prior to 0.12 landing? Or is it more a matter of prioritisation and bandwidth?

[pdecat]

This was also discussed in #195

[benishak]

we at mobfox a year ago wrote a provider that allows the user to create Custom Resources and even despites the needs to create Custom Provider.

Now we are open sourcing it so Check it out here:
https://github.com/mobfox/terraform-provider-multiverse

you can even use AWS Lambda or execute a function locally in any language you like to manage your resources, it also keep state of your resource, so you can delete, read, update them too. It creates a resource, so it is not like External Data and behaves exactly like Custom Resources in AWS CloudFormation

[awfm9]

This is going to be needed to use the upcoming v2 release of traefik as ingress, as they are using custom resources to improve configuration: https://docs.traefik.io/v2.0/providers/kubernetes-crd/

[aelbarkani]

@alexsomesan now that 0.12 has landed, are there any plans to implement this feature ? it seems CRDs is the way to go according to many Kubernetes experts, and Redhat is investing a lot on it through Openshift operators. so it seems to me that it will become very soon a must have feature for Terraform I think

[kalebwalton]

Agree. Operators for 'day 2 operations' are becoming a big thing in IBM / RedHat. We're using the 'helm_resource' right now, but soon we're going to want to use an Operator CRD to manage our applications. It'll be a shame if we have to patch it into our orchestration outside of Terraform.

[davisford]

I wish the official Terraform k8s provider supported things like CRD, Jobs, and other smaller tasks that currently are unsupported, but my guess is it won't come any time real soon. I've had some luck using the nice community provider that deals with raw yaml.

The original author indicated he does not have time to push it forward any farther, so I forked it and updated it for Terraform 0.12. It is working very nicely for us. We use the Couch infrastructure which does use an operator that pushes its own CRD. So far so good. The current state of the provider needs some work with respect to diffs and patches, but it does what we need it to do.

Fork is here that updates for Go modules and [email protected]

[gabzim]

Is there an estimate ETA for this feature? As more deployments require CRDs the number of things we can deploy to k8 with terraform grows smaller and smaller (I'm looking at you cert-manager)

[alexsomesan]

We are committed to having complete coverage of the Kubernetes API in this provider. Jobs and CronJobs are going to be supported soon (PRs already open). Other smaller missing parts will follow soon after. There is no set ETA for CRDs as of now. The main impediment is fundamental limitations in the Provider SDK (the framework that connects the provider to Terraform core). These have been mentioned as blockers by the authors of most raw manifest / CRD resource attempts so far. The good news is that we are experimenting various alternatives to work around these limitations and working with the developers of Terraform core to make sure however we end up implementing CRDs does provide the same UX in regard to diffs/patching and planning behavior that users rely on with all the other resources. Finally, we are not looking to merge any “MVP” type of change that only allows for creation but not manipulation / patching of CRDs resources. Also “planning” needs to behave as expected and show the diff of changes within the CRD manifest, not just the entire “blob”. Hope this clarifies thing a bit.

On Sat 15. Jun 2019 at 21:27, Gabriel Zimmermann ***@***.***> wrote: Is there an estimate ETA for this feature? As more deployments require CRDs the number of things we can deploy to k8 with terraform grows smaller and smaller (I'm looking at you cert-manager <https://github.com/jetstack/cert-manager>) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#215>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIL5GZOVC6UEEWUJ5AS65TP2U7CHANCNFSM4GDZFLLA> .

-- — Sent from my phone.

[curtbushko]

Adding to use cases for a simple CRD option. Basically, we only care if it just passes everything onto the kubernetes API and saves the state.

We are using the istio ingressgateway and to do so you need the VirtualService kind.

I would really like to use the kubernetes provider for setting up our services instead of helm.

[arnonki]

This is a must feature it prevents us from using terraform in production

[benishak]

@arnonki
check this, it might be helpful
https://github.com/benishak/terraform-provider-multiverse

[pescetto]

@alexsomesan I know it's only been a couple months since your last update, but any chance this can get added this year? I feel like CDR's are the biggest missing feature from the kubernetes terraform provider. Thanks again and I'm really looking forward to CDR's in the provider!

[zdevi]

Hi, any news on this ? Did they abandon implementing CRD's ??? This is a vital feature for me, CRD's are used to configure backends on GCP... Please let us know what is happening, and if there is an ETA, i see it has been more than a year people are asking for that feature.

[alexsomesan]

We have not abandoned CRD support.
However, it's a radical change that requires intervention into multiple layers of the provider SDKs (and overriding some of them). It going to be released as a separate experimental provider in Alpha state when we have it working enough for testing. We hope to be able to demo it after the end of Oct.

[zdevi]

So we are talking about Years just for this feature ? I can see it has already been discussed in early 2018. It is becoming hard to continue using kubernetes provider in production as it is lacking a lot, no custom configuration possible without CRD's.... I am gona be forced to find another solution, else than the official provider, that's sad.

[tombh]

I hope this isn't off topic, but what do people consider as alternative options? I'm thinking more of the declarative approaches like Cloudformation and maybe Pulumi?

I haven't really researched much recently but it still seems to me that Terraform is the best we have, even without CRD support. Is there something inherent about Terraform's design, or declarative Infrastructure As Code frameworks in general that makes it inextricably difficult to maintain feature parity with Kubernetes?

I believe it when I hear that implementing CRD support is a significant task. But I'm happy to wait and very grateful for the efforts being made because Terraform still seems like the best approach out there. Sure I have a few YAML files here and there that I need to manually manage but it's tiny compared to the swathes of HCL. It boggles my mind every time I see Kubernetes tutorials referencing YAML snippets, how can people be productive like that!?

[benabineri]

I believe (although I may be wrong) that Pulumi uses the Terraform providers underneath so you'd probably have the same issue there.

[stuart-c]

We use Helm charts via the Helm Terraform provider, with small pieces handled directly through this provider where it doesn't make sense for a Helm chart.

[BarnumD]

@drebes That looks great. We're using terraform cloud for our deploys. Is there a way to pull in a provider like that in the cloud that's not from the public repository?

[Sytten]

@BarnumD You have to commit it in the repo in .terraform.d/plugins. See https://www.terraform.io/docs/plugins/basics.html.

[BarnumD]

@Sytten Thanks. I had to put it in terraform.d/plugins (without the dot). I also had to run git update-index --chmod=+x .\terraform.d\plugins\linux_amd64\terraform-provider-kubernetes-alpha otherwise terraform cloud would complain about permissions. Didn't see that in the link you provided.

[alexsomesan]

Support for CRD is being rolled out in the new alpha provider. You can try out the alpha provider here: https://github.com/hashicorp/terraform-provider-kubernetes-alpha

Once it's been graduated out of alpha state, it's codebase will be merged into this provider and the features will become available here as additional resources.

[Sytten]

@alexsomesan Is there a timeline/roadmap for that to happen?

[nadworny]

Do you happen to know more about the roadmap?

[0dragosh]

Also interested in a roadmap so I know what provider to choose. HashiCorp is really lagging behind on this.

[mcfedr]

I've found this provider to do a really great job with any kubernetes resources this one cannot handle, https://registry.terraform.io/providers/banzaicloud/k8s/latest/docs/resources/manifest - and now its really easy to use with the registry.

[nadworny]

thanks @mcfedr , I'll check it out

[jurgenweber]

if you weren't aware, I think they are deprecating this provider in favour of: https://github.com/hashicorp/terraform-provider-kubernetes-alpha

[aareet]

Hi folks! We're currently evaluating the feasibility of merging the manifest resource into our official provider. We are also working on some SDK improvements that are required to support the alpha provider. We'll keep our roadmap up to date with timelines and changes as soon we have them.

[pawelprazak]

Any news on the roadmap?

CRDs are GA since Kubernetes 1.16 and almost everyone needs them now that every cloud provider and project use some kind of operator.

Btw. the plugin is available in the registry, I was able to install it on tf 0.13

terraform {
  required_providers {
    kubernetes-alpha = {
      source = "hashicorp/kubernetes-alpha"
      version = "0.2.1"
    }
  }
}

provider "kubernetes-alpha" {
  # Configuration options
}

https://registry.terraform.io/providers/hashicorp/kubernetes-alpha/latest/docs/resources/kubernetes_manifest

[viceice]

Also a workaround for deploying simple yaml manifests: https://registry.terraform.io/providers/gavinbunney/kubectl/latest

[vdahmane]

Hello, do you have ETA for this useful feature?

[jurgenweber]

yeah, in the end we abstracted away from TF deploying directly....

We have TF deploying ArgoCD apps via helm charts....

ArgoCD does the rest.

[yashbhutwala]

An update on the plans on the roadmap for this feature would help assess if/whether we should use kubernetes-alpha provider or not.

Info and links from @aareet's comment from last year has not been kept up to date, i.e.: https://github.com/hashicorp/terraform-provider-kubernetes/blob/master/_about/ROADMAP.md

Hi folks! We're currently evaluating the feasibility of merging the manifest resource into our official provider. We are also working on some SDK improvements that are required to support the alpha provider. We'll keep our roadmap up to date with timelines and changes as soon we have them.

[aareet]

Hi, the SDK improvements have now shipped and we have been working on integrating them into the alpha provider. As before, we continue to work to find a way to bring the alpha provider to a production-capable state.

[alexsomesan]

Quick update here: the recently released v0.3.1 of the Kubernetes Alpha provider has quite usable support for CRDs. I would encourage everyone to give that a try a provide us with feedback. That would help us to graduate that provider to GA as soon as possible.

[nomius]

Hi folks, any updates on moving this functionality to this (stable channel) provider?

Thanks!

[nikolay]

@alexsomesan "Usable" and "alpha" can't be used in one sentence!

[asaphe]

Hi! I love the Kubernetes Alpha provider and I'm using it here https://github.com/asaphe/terraform-keda to deploy a KEDA crd.

Would love to be able to use dynamic blocks for manifest.

When will the Kubernetes Alpha provider be merged with the Kubernetes provider? I would love to already start using it and plan ahead.

[jrhouston]

The kubernetes_manifest resource from the kubernetes-alpha project has now been merged into this provider as a beta feature under an experiment flag. Here is an example of how to use it:

provider "kubernetes" {
  experiments {
    manifest_resource = true
  }
  config_path = "~/.kube/config"
}

resource "kubernetes_manifest" "example" {
  manifest = {
    apiVersion = "v1"
    kind = "ConfigMap"
    metadata = {
      name = "example"
    }
    data = {
      provider = "Terraform"
    }
  }
}

I'm going to close this issue now, as we would like to address any issues with the resource separately. Please open a new issue on this repo if you encounter any problems.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.