Support for only v1 is very limiting

[moserke]

Is there any way to remove the limitation on only resources for v1 APIs? This is incredibly limiting in the kubernetes world as many of the core components of the system live for a long time on alpha & beta APIs. Ingress is a great example of one that is not on v1.

Just wanted to spark conversation around what it would take to be able to support some of these other resources? Or is this just a hard line and there is no way it will be done?

I really want to use this provider and was very excited to see it, but just doesn't fit the bill for that reason.

[radeksimko]

Hi @moserke
I acknowledge the fact that some providers/APIs have different conventions, specifically that they keep APIs in alpha/beta for a long time.

As I mentioned in #3 (comment), #14 (comment) and #30 (comment) there are no plans for supporting alpha/beta at this point in time. The main reason being maintenance burden caused by stability promises (or lack of). In other words we don't have a good way to deal with potential breaking changes (when they come) in a way that it doesn't affect the end user.

That said folks maintaining the Google provider are evaluating one approach to this problem (overcoming BCs and supporting multiple versions of APIs) and I plan to circle back after a while to see how it goes (how is the approach accepted/understood by users and devs). Eventually we could apply the same approach here in K8S provider.

The idea of supporting alpha/beta certainly isn't shelved - there is no hard line per say. Supporting a few alpha/beta resources right now may look simple, I'm much more concerned about an ongoing support and changing APIs as that's what we have to deal with, as maintainers.

We did discuss the idea of some core schema features that would help with this, but other things have higher priority, so there are no immediate plans to make any core changes in this area in the upcoming months.

I know it's annoying, but I hope the explanation makes sense.

[davidquarles]

Hey @radeksimko! I thought I'd check back and see if you were able to sync with the Google provider team. I've been building around the stuff in there recently and I've done a little bit of work on the provider itself, but can't really speak to what headaches their new versioning support has / hasn't created. I am just about done building out our internal infrastructure stuff around GCP, though, and our next objective will target the actual applications / kubernetes objects we deploy and maintain as admins.

One of my primary goals has been to avoid additional layers / templating, if possible, to get us from the heavily-templated multi-stage workflow we have for our AWS stuff today to a single global plan/apply. So far I've gotten by using only Terraform, and I'd love to continue heading in that direction with our applications, but we'd definitely require support for the beta API (at least the workloads) for that to be a possibility. I absolutely don't want to complain – I'm just trying to contextualize, since our window for using this stuff is somewhat narrow. If you think near-term support for multiple API versions is achievable and you're open to contributions, we'd definitely be willing to help build out resources, tackle bugs, etc.

Thanks!

[radeksimko]

@davidquarles The current way of solving this in GCP provider is expressed here:

https://github.com/terraform-providers/terraform-provider-google/blob/48192d62dc3464e67c0547f32b7bbd9097e39b42/google/resource_compute_firewall.go#L18-L24

It's based on whitelisting fields per API version and choosing the API based on the field.

I only briefly discussed this with one GCP provider maintainer and I heard mostly positive feedback (from dev's perspective). I think the next step is probably finding all the common patterns between K8S & GCP provider in this area and move some of that code over to github.com/hashicorp/terraform/helper/ so that we're all on the same page cross-providers and benefit from each other's work.

Also I'd like to see at least one example of a field transitioning from one API version to another and how it's handled from user's perspective, the GCP provider has not experienced that yet AFAIK.

I'd like to focus on building out support for 1.7 and potentially 1.8 before tackling this - there's a bunch of pending PRs and other work.

I'll keep you posted.

[mishak87]

Would not provider version lock solve this issue?
If I lock provider and I'm aware of issues that applying alpha/beta resource to different kubernetes version has, it should be no problem.

[matthewhartstonge]

@mishak87 are you thinking of supplying this in the terraform provider configuration, for example hiding it behind a channel flag such as:

provider "kubernetes" {
  config_context_auth_info = "ops"
  config_context_cluster   = "mycluster"

  api_channel = "stable|alpha|beta " # Default = stable
}

As something like that would be faaaannnnntasstic and help keep things stable, for those that require it.

[mishak87]

@matthewhartstonge I would be for no api channel and simply supporting all resources that latest currently supported kubernetes api supports.

provider	supports k8s up to	alpha/beta resources	notes
1.1.0	1.6.*	cronjob_v2_alpha1 deamonset_v1_beta1	note that there is no daemonset_v1_alpha
1.2.0	1.7.*	cronjob_v2_alpha1 daemonset_v1_beta1	note that there is no daemonset_v1_alpha
1.3.0	1.8.*	cronjob_v2_alpha1 cronjob_v2_beta1 daemonset_v1_beta1 daemonset_v1_beta2
1.4.0	1.9.*	cronjob_v2_alpha1 cronjob_v2_beta1 daemonset_v1_beta1 daemonset_v1_beta2

provider "kubernetes" {
  version = "~> 1.2.0"
}

// using this resource I understand that it might not be available
// in next feature version of kubernetes provider
// and I will have to migrate to stable cronjob_v2 manually in the future
resource "cronjob_v2_alpha1" "cleanup" {
  // unicorns rainbows and happiness
}

[mishak87]

Would it be possible to support "generic" kubernetes resource with required field kind and apiVersion optional metadata and annotations and rest of the fields would be in generic field of type map[string]interface{}?

[icereed]

@mishak87 excellent approach. The missing resources for Kubernetes is killing our adoption of Terraform for Kubernetes. A generic Kubernetes resource would open many use cases. IMO this feature is one of the most needed of all Terraform providers available.

[paultyng]

Just wanted to provide some updates here, we are working on a plan similar to what @mishak87 described, its still a work in progress as some of the generic handling in kubectl isn't packaged up as easily as it could be to replicate some of its functionality, but we hope to have something soon.

[rawkode]

Did you have any luck with this implementation, @paultyng ?

[costimuraru]

Unfortunately I think this provider is ice cold. No progress whatsoever in the last 2 months, despite a lot of effort from the community.
Using the terraform kubernetes provider would have been nice, just because it meant not having to learn yet another tool.
However, given the state of things, use helm instead. Similar templating approach, but supports all k8s resources and is more mature.
Just as easy to create custom charts (aka modules) https://docs.helm.sh/chart_template_guide/#getting-started-with-a-chart-template

Plus it has a lot of preexisting charts (aka modules) https://github.com/kubernetes/charts/tree/master/stable

And because helm also uses tiller inside the actual k8s cluster, it's very convenient when you have multiple people managing the cluster.

[mishak87]

@costimuraru What about creating helm provider/module? It would solve plenty of other issues.

[ktham]

@paultyng or @radeksimko Can you comment on whether work on this provider is still ongoing despite lack of public activity on this repo, or has the effort been dropped?

[rawkode]

@mishak87 There already is a Helm provider, it works well:

https://github.com/mcuadros/terraform-provider-helm

[costimuraru]

That's actually quite nice! Thanks for sharing @rawkode

[paultyng]

@ktham see this response on the deployments issue: #3 (comment)

[alexsomesan]

Closing this as we've documented the availability to include Beta resources: https://github.com/terraform-providers/terraform-provider-kubernetes#contributing-resources