some annotations are being re-applied on every terraform run

[mdz]

Thanks for releasing this provider! It has saved me a lot of effort, being able to use these newer resources with terraform.

Terraform Version

Run terraform -v to show the version. If you are not running the latest version of Terraform, please upgrade because your issue may have already been fixed.

Terraform v0.11.9

• provider.google (unversioned)
• provider.google v1.17.1
• provider.kubernetes (unversioned) <--- 7974cd5
• provider.random (unversioned)
• provider.template (unversioned)

My kubernetes clusters are running in GKE:

Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.7-gke.6", GitCommit:"06898a4d0f2b96f82b43d9e59fa2825bd3d616a2", GitTreeState:"clean", BuildDate:"2018-10-02T17:32:01Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}

Affected Resource(s)

Please list the resources as a list, for example:

• kubernetes_stateful_set

Terraform Configuration Files

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key.

I have several terraform modules, each of which defines a kubernetes_stateful_set with a few annotations, like so:

resource "kubernetes_stateful_set" "foo" {
  lifecycle {
    ignore_changes = [
      # Deployments update this parameter, so terraform mustn't undo those changes
      "spec.0.template.0.spec.0.container.0.image",
      "spec.0.template.0.spec.0.container.1.image",
      "spec.0.template.0.spec.0.init_container.0.image",
    ]
  }
  metadata {
    name = "${var.name}"
    labels {
      app = "${var.name}"
    }
  }
  spec {
    replicas = "${var.nodes}"
    selector {
      app = "${var.name}"
    }
    service_name = "${var.name}"
    template {
      metadata {
        labels {
          app = "${var.name}"
        }
        annotations {
          "kubernetes.io/egress-bandwidth" = "${var.egress_bandwidth}"
          "prometheus.io/scrape" = "true"
          "prometheus.io/port" = "8080"
        }
[...]

Debug Output

With TF_LOG=DEBUG, I can see the annotations included here:

2018-10-23T04:17:52.631Z [DEBUG] plugin.terraform-provider-kubernetes: 2018/10/23 04:17:52 [INFO] Received statefulSet: &v1.StatefulSet{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, 
[...]
Annotations:map[string]string{"kubernetes.io/egress-bandwidth":"1M", "prometheus.io/port":"8080", "prometheus.io/scrape":"true"}

Expected Behavior

What should have happened?

An empty plan (because all three annotations already exist on the target resource)

Actual Behavior

What actually happened?

Terraform will perform the following actions:

  ~ module.mymodule.foo
      spec.0.template.0.metadata.0.annotations.%:                              "2" => "3"
      spec.0.template.0.metadata.0.annotations.kubernetes.io/egress-bandwidth: "" => "1M"

Even if I apply the (noop) change, it still appears the next time I run terraform. Note that the prometheus.io/ annotations seem to work as expected, but the kubernetes.io/egress-bandwidth annotations all have this issue.

Steps to Reproduce

1. terraform plan

[itmecho]

I've run into this with an AWS LoadBalancer service and the annotation that makes it an internal load balancer. Narrowed it down to a function that filters out any keys that contain kubernetes.io:

kubernetes/structures.go:165-175

func isInternalKey(annotationKey string) bool {
	u, err := url.Parse("//" + annotationKey)
	if err == nil && strings.Contains(u.Hostname(), "kubernetes.io") {
		log.Printf("[DEBUG] %s is internal key", annotationKey)
		return true
	} else if strings.Contains(annotationKey, "deprecated.daemonset.template.generation") {
		return true
	}

	return false
}

The path I took to get to that function:

kubernetes/resource_kubernetes_service.go:192 - resourceKubernetesServiceRead
kubernetes/structures.go:100 - flattenMetadata
kubernetes/structures.go:141 - removeInternalKeys
kubernetes/structures.go:165 - isInternalKey

The annotation I'm experiencing this with is:

annotations {
    "service.beta.kubernetes.io/aws-load-balancer-internal" = "0.0.0.0/0"
}

My plan output:

  ~ module.prometheus.kubernetes_service.prometheus
      metadata.0.annotations.%:                                                     "1" => "2"
      metadata.0.annotations.service.beta.kubernetes.io/aws-load-balancer-internal: "" => "0.0.0.0/0"

Output of kubectl get svc prometheus -o yaml:

metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0

and finally the line from the TRACE:

2018-10-31T15:59:48.831Z [DEBUG] plugin.terraform-provider-kubernetes: 2018/10/31 15:59:48 [DEBUG] service.beta.kubernetes.io/aws-load-balancer-internal is internal key
2018-10-31T15:59:48.831Z [DEBUG] plugin.terraform-provider-kubernetes: 2018/10/31 15:59:48 [DEBUG] removing service.beta.kubernetes.io/aws-load-balancer-internal

[itmecho]

The official provider has a "workaround" for this. It would require manually editing the provider and rebuilding it.

hashicorp#60 (comment)

[itmecho]

Kubernetes also recommends a standard set of labels, all of which container kubernetes.io
https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/

[mdz]

Thank you!