Can not set internal annotations to specify load balancer ssl certs #60

udangel-r7 · 2017-08-25T16:16:38Z

With the recent change of disallowing internal annotations it is no longer possible to setup internal annotations.

I want to use these internal annotations to provide SSL certificates/arn configuration for the load balancers. Any idea for a workaround?

Affected Resource(s)

service

Terraform Configuration Files

"aws_elb_hosted_zone_id" "main" {}

data "aws_acm_certificate" "hello" {
  domain   = "*.hello.com"
  statuses = ["ISSUED"]
}

resource "kubernetes_service" "hello" {
  metadata {
    name = "terraform-hello-integrated-example"

    annotations {
      "service.beta.kubernetes.io/aws-load-balancer-ssl-cert" = "${data.aws_acm_certificate.hello.arn}"
    }
  }

  spec {
    selector {
      app  = "hello"
    }

    port {
      port             = 80
      target_port = 80
    }

    type = "LoadBalancer"
  }
}

resource "aws_route53_record" "hello" {
  zone_id = "HELLO"
  name    = "hello"
  type    = "A"

  alias {
    name                   = "${kubernetes_service.hello.load_balancer_ingress.0.hostname}"
    zone_id                = "${data.aws_elb_hosted_zone_id.main.id}"
    evaluate_target_health = true
  }
}

radeksimko · 2017-09-01T19:19:24Z

Hi @udangel-r7
I'm sorry for any inconvenience caused by this.

Allowing users to specify internal (kubernetes.io) annotations is something I plan to look into eventually, but it's non trivial for us to support. See full explanation at #50 (comment)

fbsb · 2017-09-05T19:50:27Z

Hi, I am also affected by this change. Is there any known workaround to configure SSL for the load balancers on AWS?

frankthelen · 2017-10-05T11:59:36Z

The same problem exists when your LB shall not have an external IP, e.g., in development environments with only internal IPs. I'd be happy if this issue can be solved. Thank you.

annotations {
   "service.beta.kubernetes.io/aws-load-balancer-internal" = "0.0.0.0/0"
}

jsullivanLI · 2017-10-25T15:42:05Z

Is there a previous version of Terraform where internal annotations are allowed?

pdecat · 2017-10-30T09:45:30Z

Hi, same issue for me, as I need to specify the well known pv.beta.kubernetes.io/gid annotation on my persistent volumes.

I've read #50 (comment) and understand that the root issue from the terraform point of view is more with dynamic kubernetes annotations that potentially change all the time like those listed at https://kubernetes.io/docs/reference/labels-annotations-taints/ :

beta.kubernetes.io/arch
beta.kubernetes.io/os
kubernetes.io/hostname
beta.kubernetes.io/instance-type
failure-domain.beta.kubernetes.io/region
failure-domain.beta.kubernetes.io/zone

As maintaining the list of such annotations exhaustively can seem overwhelming if not impossible as time goes, what about allowing to whitelist annotations by configuration?

I guess doing so at the provider level would be practical:

provider "kubernetes" {
  internal_annotations_whitelist = [
    "pv.beta.kubernetes.io/gid",
    "service.beta.kubernetes.io/aws-load-balancer-internal",
    "service.beta.kubernetes.io/aws-load-balancer-ssl-cert",
 ]
}

What do you think?

PS: I'll use the pod security context's fs_group field to resolve for my current FS GID issue, but I believe whitelisting will come in handy anyway.

Edit: #325 is a better alternative as it does not require the user to actively maintain the whitelist.

bobbytables · 2018-01-10T21:54:21Z

This is pretty annoying, why enforce this at all? Doesn't kubernetes API prevent this?

Phylu · 2018-02-26T16:34:05Z

Actually there is a good list regarding the aws specific annotations here: https://gist.github.com/mgoodness/1a2926f3b02d8e8149c224d25cc57dc1

I think a whitelist would be the best solution currently.

bitglue · 2018-04-02T00:21:07Z

Further broken use cases include assigning an ELB to a service: https://gist.github.com/mgoodness/1a2926f3b02d8e8149c224d25cc57dc1

And allowing syscalls in pods: https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/#setting-sysctls-for-a-pod

offlinehacker · 2018-04-09T07:23:44Z

By my opinion this check is unneeded, it's up to user to not use these annotations, and use them when he knows what he is doing.

r-moiseev · 2018-04-18T08:21:11Z

Same here, I am trying to make storage class to be default. And I need storageclass.kubernetes.io/is-default-class annotation.

Using of this annotation is not restricted by Kubernetes and documented there https://kubernetes.io/docs/tasks/administer-cluster/change-default-storage-class/

kattelus · 2018-05-14T10:54:55Z

Same problem with azure internal load balancer:
https://docs.microsoft.com/en-us/azure/aks/internal-lb
service.beta.kubernetes.io/azure-load-balancer-internal: "true"

kattelus · 2018-05-16T07:19:11Z

Any possibility allow annotation usage e.g. change this check to just output warning or something? Now this means that everyone using e.g. internal load balancers in azure or aws or otherwise defining additional parameters to those fail? Or is there any easy way to make those work in other ways?

bitglue · 2018-05-17T17:09:47Z

Please read #50 (comment), which explains why this check exists.

Phylu · 2018-05-18T07:56:52Z

I think that especially because of the reasoning behind #50 this is a big issue for many of us.

Unfortunately, the usage after this change showed that some tasks that were possible before are not possible anymore. For those of us who rely on these features (internal load balancers, aws certificates, ...) this makes those things really hard do achieve.

Therefore, I would be really interested in feasible solution such as some whitelisting of certain annotations or similar.

chreichert · 2018-05-25T10:03:54Z

Same problem here, when having the external IP for a loadbalancer inside a different resource group on azure:
The annotation needed for this is not allowed:
service.beta.kubernetes.io/azure-load-balancer-resource-group: RG_SANDBOX_006_QA_PROD_PublicIP_Pool

Whitelisting these necessary annotations would be highly appreciated.
The disallowing renders Terraform useless for us atm.

TechyMatt · 2018-06-12T17:53:51Z

I'm also now hit with the bug and having the list of approved annotations would seriously help. I'm not a developer and had never even ventured into Go before, but managed to piece together the following extension to the existing function validateAnnotations in https://github.com/terraform-providers/terraform-provider-kubernetes/blob/master/kubernetes/validators.go.

func validateAnnotations(value interface{}, key string) (ws []string, es []error) {
	m := value.(map[string]interface{})
	for k, _ := range m {
		errors := utilValidation.IsQualifiedName(strings.ToLower(k))
		if len(errors) > 0 {
			for _, e := range errors {
				es = append(es, fmt.Errorf("%s (%q) %s", key, k, e))
			}
		}

		permittedInternalKubAnnotations := map[string]bool {
			"service.beta.kubernetes.io/azure-load-balancer-internal": true,
			"service.beta.kubernetes.io/azure-load-balancer-resource-group": true,
			"service.beta.kubernetes.io/aws-load-balancer-access-log-emit-interval": true,
			"service.beta.kubernetes.io/aws-load-balancer-access-log-enabled": true,
			"service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name": true,
			"service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix": true,
			"service.beta.kubernetes.io/aws-load-balancer-backend-protocol": true,
			"service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled": true,
			"service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout": true,
			"service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout": true,
			"service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled": true,
			"service.beta.kubernetes.io/aws-load-balancer-internal": true,
			"service.beta.kubernetes.io/aws-load-balancer-proxy-protocol": true,
			"service.beta.kubernetes.io/aws-load-balancer-ssl-cert": true,
			"service.beta.kubernetes.io/aws-load-balancer-ssl-ports": true,
		}
		
		val := permittedInternalKubAnnotations[k];
	
		if isInternalKey(k) && val != true {
			es = append(es, fmt.Errorf("%s: %q is internal Kubernetes annotation", key, k))
		}
	}
	return
}

I've tested the above and it works with my specific use case of needing an internal load balancer in Azure for provisioning a POC. Whilst i understand why the check for internal providers was built in the first place, this is severely impacting the usefulness of the Kubernetes provider without the ability for this exception.

If there is appetite for me to submit a pull request i'm more than happy to but I suspect there is a significantly more efficient way of the above check :). I'll also have to figure out the part of that stops the attempt to re-enable the load balancer every time as this is what happens when I re-run:

  ~ kubernetes_service.nginx
      metadata.0.annotations.%:                                                       "0" => "1"
      metadata.0.annotations.service.beta.kubernetes.io/azure-load-balancer-internal: "" => "true"

juhan88 · 2018-10-26T22:40:39Z

Are there plan to support annotations? Specifically creating kubernetes service's with internal private elbs

mbarrien · 2018-10-30T07:26:11Z

Here's another set of "internal" kubernetes.io annotations that are disallowed: https://github.com/kubernetes-sigs/aws-alb-ingress-controller/blob/master/docs/ingress-resources.md#annotations. This ingress controller is absolutely unusable without whitelisting the large list.

C'mon guys, the examples where this is blocking people for legitimate usage are far outweighing and far outnumbering the original reasons for blocking it, and severely degrades the usability of this provider. Please revert #50 or reduce its scope.

andyvanosdale · 2018-10-30T23:07:41Z

Now that development is starting back up on this, it would be great to have this revisited. I would love to use this in some different projects, but kube2iam used the internal annotations to allow pods to assume an IAM role.

dh-harald · 2018-11-16T10:06:45Z

I'm not a go developer, so I won't open a pull request, but here's my solution for this problem:
dh-harald@d0b48e8

Idea: @pdecat / Solution: @mb290

Here's an example:

provider "kubernetes" {
  internal_annotations_whitelist = [
    "storageclass.kubernetes.io/is-default-class",
  ]
}

resource "kubernetes_storage_class" "gp2" {
  metadata {
    name = "gp2"

    annotations {
      "storageclass.kubernetes.io/is-default-class" = "true"
    }
  }
  storage_provisioner = "kubernetes.io/aws-ebs"
  parameters {
    type      = "gp2"
    encrypted = "true"
  }
}

I don't know is it known, but if you're using true instead of "true", terraform translates it into 1, and it won't work

jakubigla · 2018-12-10T12:51:30Z

Haha, lack of support of internal annotations just made this and couple of other Kubernetes resources useless. Especially in enterprise. Thanks for removing this. Now automation of this just got much complicated.

dh-harald · 2018-12-10T15:12:19Z

@jakubigla For similar reason, we've started using https://github.com/sl1pm4t/terraform-provider-kubernetes instead of the official... It has no problem with internal annotations, there's a lot of beta APIs are supported, which we're using (ingress, clusterrole, etc)... Worth a try instead of using (for example) dumb hacks with null_provider
I though, that the official provider could be usable if we're helping them, but it looks nothing has changed... So I'm very disappointed

holms · 2019-04-08T18:53:56Z

So is there's any progress with this? I mean it's 2019 and April already hello? @pdecat @AlexBoulyga Anyone? Help? Devops can't live without terraform!

Xyrodileas · 2019-04-10T14:00:07Z

And here's a dodgy workaround (partial implementation - it provisions, but not updates or destroys). YMMV.

resource "null_resource" "eks_service_metadata_annotations" {
  provisioner "local-exec" {
    command = <<EOF
kubectl annotate --overwrite --namespace ${local.eks_namespace} \
  service ${kubernetes_service.sample.metadata.0.name} \
  'service.beta.kubernetes.io/aws-load-balancer-backend-protocol=http' \
  'service.beta.kubernetes.io/aws-load-balancer-ssl-cert=${local.acm_certificate_arn}'
EOF
  }
}

Thanks a lot, I would prefer that the official provider support k8s annotation, but ¯_(ツ)_/¯

rpatrick00 · 2019-04-10T16:53:10Z

@Xyrodileas, most of us have gone off the reservation and built our own version of the provider to get around this problem. I created PR #244 for the main provider to address the problem in the way suggested by one of the contributors. Others, like sl1pm4t / terraform-provider-kubernetes, have forked the code since Hashicorp seems uninterested in making this provider capable of real-world deployments. Either way, the only real workaround at this point is to either build your own custom provider or use the local-exec hack to run kubectl.

erick-thompson · 2019-04-16T00:03:03Z

Is this still stalled? I'm trying to use External DNS, and I've run into this issue. It's very annoying. I don't like this idea of terraform making decisions about what I can and can't do in k8s. I've run into other issues like this (not being able to set automountServiceAccountToken on deployments).

Is there a workaround for this particular issue?

tdmalone · 2019-04-16T02:08:37Z

@erick-thompson Workaround is currently to shell out to kubectl using a null_resource and local-exec provisioner (see a few comments up for a simple example). Fix appears to not be coming anytime soon, I think likely not to happen until well after Terraform 0.12 launches.

Regarding automountServiceAccountToken, there is a potential workaround here.

Alternatively there's also a number of forks of the provider, such as this one, which appear to be a good option (I haven't gone down this path just yet). Or of course you could also build a provider locally based on any of the PRs that have been submitted to solve this issue.

erick-thompson · 2019-04-16T17:36:32Z

I ended up using that workaround for the service account token, but I wish I had those hours back, when I was trying to debug what was going on and how to fix it.

For this issue, I'll likely just use terraform to create the yaml file and apply it later. The problem is that I provision the cluster (in my case AKS) from the same terraform as I wanted to configure the service for, so I can't use local-exec unless I do something to ensure that the kubectl config is set up correctly.

tdmalone · 2019-04-17T01:08:44Z

but I wish I had those hours back, when I was trying to debug what was going on and how to fix it.

Yeah, I hear you. It's one thing to make questionable design decisions; it's another thing entirely to not document them!

I can't use local-exec unless I do something to ensure that the kubectl config is set up correctly.

I'm on EKS so I'm not sure what's available for AKS, but I'm using aws eks update-kubeconfig --name "my-cluster-name" && kubectl apply ... in my local-exec. I'm not sure if Azure has a similar tool or automated way to configure kubeconfig.

erick-thompson · 2019-04-19T01:50:05Z

I took a similar approach and used "az aks get-credential". It works well enough, but I find I have to be a lot more explicit about my dependencies.

dayglojesus · 2019-05-05T20:31:46Z

Plan works, apply produces this...

Error: kubernetes_service.ingress_nginx: metadata.0.annotations: "service.beta.kubernetes.io/aws-load-balancer-type" is internal Kubernetes annotation

tdmalone · 2019-05-11T04:53:00Z

This should now be fixed by the merge of #325

selslack · 2019-05-17T17:46:44Z

Can we have a release tagged with the fix included, please?

alexsomesan · 2019-05-17T18:18:47Z

Hi. Yes, a release is coming in the next couple of days. I’m aiming for it to include PodAffinity as well, so I'm looking into getting that ready as well.

On Fri 17. May 2019 at 19:46, Pavel Eremeev ***@***.***> wrote: Can we have a release tagged with the fix included, please? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#60>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIL5G7ACG2BGC4GUM6OMI3PV3VQLANCNFSM4DYLP6HQ> .

-- — Sent from my phone.

tdmalone · 2019-05-23T04:22:23Z

This has now been released in 1.7.0: https://github.com/terraform-providers/terraform-provider-kubernetes/blob/master/CHANGELOG.md#170-may-22-2019
🎉 🚀

bobbytables · 2019-05-23T05:02:09Z

Rock on 🤘

On Thu, May 23, 2019 at 12:22 AM Tim Malone ***@***.***> wrote: This has now been released in 1.7.0: https://github.com/terraform-providers/terraform-provider-kubernetes/blob/master/CHANGELOG.md#170-may-22-2019 🎉 🚀 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#60>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAD7LG3B6DRAAGEAKPQZPTDPWYLYHANCNFSM4DYLP6HQ> .

-- - Robert Ross

andyvanosdale · 2019-05-23T13:19:09Z

Thank you @tdmalone for taking the torch on this provider!

andoneve · 2019-06-12T13:20:35Z

I am still getting an error - maybe my syntax is incorrect?

* provider.kubernetes: version = "~> 1.7"

resource "kubernetes_secret" "my-secret" {
  metadata {
    name = "my-secret"
    annotations {
      kubernetes.io/service-account.name = "my-account"
    }
  }
  type = "kubernetes.io/service-account-token"
}

Error:

Error: Argument or block definition required

  on x.tf line x, in resource "kubernetes_secret" "my-secret":
  x:       kubernetes.io/service-account = "my-account"

An argument or block definition is required here. To set an argument, use the
equals sign "=" to introduce the argument value.

In the error message kubernetes is underlined, as if it parses it as a keyword.

dh-harald · 2019-06-12T13:29:01Z

Can you try with quotes?

resource "kubernetes_secret" "my-secret" {
  metadata {
    name = "my-secret"
    annotations {
      "kubernetes.io/service-account" = "my-account"
    }
  }
  type = "kubernetes.io/service-account-token"
}

andoneve · 2019-06-12T14:06:17Z

I tried before with quotes and I got:

Error: Invalid argument name

  on xx.tf line 17, in resource "kubernetes_secret" "my-token":
  x:       "kubernetes.io/service-account" = "my-account"

Argument names must not be quoted.

pdecat · 2019-06-12T14:08:56Z

With terraform 0.12+, you need to specify annotations as a map and with an equal sign:

resource "kubernetes_secret" "my-secret" {
  metadata {
    name = "my-secret"
    annotations = {
      "kubernetes.io/service-account.name" = "my-account"
    }
  }
  type = "kubernetes.io/service-account-token"
}

andoneve · 2019-06-12T14:12:13Z

With terraform 0.12+, you need to specify annotations as a map and with an equal sign:

resource "kubernetes_secret" "my-secret" {
  metadata {
    name = "my-secret"
    annotations = {
      "kubernetes.io/service-account" = "my-account"
    }
  }
  type = "kubernetes.io/service-account-token"
}

This worked! Thank you. I was losing my mind. I'm new to Terraform - where could I have found this in reference docs? For this specific issue, neither https://www.terraform.io/docs/providers/kubernetes/r/secret.html#annotations nor https://www.terraform.io/docs/configuration/syntax.html were that helpful.

p.s. for posterity's sake, the correct annotation is "kubernetes.io/service-account.name" - just in case someone ends up reading this comment in more detail. Changed in my comment above.

pdecat · 2019-06-12T14:20:56Z

where could I have found this in reference docs? For this specific issue, https://www.terraform.io/docs/providers/kubernetes/r/secret.html#annotations nor https://www.terraform.io/docs/configuration/syntax.html were that helpful.

That specific use case is missing from the docs.
The most extensive source for examples are the acceptance test cases, e.g. https://github.com/terraform-providers/terraform-provider-kubernetes/blob/v1.7.0/kubernetes/resource_kubernetes_secret_test.go#L342
Maybe they could be extracted and put somewhere more easily discoverable.

Edit: I've noticed a few examples were outdated for terraform 0.12, working on a PR.

tdmalone · 2019-06-13T07:02:58Z

This ticket should probably be closed - as the original issue is now resolved :)
Might need a new ticket if the docs aren't right for 0.12 though.

pdecat · 2019-06-13T07:22:39Z

+1 to close this issue.

The docs should be fine now: #501

alexsomesan · 2019-06-13T07:39:39Z

Closed

NZenitram · 2020-02-08T19:57:05Z

This still isn't working for me with an AWS NLB. I've checked the versions as well.

radeksimko added upstream enhancement labels Aug 29, 2017

shadycuz mentioned this issue Sep 1, 2017

Service errors when optional param is not supplied and fails to set annotation #67

Closed

sdavids13 mentioned this issue Jul 3, 2018

Allow storage classes to be designated as the default class #173

Closed

mbarrien mentioned this issue Oct 30, 2018

Revert "meta: Treat internal k8s annotations as invalid" #199

Closed

itmecho mentioned this issue Oct 31, 2018

some annotations are being re-applied on every terraform run sl1pm4t/terraform-provider-kubernetes#72

Closed

pdecat mentioned this issue Nov 6, 2018

meta: Treat internal k8s annotations as invalid #50

Merged

rpatrick00 mentioned this issue Dec 5, 2018

add internal annotation whitelist #244

Closed

tdmalone mentioned this issue Mar 22, 2019

Can not create a persistent volume claim with a storage class annotation #368

Closed

pdecat mentioned this issue Apr 26, 2019

Whitelist "internal" annotations/labels that the user is managing in the resource #325

Merged

pdecat mentioned this issue Jun 12, 2019

Adapt all documentation examples to terraform 0.12 #501

Merged

alexsomesan closed this as completed Jun 13, 2019

ghost locked and limited conversation to collaborators Apr 21, 2020

Can not set internal annotations to specify load balancer ssl certs #60

Can not set internal annotations to specify load balancer ssl certs #60

Comments

udangel-r7 commented Aug 25, 2017

Affected Resource(s)

Terraform Configuration Files

radeksimko commented Sep 1, 2017

fbsb commented Sep 5, 2017

frankthelen commented Oct 5, 2017

jsullivanLI commented Oct 25, 2017

pdecat commented Oct 30, 2017 • edited Loading

bobbytables commented Jan 10, 2018

Phylu commented Feb 26, 2018

bitglue commented Apr 2, 2018

offlinehacker commented Apr 9, 2018

r-moiseev commented Apr 18, 2018

kattelus commented May 14, 2018 • edited Loading

kattelus commented May 16, 2018

bitglue commented May 17, 2018

Phylu commented May 18, 2018

chreichert commented May 25, 2018

TechyMatt commented Jun 12, 2018 • edited Loading

juhan88 commented Oct 26, 2018

mbarrien commented Oct 30, 2018 • edited Loading

andyvanosdale commented Oct 30, 2018

dh-harald commented Nov 16, 2018 • edited Loading

jakubigla commented Dec 10, 2018 • edited Loading

dh-harald commented Dec 10, 2018 • edited Loading

holms commented Apr 8, 2019 • edited Loading

Xyrodileas commented Apr 10, 2019

rpatrick00 commented Apr 10, 2019

erick-thompson commented Apr 16, 2019

tdmalone commented Apr 16, 2019 • edited Loading

erick-thompson commented Apr 16, 2019

tdmalone commented Apr 17, 2019

erick-thompson commented Apr 19, 2019

dayglojesus commented May 5, 2019

tdmalone commented May 11, 2019

selslack commented May 17, 2019

alexsomesan commented May 17, 2019 via email

tdmalone commented May 23, 2019

bobbytables commented May 23, 2019 via email

andyvanosdale commented May 23, 2019 • edited Loading

andoneve commented Jun 12, 2019 • edited Loading

dh-harald commented Jun 12, 2019

andoneve commented Jun 12, 2019

pdecat commented Jun 12, 2019 • edited Loading

andoneve commented Jun 12, 2019 • edited Loading

pdecat commented Jun 12, 2019 • edited Loading

tdmalone commented Jun 13, 2019

pdecat commented Jun 13, 2019

alexsomesan commented Jun 13, 2019

NZenitram commented Feb 8, 2020

pdecat commented Oct 30, 2017 •

edited

Loading

kattelus commented May 14, 2018 •

edited

Loading

TechyMatt commented Jun 12, 2018 •

edited

Loading

mbarrien commented Oct 30, 2018 •

edited

Loading

dh-harald commented Nov 16, 2018 •

edited

Loading

jakubigla commented Dec 10, 2018 •

edited

Loading

dh-harald commented Dec 10, 2018 •

edited

Loading

holms commented Apr 8, 2019 •

edited

Loading

tdmalone commented Apr 16, 2019 •

edited

Loading

andyvanosdale commented May 23, 2019 •

edited

Loading

andoneve commented Jun 12, 2019 •

edited

Loading

pdecat commented Jun 12, 2019 •

edited

Loading

andoneve commented Jun 12, 2019 •

edited

Loading

pdecat commented Jun 12, 2019 •

edited

Loading