Unauthorized on resource deletion in EKS

[nikitazernov]

Terraform Version, Provider Version and Kubernetes Version

Terraform version: 0.14.4
Kubernetes provider version: 1.13.3
Kubernetes version: 1.18

Affected Resource(s)

All resources

Terraform Configuration Files

resource "kubernetes_service" "admin-panel" {
  metadata {
    name = "admin-panel"
    namespace = kubernetes_namespace.platform.id
  }
  spec {
    selector = {
      app = "admin-panel"
    }
    port {
      protocol = "TCP"
      port = 80
    }
  }
}

resource "kubernetes_ingress" "admin-panel" {
  depends_on = [helm_release.nginx-ingress, aws_route53_record.nginx, helm_release.cert-manager]

  metadata {
    name = "admin-panel-ingress"
    namespace = kubernetes_namespace.platform.id
    annotations = {
      "kubernetes.io/ingress.class" = "nginx"
      "acme.cert-manager.io/http01-edit-in-place" = "true"
      "cert-manager.io/cluster-issuer" = "letsencrypt"
      "cert-manager.io/issue-temporary-certificate" = "true"
      "nginx.ingress.kubernetes.io/configuration-snippet" = <<-EOT
        proxy_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;
        grpc_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;
      EOT
    }
  }
  spec {
    rule {
      host = "panel.adm.${var.admin_zone}"
      http {
        path {
          backend {
            service_name = "admin-panel"
            service_port = 80
          }
        }
      }
    }
  }
}

Debug Output

https://gist.github.com/nikitazernov/282139c480c47f2a7df47ed846b1f774

Panic Output

Steps to Reproduce

1. terraform apply
2. terraform destroy

Expected Behaviour

1. On apply all resources created successfully
2. On destroy all resources deleted successfully

Actual Behaviour

1. On apply all resources created successfully
2. On destroy all resources are failed to delete because unauthorized

Important Factoids

Cluster is running on AWS EKS. Resource deletion using kubectl with the same AWS credentials is successful.
Here are snippets from EKS logs:
https://gist.github.com/nikitazernov/6db450ac4b10d2779c571fe0281c1852
The user is empty on deletion, but persists in all other requests.

References

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[thpang]

@nikitazernov what did you do to fix this issue? I am seeing this as well. No explanation here on what was fixed.

[nikitazernov]

@thpang hello! Later I've figured out this issue appeared after update to terraform 0.14. After downgrading Terraform to 0.13.5 it worked.

[thpang]

Yes I have come to the same conclusion, that moving back to tf 0.13.6 works, but that tf 0.14.4 does not. Have not turned on tf debugging to capture more. Hopefully someone from Hashicorp will pick this up.

[TsvetanMilanov]

The problem seems to be in terraform, not in the kubernetes provider - hashicorp/terraform#27172
The fix is merged but terraform 0.14.5 still reproduces the issue. I guess it is not released yet. However, the workaround with manually refreshing the state before destroying, worked for me.

[eaterm]

Have the same issue. Running refresh before destroy seemed to work.
But looking at the output of the log data sources are still not refreshed.

Looks like aws_eks_cluster_auth is not refreshing. During destroy (even with a refresh before) and plan/apply.
Could this be the issue?

[dak1n1]

This is unfortunately a common problem when EKS credentials expire and the Kubernetes provider attempts to initialize using the outdated credentials. See this comment for more details and a work-around. #1131 (comment)

[derjust]

The fix is merged but terraform 0.14.5 still reproduces the issue. I guess it is not released yet. However, the workaround with manually refreshing the state before destroying, worked for me.

The fix is announced only in the changelog of the (upcoming) v0.15.0 release

[dak1n1]

Closing since it's fixed upstream.

[thpang]

Just as a side note, given this issue was not fixed at all in v0.14.x this is going to cause folks that need this functionality to completely skip the v0.14.x of terraform for their work. I know we will ;)

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!