Token not being set in provider when trying to upgrade the cluster #1095

gappan · 2020-12-15T17:13:51Z

Terraform Version, Provider Version and Kubernetes Version

Terraform version: 0.14.1
Kubernetes provider version: 1.13.0
Kubernetes version: 1.15

Affected Resource(s)

Terraform Configuration Files

# Provider config 
provider "kubernetes" {
  alias                  = "cluster-0"
  host                   = aws_eks_cluster.cluster-0_cluster.endpoint
  cluster_ca_certificate = base64decode(aws_eks_cluster.cluster-0_cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.cluster-0_cluster-0.token
  load_config_file       = false
}

data "aws_eks_cluster_auth" "cluster-0_cluster-0" {
  name = aws_eks_cluster.cluster-0_cluster.name
}

resource "kubernetes_namespace" "cluster-0_dwh" {
  provider = kubernetes.cluster-0

  metadata {
    name = "dwh"
  }
}

resource "aws_eks_cluster" "cluster-0_cluster" {
  enabled_cluster_log_types = ["api", "controllerManager", "scheduler"]
  name                      = "${local.prefix}-cluster-0"
  role_arn                  = aws_iam_role.cluster-0_cluster_role.arn
  tags                      = local.default_tags

  version = "1.16"

  vpc_config {
    security_group_ids      = [aws_security_group.cluster-0_control_plane.id]
    subnet_ids              = aws_subnet.cluster-0_private.*.id
    endpoint_private_access = true
    endpoint_public_access  = false 
  }
  depends_on = [
    aws_iam_role_policy_attachment.cluster-0_cluster_role-AmazonEKSClusterPolicy,
    aws_iam_role_policy_attachment.cluster-0_cluster_role-AmazonEKSServicePolicy,
  ]
}


Just for testing purpose I ran the plan with a placeholder token as well, output is below 

provider "kubernetes" {
  alias                  = "cluster-0"
  host                   = aws_eks_cluster.cluster-0_cluster.endpoint
  cluster_ca_certificate = base64decode(aws_eks_cluster.cluster-0_cluster.certificate_authority.0.data)
  token                  = "MYAWESOMETOKEN"
  load_config_file       = false
}

Debug Output

Plan outputs of both cases.
As you can see the Authorization Header with Bearer token is not being set in the request when the auth token is set by the data.aws_eks_cluster_auth.cluster-0_cluster-0.token , and vice versa when i ran the plan hardcoding a placeholder token.

2020-12-15T16:36:37.334Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: 2020/12/15 16:36:37 [INFO] Checking namespace dwh
2020-12-15T16:36:37.335Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: 2020/12/15 16:36:37 [DEBUG] Kubernetes API Request Details:
2020-12-15T16:36:37.335Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: ---[ REQUEST ]---------------------------------------
2020-12-15T16:36:37.335Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: GET /api/v1/namespaces/dwh HTTP/1.1
2020-12-15T16:36:37.335Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: Host: XXXXXXXXXXXXX.yl4.ap-northeast-2.eks.amazonaws.com
2020-12-15T16:36:37.335Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: User-Agent: HashiCorp/1.0 Terraform/0.14.1
2020-12-15T16:36:37.335Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: Accept: application/json, */*
2020-12-15T16:36:37.335Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: Accept-Encoding: gzip
2020-12-15T16:36:37.335Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: 
2020-12-15T16:36:37.335Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: 
2020-12-15T16:36:37.335Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: -----------------------------------------------------
2020-12-15T16:36:38.374Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: 2020/12/15 16:36:38 [DEBUG] Kubernetes API Response Details:
2020-12-15T16:36:38.374Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: ---[ RESPONSE ]--------------------------------------
2020-12-15T16:36:38.374Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: HTTP/2.0 403 Forbidden
2020-12-15T16:36:38.374Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: Content-Length: 295
2020-12-15T16:36:38.374Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: Audit-Id: 86a58235-11ef-477b-b7ec-603d0c195f42
2020-12-15T16:36:38.374Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: Content-Type: application/json
2020-12-15T16:36:38.374Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: Date: Tue, 15 Dec 2020 16:36:38 GMT
2020-12-15T16:36:38.374Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: X-Content-Type-Options: nosniff
2020-12-15T16:36:38.374Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: 
2020-12-15T16:36:38.374Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: {
2020-12-15T16:36:38.374Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4:  "kind": "Status",
2020-12-15T16:36:38.374Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4:  "apiVersion": "v1",
2020-12-15T16:36:38.374Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4:  "metadata": {},
2020-12-15T16:36:38.374Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4:  "status": "Failure",
2020-12-15T16:36:38.374Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4:  "message": "namespaces \"dwh\" is forbidden: User \"system:anonymous\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"dwh\"",
2020-12-15T16:36:38.374Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4:  "reason": "Forbidden",
2020-12-15T16:36:38.374Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4:  "details": {
2020-12-15T16:36:38.374Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4:   "name": "dwh",
2020-12-15T16:36:38.374Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4:   "kind": "namespaces"
2020-12-15T16:36:38.374Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4:  },
2020-12-15T16:36:38.375Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4:  "code": 403
2020-12-15T16:36:38.375Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: }



Plan when ran with hardcodding a placeholder token 

2020-12-15T18:55:03.091Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: 2020/12/15 18:55:03 [INFO] Checking namespace dwh
2020-12-15T18:55:03.091Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: 2020/12/15 18:55:03 [DEBUG] Kubernetes API Request Details:
2020-12-15T18:55:03.091Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: ---[ REQUEST ]---------------------------------------
2020-12-15T18:55:03.091Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: GET /api/v1/namespaces/dwh HTTP/1.1
2020-12-15T18:55:03.091Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: Host: XXXXXXXXXXXXXXXXXX.yl4.ap-northeast-2.eks.amazonaws.com
2020-12-15T18:55:03.091Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: User-Agent: HashiCorp/1.0 Terraform/0.14.1
2020-12-15T18:55:03.091Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: Accept: application/json, */*
2020-12-15T18:55:03.091Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: Authorization: Bearer MYAWESOMETOKEN
2020-12-15T18:55:03.091Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: Accept-Encoding: gzip
2020-12-15T18:55:03.091Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: 
2020-12-15T18:55:03.091Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: 
2020-12-15T18:55:03.091Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: -----------------------------------------------------
2020-12-15T18:55:04.099Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: 2020/12/15 18:55:04 [DEBUG] Kubernetes API Response Details:
2020-12-15T18:55:04.099Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: ---[ RESPONSE ]--------------------------------------
2020-12-15T18:55:04.099Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: HTTP/2.0 401 Unauthorized
2020-12-15T18:55:04.099Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: Content-Length: 129
2020-12-15T18:55:04.099Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: Audit-Id: edf94a31-5bb5-4e53-9dcf-de60c3a6911f
2020-12-15T18:55:04.099Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: Content-Type: application/json
2020-12-15T18:55:04.099Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: Date: Tue, 15 Dec 2020 18:55:03 GMT
2020-12-15T18:55:04.099Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: 
2020-12-15T18:55:04.099Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: {
2020-12-15T18:55:04.099Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4:  "kind": "Status",
2020-12-15T18:55:04.099Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4:  "apiVersion": "v1",
2020-12-15T18:55:04.099Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4:  "metadata": {},
2020-12-15T18:55:04.099Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4:  "status": "Failure",
2020-12-15T18:55:04.099Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4:  "message": "Unauthorized",
2020-12-15T18:55:04.099Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4:  "reason": "Unauthorized",
2020-12-15T18:55:04.099Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4:  "code": 401
2020-12-15T18:55:04.099Z [DEBUG] plugin.terraform-provider-kubernetes_v1.13.0_x4: }

Steps to Reproduce

Expected Behavior

We hit this issue when we were trying to upgrade the eks cluster, the provider fails to get the instantiate with the auth token and hence the plan fails. These resources should not be affected during the plan.

Actual Behavior

Terraform plan fails as kubernetes provider fails to authenticate to the cluster.

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

The text was updated successfully, but these errors were encountered:

aareet · 2021-01-06T17:17:33Z

@gappan are you using the default token that's coming from the EKS resource? It may be expiring before terraform makes the request

gappan · 2021-01-20T14:39:29Z

@aareet sorry for the late response, it is not about the token expiring, if you look into the logs, the token itself is not getting set in the first place when the api call is made . Also when I am not trying to upgrade the cluster it works as expected.

dak1n1 · 2021-02-10T00:22:11Z

I'm curious about the cluster upgrade -- is this replacing the underlying EKS cluster? Or replacing the cluster's authentication credentials (host, certs, token)? If so, the Kubernetes provider will be initialized before these credentials exist. I think the initialization order can even cause errors like this where the token is omitted completely from the API call, (though I'd have to test that to know for sure).

There is a known limitation with using a single apply to configure a cluster with Kubernetes resources. I wrote this short example guide that will walk you through updating or replacing an EKS cluster as needed. It also contains example code which uses the exec block to fetch a fresh token during each apply, which could help mitigate issues with the data source token expiring.

Here's the exec block example:

provider "kubernetes" {
  host                   = var.cluster_endpoint
  cluster_ca_certificate = base64decode(var.cluster_ca_cert)
  exec {
    api_version = "client.authentication.k8s.io/v1alpha1"
    args        = ["eks", "get-token", "--cluster-name", var.cluster_name]
    command     = "aws"
  }
}

EDIT: I forgot to mention one other option: if you apply the EKS changes in a separate apply from the Kubernetes resources, it should work every time, and then you won't have to worry about expired credentials and work-arounds. terraform apply -target=module.eks.

github-actions · 2022-04-28T00:00:57Z

Marking this issue as stale due to inactivity. If this issue receives no comments in the next 30 days it will automatically be closed. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. This helps our maintainers find and focus on the active issues. Maintainers may also remove the stale label at their discretion. Thank you!

github-actions · 2022-05-29T02:36:28Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

gappan added the bug label Dec 15, 2020

aareet added the waiting-response label Jan 6, 2021

ghost removed the waiting-response label Jan 20, 2021

dak1n1 mentioned this issue Feb 10, 2021

Removing state of dependent providers (partial fix for progressive apply) hashicorp/terraform#27728

Closed

Nuru mentioned this issue Feb 10, 2021

Fail with I/O timeout due to bad configuration of the Kubernetes provider cloudposse/terraform-aws-eks-cluster#104

Closed

Nuru mentioned this issue Mar 17, 2021

Set hashicorp/kubernetes lower bound to 1.0 cloudposse/terraform-aws-eks-cluster#109

Merged

dak1n1 added the theme/auth label Apr 27, 2021

dak1n1 mentioned this issue Jul 15, 2021

A way to refresh provider credentials hashicorp/terraform#29182

Open

github-actions bot added the stale label Apr 28, 2022

alexsomesan closed this as completed Apr 28, 2022

github-actions bot locked as resolved and limited conversation to collaborators May 29, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Token not being set in provider when trying to upgrade the cluster #1095

Token not being set in provider when trying to upgrade the cluster #1095

gappan commented Dec 15, 2020 •

edited

Loading

aareet commented Jan 6, 2021

gappan commented Jan 20, 2021

dak1n1 commented Feb 10, 2021 •

edited

Loading

github-actions bot commented Apr 28, 2022

github-actions bot commented May 29, 2022

Token not being set in provider when trying to upgrade the cluster #1095

Token not being set in provider when trying to upgrade the cluster #1095

Comments

gappan commented Dec 15, 2020 • edited Loading

Terraform Version, Provider Version and Kubernetes Version

Affected Resource(s)

Terraform Configuration Files

Debug Output

Steps to Reproduce

Expected Behavior

Actual Behavior

Community Note

aareet commented Jan 6, 2021

gappan commented Jan 20, 2021

dak1n1 commented Feb 10, 2021 • edited Loading

github-actions bot commented Apr 28, 2022

github-actions bot commented May 29, 2022

gappan commented Dec 15, 2020 •

edited

Loading

dak1n1 commented Feb 10, 2021 •

edited

Loading