Kubernetes provider 2.6.0 breaks EKS creation #1658

jabbrwcky · 2021-10-25T07:58:53Z

Description

I ran into the issue that Terraform fails to apply the calculated plan to set up EKS and import it into Rancher on a new project without existing state.

Plan fails with an error where the kubernetes provider (released on 2021-10-19) complains about an nonexisting context.

After some experimenting it turns out that the issue can be avoided by using kubernetes provider in Vaersion <2.6.0.

I am not sure whether it is an issue of the EKS module or in kubernetes provider.

Versions

Terraform: 1.0.9
Provider(s):

.
├── provider[registry.terraform.io/gavinbunney/kubectl] ~> 1.7
├── provider[registry.terraform.io/hashicorp/aws] ~> 3.15
├── provider[registry.terraform.io/rancher/rancher2] < 1.20.0
├── provider[registry.terraform.io/gitlabhq/gitlab] ~> 3.1
├── provider[registry.terraform.io/hashicorp/helm] ~> 2.2
├── provider[registry.terraform.io/hashicorp/random] ~> 3.0
├── provider[registry.terraform.io/hashicorp/kubernetes] ~> 2.0
├── module.eks
│   ├── provider[registry.terraform.io/hashicorp/aws] >= 3.44.0
│   ├── provider[registry.terraform.io/rancher/rancher2] >= 1.12.0
│   ├── provider[registry.terraform.io/hashicorp/kubernetes]
│   ├── module.vpc
│   │   └── provider[registry.terraform.io/hashicorp/aws] >= 3.63.0
│   └── module.eks
│       ├── provider[registry.terraform.io/hashicorp/cloudinit] >= 2.0.0
│       ├── provider[registry.terraform.io/terraform-aws-modules/http] >= 2.4.1
│       ├── provider[registry.terraform.io/hashicorp/aws] >= 3.56.0
│       ├── provider[registry.terraform.io/hashicorp/local] >= 1.4.0
│       ├── provider[registry.terraform.io/hashicorp/kubernetes] >= 1.11.1
│       ├── module.fargate
│           └── provider[registry.terraform.io/hashicorp/aws] >= 3.40.0
│       └── module.node_groups
│           ├── provider[registry.terraform.io/hashicorp/aws] >= 3.56.0
│           └── provider[registry.terraform.io/hashicorp/cloudinit] >= 2.0.0
└── module.subnets

Reproduction

Steps to reproduce the behavior:

Createan EKS cluster with enabled creation of aws-auth using the latest versions of the EKS module and kubernetes provider.

Code Snippet to Reproduce

module "eks" {
  source                                = "terraform-aws-modules/eks/aws"
  version                               = "~> 17.0"
  cluster_name                          = var.cluster_name
  cluster_version                       = var.cluster_version
  subnets                               = module.vpc.private_subnets
  vpc_id                                = module.vpc.vpc_id
  enable_irsa                           = true
  write_kubeconfig                      = true
  kubeconfig_output_path                = ".kube/config"
  cluster_enabled_log_types             = var.cluster_enabled_log_types
  cluster_log_retention_in_days         = var.cluster_log_retention
  manage_aws_auth                       = true
  worker_create_initial_lifecycle_hooks = true
  map_roles = ...
  node_groups = ...
}

(Omitted in the snippet is the map_roles and node_groups parameter values)

Expected behavior

Actual behavior

Terminal Output Screenshot(s)

 tg plan -out tf.plan
╷
│ Error: Provider configuration: cannot load Kubernetes client config
│
│   with module.eks.provider["registry.terraform.io/hashicorp/kubernetes"],
│   on .terraform/modules/eks/main.tf line 69, in provider "kubernetes":
│   69: provider "kubernetes" {
│
│ context "eks" does not exist
╵
╷
│ Error: error reading EKS Cluster (eks): couldn't find resource
│
│   with module.eks.module.eks.module.node_groups.data.aws_eks_cluster.default[0],
│   on .terraform/modules/eks.eks/modules/node_groups/locals.tf line 1, in data "aws_eks_cluster" "default":
│    1: data "aws_eks_cluster" "default" {
│
╵
ERRO[0018] 1 error occurred:
	* exit status 1

Additional context

Fixing kubernetes provider to <2.6.0 fixes the issue:

Providers required by configuration:
.
├── provider[registry.terraform.io/gavinbunney/kubectl] ~> 1.7
├── provider[registry.terraform.io/hashicorp/aws] ~> 3.15
├── provider[registry.terraform.io/rancher/rancher2] < 1.20.0
├── provider[registry.terraform.io/gitlabhq/gitlab] ~> 3.1
├── provider[registry.terraform.io/hashicorp/helm] ~> 2.2
├── provider[registry.terraform.io/hashicorp/random] ~> 3.0
├── provider[registry.terraform.io/hashicorp/kubernetes] ~> 2.0, < 2.6.0
├── module.eks
│   ├── provider[registry.terraform.io/hashicorp/aws] >= 3.44.0
│   ├── provider[registry.terraform.io/rancher/rancher2] >= 1.12.0
│   ├── provider[registry.terraform.io/hashicorp/kubernetes]
│   ├── module.vpc
│   │   └── provider[registry.terraform.io/hashicorp/aws] >= 3.63.0
│   └── module.eks
│       ├── provider[registry.terraform.io/hashicorp/cloudinit] >= 2.0.0
│       ├── provider[registry.terraform.io/terraform-aws-modules/http] >= 2.4.1
│       ├── provider[registry.terraform.io/hashicorp/aws] >= 3.56.0
│       ├── provider[registry.terraform.io/hashicorp/local] >= 1.4.0
│       ├── provider[registry.terraform.io/hashicorp/kubernetes] >= 1.11.1
│       ├── module.fargate
│           └── provider[registry.terraform.io/hashicorp/aws] >= 3.40.0
│       └── module.node_groups
│           ├── provider[registry.terraform.io/hashicorp/aws] >= 3.56.0
│           └── provider[registry.terraform.io/hashicorp/cloudinit] >= 2.0.0
└── module.subnets

The text was updated successfully, but these errors were encountered:

daroga0002 · 2021-10-25T08:25:37Z

could you paste terraform version for problematic run?

daroga0002 · 2021-10-25T08:41:28Z

seems be related to hashicorp/terraform-provider-kubernetes#1469

daroga0002 · 2021-10-25T08:50:39Z

and can be potentially related to hashicorp/terraform-provider-kubernetes#1467

daroga0002 · 2021-10-25T08:51:37Z

Looks they released 2.6.1 (there is just a tag without github release) so you can try it

jabbrwcky · 2021-10-25T10:01:25Z

Issue unfornuately still persists with 2.6.1, but it already fails on plan:

╷
│ Error: Provider configuration: cannot load Kubernetes client config
│
│   with module.eks.provider["registry.terraform.io/hashicorp/kubernetes"],
│   on .terraform/modules/eks/main.tf line 69, in provider "kubernetes":
│   69: provider "kubernetes" {
│
│ context "eks" does not exist
╵
ERRO[0017] 1 error occurred:
	* exit status 1

terraform version
Terraform v1.0.9
on darwin_amd64
+ provider registry.terraform.io/gavinbunney/kubectl v1.13.0
+ provider registry.terraform.io/gitlabhq/gitlab v3.7.0
+ provider registry.terraform.io/hashicorp/aws v3.63.0
+ provider registry.terraform.io/hashicorp/cloudinit v2.2.0
+ provider registry.terraform.io/hashicorp/helm v2.3.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.6.1
+ provider registry.terraform.io/hashicorp/local v2.1.0
+ provider registry.terraform.io/hashicorp/random v3.1.0
+ provider registry.terraform.io/rancher/rancher2 v1.17.2
+ provider registry.terraform.io/terraform-aws-modules/http v2.4.1

daroga0002 · 2021-10-25T10:09:46Z

please paste how you initialize kubernetes provider in terraform and check does you have in kube config context context "eks" does not exist

jabbrwcky · 2021-10-25T15:31:46Z

Apart from the module invocation I use the code from the Module documentation:

data "aws_eks_cluster" "eks" {
  name = module.eks.cluster_id
}

data "aws_eks_cluster_auth" "eks" {
  name = module.eks.cluster_id
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.eks.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.eks.token
}

module "eks" {
 ...
}

daroga0002 · 2021-11-02T10:45:03Z

So I tested this and using following providers:

Initializing provider plugins...
- Reusing previous version of hashicorp/kubernetes from the dependency lock file
- Reusing previous version of hashicorp/cloudinit from the dependency lock file
- Reusing previous version of terraform-aws-modules/http from the dependency lock file
- Reusing previous version of hashicorp/aws from the dependency lock file
- Reusing previous version of hashicorp/local from the dependency lock file
- Reusing previous version of hashicorp/random from the dependency lock file
- Using previously-installed terraform-aws-modules/http v2.4.1
- Using previously-installed hashicorp/aws v3.63.0
- Using previously-installed hashicorp/local v2.1.0
- Using previously-installed hashicorp/random v3.1.0
- Using previously-installed hashicorp/kubernetes v2.6.1
- Using previously-installed hashicorp/cloudinit v2.2.0

on our example https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/examples/launch_templates_with_managed_node_groups and no issue observed

where kubernetes provider is initialized:

terraform-aws-eks/examples/launch_templates_with_managed_node_groups/main.tf

Lines 97 to 101 in 967403b

    
           provider "kubernetes" { 
        
             host                   = data.aws_eks_cluster.cluster.endpoint 
        
             cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data) 
        
             token                  = data.aws_eks_cluster_auth.cluster.token 
        
           }

jabbrwcky · 2021-11-19T08:42:12Z

Just encountered the same issue again with a different cluster. A downgrade from 2.6.1 to 2.5.1 resolved the issue

github-actions · 2021-12-20T00:43:06Z

This issue has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this issue will be closed in 10 days

github-actions · 2021-12-31T00:43:49Z

This issue was automatically closed because of stale in 10 days

antonbabenko · 2022-01-05T19:59:02Z

This issue has been resolved in version 18.0.0 🎉

github-actions · 2022-11-16T02:29:20Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

daroga0002 mentioned this issue Oct 27, 2021

terraform AWS EKS 'destroy' failed module.eks.kubernetes_config_map.aws_auth[0] Error: Unauthorized #1661

Closed

bryantbiggs mentioned this issue Dec 7, 2021

feat!: Removed support for launch configuration and replace count with for_each #1680

Merged

1 task

github-actions bot added the stale label Dec 20, 2021

github-actions bot closed this as completed Dec 31, 2021

github-actions bot locked as resolved and limited conversation to collaborators Nov 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kubernetes provider 2.6.0 breaks EKS creation #1658

Kubernetes provider 2.6.0 breaks EKS creation #1658

jabbrwcky commented Oct 25, 2021

daroga0002 commented Oct 25, 2021

daroga0002 commented Oct 25, 2021

daroga0002 commented Oct 25, 2021 •

edited

Loading

daroga0002 commented Oct 25, 2021

jabbrwcky commented Oct 25, 2021

daroga0002 commented Oct 25, 2021 •

edited

Loading

jabbrwcky commented Oct 25, 2021

daroga0002 commented Nov 2, 2021

jabbrwcky commented Nov 19, 2021

github-actions bot commented Dec 20, 2021

github-actions bot commented Dec 31, 2021

antonbabenko commented Jan 5, 2022

github-actions bot commented Nov 16, 2022

Kubernetes provider 2.6.0 breaks EKS creation #1658

Kubernetes provider 2.6.0 breaks EKS creation #1658

Comments

jabbrwcky commented Oct 25, 2021

Description

Versions

Reproduction

Code Snippet to Reproduce

Expected behavior

Actual behavior

Terminal Output Screenshot(s)

Additional context

daroga0002 commented Oct 25, 2021

daroga0002 commented Oct 25, 2021

daroga0002 commented Oct 25, 2021 • edited Loading

daroga0002 commented Oct 25, 2021

jabbrwcky commented Oct 25, 2021

daroga0002 commented Oct 25, 2021 • edited Loading

jabbrwcky commented Oct 25, 2021

daroga0002 commented Nov 2, 2021

jabbrwcky commented Nov 19, 2021

github-actions bot commented Dec 20, 2021

github-actions bot commented Dec 31, 2021

antonbabenko commented Jan 5, 2022

github-actions bot commented Nov 16, 2022

daroga0002 commented Oct 25, 2021 •

edited

Loading

daroga0002 commented Oct 25, 2021 •

edited

Loading