ValidationError: gp3 is invalid #1205

ichasco-heytrade · 2021-01-29T08:36:42Z

I have issues

GP3 is unsupported in launch configuration

I'm submitting a...

[ x] bug report
feature request
support request - read the FAQ first!
kudos, thank you, warm fuzzy

What is the current behavior?

The cluster can't be created with gp3 launch configuration

If this is a bug, how to reproduce? Please include a code sample if relevant.

module "eks" {
  source          = "terraform-aws-modules/eks/aws"
  cluster_name    = var.stage
  cluster_version = "1.18"
  subnets         = module.vpc.private_subnets
  vpc_id          = module.vpc.vpc_id

  cluster_endpoint_private_access = true
  cluster_endpoint_public_access  = true

  worker_additional_security_group_ids  = [aws_security_group.bastion-sg.id]
  worker_ami_name_filter                = "amazon-eks-node-1.18-v20201211"
  manage_worker_iam_resources           = true
  enable_irsa                           = true

  map_roles = [
    {
      rolearn = module.eks_devops_user-role.role-arn
      username = "devops"
      groups = ["system:masters"]
    },
  ]

  cluster_encryption_config = [
    {
      provider_key_arn = aws_kms_key.eks_secrets.arn
      resources        = ["secrets"]
    }
  ]

  worker_groups = [
    {
      name                  = "spot-1"
      spot_price            = "0.50"
      instance_type         = "m4.large"
      asg_desired_capacity  = 1
      asg_max_size          = 3
      asg_min_size          = 0
      pre_userdata          = "hostnamectl set-hostname $( cat /etc/hostname ).${var.region}.compute.internal"
      kubelet_extra_args    = "--node-labels=node.kubernetes.io/lifecycle=spot,stage=${var.stage}"
      enable_monitoring     = false
      public_ip             = false
      suspended_processes   = ["AZRebalance"]
      tags = [
        {
          "key"                 = "Stage"
          "value"               = var.stage
          "propagate_at_launch" = true
        },
        {
          "key"                 = "k8s.io/cluster-autoscaler/node-template/label/kubernetes.io/lifecycle"
          "value"               = "spot"
          "propagate_at_launch" = false
        },
        {
          "key"                 = "k8s.io/cluster-autoscaler/enabled"
          "value"               = "true"
          "propagate_at_launch" = false
        },
        {
          "key"                 = "k8s.io/cluster-autoscaler/${var.stage}"
          "propagate_at_launch" = "false"
          "value"               = "true"
        }
      ]
    }
  ]
  tags = {
    Stage = var.stage
  }
}

Environment details

EKS module version: 14.0.0
AWS module version: v3.26.0
Terraform version: v0.14.5

Any other relevant info

Error:

Error: Error creating launch configuration: ValidationError: gp3 is invalid. Valid volume types are standard, io1, gp2, st1 and sc1.
        status code: 400, request id: 9d6826bb-a513-4647-9169-c1b49e2dbbd7

Change:

 ~ root_block_device {
          ~ volume_type           = "gp2" -> "gp3" # forces replacement
            # (4 unchanged attributes hidden)
        }

The text was updated successfully, but these errors were encountered:

oscr · 2021-01-29T08:58:12Z

@ichasco-heytrade What region are you using? Since it's a new volume type it might be that it's not available yet.

ichasco-heytrade · 2021-01-29T09:05:10Z

Hi @oscr

The region is eu-west-1

It suppose that every regions support it
https://aws.amazon.com/es/blogs/aws/new-amazon-ebs-gp3-volume-lets-you-provision-performance-separate-from-capacity-and-offers-20-lower-price/

Available Today
The gp3 volume type is available for all AWS Regions. You can access the AWS Management Console to launch your first gp3 volume.

Thanks!! :)

oscr · 2021-01-29T09:08:40Z

@ichasco-heytrade I just got the same error message in the same region. Even if you can create that volume type via the console.

barryib · 2021-01-29T09:21:50Z

It sounds like we have to update the aws provider version https://github.com/hashicorp/terraform-provider-aws/blob/main/CHANGELOG.md#3250-january-22-2021.

Can you please change the required provider https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/versions.tf#L5 to 3.25.0 ?

oscr · 2021-01-29T09:22:38Z

@barryib I'm testing it right now.

ichasco-heytrade · 2021-01-29T09:38:00Z

I have tested with 3.26.0 provider version. - Installing hashicorp/aws v3.26.0... - Installed hashicorp/aws v3.26.0 (signed by HashiCorp)

GerardoGR · 2021-01-29T09:41:28Z

For the time being, setting the workers_group_defaults.root_volume_type to gp2 seems to work:

  workers_group_defaults = {
  	root_volume_type = "gp2"
  }

oscr · 2021-01-29T10:27:23Z

This might be an issue with aws launch configuration validation:

https://docs.aws.amazon.com/cli/latest/reference/autoscaling/create-launch-configuration.html

VolumeType -> (string)

The volume type, which can be standard for Magnetic, io1 for Provisioned IOPS SSD, gp2 for General Purpose SSD, st1 for Throughput Optimized HDD, or sc1 for Cold HDD. For more information, see Amazon EBS Volume Types in the Amazon EC2 User Guide for Linux Instances .

Valid Values: standard | io1 | gp2 | st1 | sc1

marcuz · 2021-01-29T11:46:22Z

I have tested with 3.26.0 provider version.

Installing hashicorp/aws v3.26.0...

Installed hashicorp/aws v3.26.0 (signed by HashiCorp)

@ichasco-heytrade what are your results ?

I have tested with AWS provider 3.26.0 too and getting the same error:

Error: Error creating launch configuration: ValidationError: gp3 is invalid. Valid volume types are standard, io1, gp2, st1 and sc1.
	status code: 400, request id: bc4139d6-d9ce-41fc-8998-1d902bdd3e9d

barryib · 2021-01-29T11:59:53Z

It sounds like the aws api doesn't support gp3 for launch configuration https://docs.aws.amazon.com/autoscaling/ec2/APIReference/API_Ebs.html.

Does anyone know if this will change in a near future ?

So for now, launch configuration users must set they root_device_type to gp2 as @GerardoGR suggested. We can update the FAQ with this.

mzupan · 2021-01-29T13:15:29Z

gp3 isn't in launch configs yet. If you switch to launch templates gp3 should work. The default for launch configs for this module should be reverted back to gp2 though

edit: I believe AWS is trying to push people to templates over configs. I'd honestly just use templates as I wouldn't expect them to be too quick to add support to configs

barryib · 2021-01-29T13:58:47Z

gp3 isn't in launch configs yet. If you switch to launch templates gp3 should work. The default for launch configs for this module should be reverted back to gp2 though

@mzupan there is no default config for LC only. So if we rollback, we will rollback for both LC and LT. FWIW, we plan to drop launch configuration soon to let peoples move to launch template.

margamanterola · 2021-01-29T15:22:53Z

This change breaks launching clusters with the default config shown in the basic example. If launch config is no longer supported, then it should be removed from all docs. Having a default value that breaks the default use of the module seems like a recipe for frustrated users.

barryib · 2021-01-29T16:12:46Z

Just opened #1208.

sharjeelsayed · 2021-01-31T07:43:14Z

For the time being, setting the workers_group_defaults.root_volume_type to gp2 seems to work:
  workers_group_defaults = {
  	root_volume_type = "gp2"
  }

Where do you place this code?

ichasco-heytrade · 2021-01-31T08:46:31Z

Hi @sharjeelsayed, you have to place it in the module:

module "eks" {
  source          = "terraform-aws-modules/eks/aws"
...

  workers_group_defaults = {
  	root_volume_type = "gp2"
  }
...
}

sharjeelsayed · 2021-01-31T08:49:56Z

Hi @sharjeelsayed, you have to place it in the module:

module "eks" {
  source          = "terraform-aws-modules/eks/aws"
...

  workers_group_defaults = {
  	root_volume_type = "gp2"
  }
...
}

What if I am using the eks_example_basic module?

module "eks_example_basic" {
  source  = "terraform-aws-modules/eks/aws//examples/basic"
  version = "14.0.0"
}

[#176721566](https://www.pivotaltracker.com/story/show/176721566) See terraform-aws-modules/terraform-aws-eks#1205 Co-authored-by: James Pollard <[email protected]>

shahafy-ma · 2021-02-10T17:35:24Z

Hey guys

I am still getting this one

      ~ root_block_device {
            delete_on_termination = true
          ~ encrypted             = false -> (known after apply)
            iops                  = 0
            volume_size           = 100
          ~ volume_type           = "gp2" -> "gp3" # forces replacement
        }
    }
...

Error: Error creating launch configuration: ValidationError: gp3 is invalid. Valid volume types are standard, io1, gp2, st1 and sc1.
        status code: 400, request id: 2cf54f3d-de26-47be-a989-e92af332ed17

Is there any solution? I am using
provider registry.terraform.io/hashicorp/aws v3.27.0

Saeger · 2021-02-10T17:52:17Z

Hey guys

I am still getting this one

      ~ root_block_device {
            delete_on_termination = true
          ~ encrypted             = false -> (known after apply)
            iops                  = 0
            volume_size           = 100
          ~ volume_type           = "gp2" -> "gp3" # forces replacement
        }
    }
...

Error: Error creating launch configuration: ValidationError: gp3 is invalid. Valid volume types are standard, io1, gp2, st1 and sc1.
        status code: 400, request id: 2cf54f3d-de26-47be-a989-e92af332ed17

Is there any solution? I am using
provider registry.terraform.io/hashicorp/aws v3.27.0

I would recommend you for Switching from aws_launch_configuration to aws_launch_template as it already have support for gp3 instances.

albertoal · 2021-02-10T18:39:04Z

Also still getting this on aws provider 3.27.0 and us-east-1. The workaround suggested is working but it will be good to fix the example config so users don't get this error when using it for the first time. Thanks!

kaushikarvind · 2021-02-13T22:00:07Z

Hey guys
I am still getting this one
      ~ root_block_device {
            delete_on_termination = true
          ~ encrypted             = false -> (known after apply)
            iops                  = 0
            volume_size           = 100
          ~ volume_type           = "gp2" -> "gp3" # forces replacement
        }
    }
...

Error: Error creating launch configuration: ValidationError: gp3 is invalid. Valid volume types are standard, io1, gp2, st1 and sc1.
        status code: 400, request id: 2cf54f3d-de26-47be-a989-e92af332ed17
Is there any solution? I am using
provider registry.terraform.io/hashicorp/aws v3.27.0
I would recommend you for Switching from aws_launch_configuration to aws_launch_template as it already have support for gp3 instances.

How to switch to 'aws_launch_template'

innovate-invent · 2021-02-17T19:47:29Z

How to switch to 'aws_launch_template'

Someone correct me if I am wrong but in your module config you just change worker_groups = [...] to worker_groups_launch_template = [...]

LukaszRacon · 2021-02-17T19:52:33Z

If you switch from worker_groups to worker_groups_launch_template - you might also want to preserve ASG (adjust for location of your module):
terraform state mv module.eks.aws_autoscaling_group.workers[0] module.eks.aws_autoscaling_group.workers_launch_template[0]

luis02lopez · 2021-02-17T20:13:58Z

For "aws" (2.70.0), using launch templates, still, receive this error:

Error: aws_launch_template.module: expected block_device_mappings.0.ebs.0.volume_type to be one of [standard gp2 io1 sc1 st1], got gp3

innovate-invent · 2021-02-17T20:26:09Z

I tried this in us-east-1 and it succeeded after making the change I suggested.

Edit: It appears with my suggested change the autoscaler no longer works.
invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
After the first node fills the remaining pods fail to schedule.

Edit2: This may be related to the pods failing to match a PVC on the new node

Edit3: I think the launch template needs to specify the PVC somehow

Edit4: Everything works now. I somehow managed to unlink a role binding for the CSI driver...

vbagmar · 2021-03-07T00:55:04Z

What's the fix? I had to change to gp2 for us-west-1

innovate-invent · 2021-03-07T00:58:32Z

@vbagmar #1205 (comment)

vsenapathy · 2021-03-08T18:03:03Z

Do we know when gp3 will be available for Launch configuration?

innovate-invent · 2021-03-08T18:06:11Z

@vsenapathy is there a reason to not switch to launch templates? From what I understand, launch configurations are depreciated.

vsenapathy · 2021-03-08T18:07:27Z

Oh i can move to Launch templates and test this.

vsenapathy · 2021-03-08T23:58:53Z

@innovate-invent: I was able to get this to work with Launch template.

sollie · 2021-03-10T09:53:45Z

@innovate-invent

From what I understand, launch configurations are depreciated.

Do you have a source for that? Couldn't find anything with a quick google search.

innovate-invent · 2021-03-10T15:06:27Z

@sollie #1205 (comment)

sollie · 2021-03-11T15:11:21Z

@innovate-invent ah, thanks. Missed that while skimming.

razor-x · 2021-03-26T04:17:26Z

@vsenapathy is there a reason to not switch to launch templates? From what I understand, launch configurations are depreciated.

At least for me this is the main blocker for moving to launch templates: terraform-aws-modules/terraform-aws-autoscaling#54

fabianomartinsrj · 2021-04-13T15:41:47Z

I've made an override to default values on my template, with no need to change the values from module.

    {
     ...
      root_volume_type   = "gp2"
      ...
    }
  ]

github-actions · 2022-11-22T02:27:56Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

barryib mentioned this issue Jan 29, 2021

fix: Change back the default disk type to gp2 #1208

Merged

barryib mentioned this issue Jan 29, 2021

feat: Use gp3 as default as it saves 20% and is more performant #1134

Merged

1 task

sharjeelsayed mentioned this issue Jan 31, 2021

ValidationError: gp3 is invalid. Valid volume types are standard, io1, gp2, st1 and sc1 #1210

Closed

DmitriyStoyanov mentioned this issue Feb 17, 2021

Unresponsive aws-node CNI 1.7.5 aws/amazon-vpc-cni-k8s#1372

Closed

soulchips mentioned this issue Mar 17, 2021

Fix/expose volume type CircleCI-Public/server-terraform#94

Merged

james-callahan mentioned this issue Mar 23, 2021

cannot use GP3 root volumes #1244

Closed

1 task

barryib closed this as completed in #1208 Apr 19, 2021

inge4pres mentioned this issue Apr 29, 2021

Invalid dynamic for_each value for launch templates #1325

Closed

github-actions bot locked as resolved and limited conversation to collaborators Nov 22, 2022

ValidationError: gp3 is invalid #1205

ValidationError: gp3 is invalid #1205

Comments

ichasco-heytrade commented Jan 29, 2021

I have issues

I'm submitting a...

What is the current behavior?

If this is a bug, how to reproduce? Please include a code sample if relevant.

Environment details

Any other relevant info

oscr commented Jan 29, 2021

ichasco-heytrade commented Jan 29, 2021

oscr commented Jan 29, 2021

barryib commented Jan 29, 2021

oscr commented Jan 29, 2021

ichasco-heytrade commented Jan 29, 2021 via email • edited Loading

GerardoGR commented Jan 29, 2021

oscr commented Jan 29, 2021

marcuz commented Jan 29, 2021

barryib commented Jan 29, 2021

mzupan commented Jan 29, 2021 • edited Loading

barryib commented Jan 29, 2021

margamanterola commented Jan 29, 2021

barryib commented Jan 29, 2021 • edited Loading

sharjeelsayed commented Jan 31, 2021

ichasco-heytrade commented Jan 31, 2021

sharjeelsayed commented Jan 31, 2021

shahafy-ma commented Feb 10, 2021 • edited Loading

Saeger commented Feb 10, 2021

albertoal commented Feb 10, 2021 • edited Loading

kaushikarvind commented Feb 13, 2021

innovate-invent commented Feb 17, 2021

LukaszRacon commented Feb 17, 2021

luis02lopez commented Feb 17, 2021

innovate-invent commented Feb 17, 2021 • edited Loading

vbagmar commented Mar 7, 2021

innovate-invent commented Mar 7, 2021

vsenapathy commented Mar 8, 2021

innovate-invent commented Mar 8, 2021

vsenapathy commented Mar 8, 2021

vsenapathy commented Mar 8, 2021 • edited Loading

sollie commented Mar 10, 2021

innovate-invent commented Mar 10, 2021

sollie commented Mar 11, 2021

razor-x commented Mar 26, 2021

fabianomartinsrj commented Apr 13, 2021 • edited Loading

github-actions bot commented Nov 22, 2022

ichasco-heytrade commented Jan 29, 2021 via email •

edited

Loading

mzupan commented Jan 29, 2021 •

edited

Loading

barryib commented Jan 29, 2021 •

edited

Loading

shahafy-ma commented Feb 10, 2021 •

edited

Loading

albertoal commented Feb 10, 2021 •

edited

Loading

innovate-invent commented Feb 17, 2021 •

edited

Loading

vsenapathy commented Mar 8, 2021 •

edited

Loading

fabianomartinsrj commented Apr 13, 2021 •

edited

Loading