[EKS] [request]: Allow feature gates to be set on master components

[jammerful]

Tell us about your request
I would like to be able to set feature gates on master components such as the kube-apiserver, this would allow me to enable features early.

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
Right now I can't enable feature gates on any master components, while I have a particular feature gate in mind I would like the general ability to toggle feature flags upon creating/updating a cluster. I understand that the feature gates enable alpha features, so I would simply note in the documentation and a warning upon any feature flag being set that using this feature can make your cluster unstable and all feature gates set should be throughly tested before releasing to production.

The particular feature gate is the VolumeSnapshotDataSource which is needed for the EBS CSI Driver's snapshotting feature.

Are you currently working around this issue?
I will deploy my own container to do the snapshotting myself.

[lanefu]

@jammerful hey I'm working on similar issue.... would you be able to explain your workaround a little more in-depth? What container are you deploying?

[jammerful]

@lanefu I'm assuming you want the volume snapshot feature, if you're trying to enable a random feature gate I don't think you can do that.
For volume snapshots, you can use the now deprecated external storage code located here https://github.com/kubernetes-incubator/external-storage/tree/master/snapshot. I've been using it for the past few weeks, and it's working well. In the repo you will find the manifest to deploy and instructions on how to use the CRDs.
Lastly, snapshotting in the CSI is beta in 1.17, so maybe by next spring/summer will be available on EKS.

[rinrailin]

Another good feature gate to be able to enable is VolumeSubpathEnvExpansion which is in GA for k8s v1.17 but in alpha for k8s v1.14

[dgarbus]

Shouldn't the label be EKS rather than ECS?

[jammerful]

@ellenthsu Please remove the ECS label and add the EKS label.

[jammerful]

Apparently alpha feature gates are not enabled on the stable channels of AKS or GKE, but they do have unstable channels that enable beta feature gates. Given that I don't know if AWS would consider doing this for production clusters, wish they would speak up.

[ellenthsu]

@ellenthsu Please remove the ECS label and add the EKS label.

done

[chris-vest]

I'm wanting to enable the ProcMountType feature gate in order to run Docker rootless, without having to use privileged pods.

[tokiwong]

I would like this feature to enable ServerSideApply

[spanky-medal]

We're looking for ServiceNodeExclusion since at scale, we can hit an AWS NLB service limit of 500 targets. The services we scale will not be used via the NLB (they're queue-based) so they just fill up NLB slots for no reason.

[themish95]

My cx is looking to enable SCTP and it requires the SCTPSupport feature gate

https://kubernetes.io/docs/concepts/services-networking/network-policies/#sctp-support

[hymgg]

Need to support SCTP for our telco apps too.

[mikeaorlando]

I am interested in this to enable Service Topology in EKS 1.17. I am following an example to reduce cross availability zone traffic (https://v1-17.docs.kubernetes.io/docs/concepts/services-networking/service-topology/#prefer-node-local-zonal-then-regional-endpoints) which require feature-flags to be accessible: https://v1-17.docs.kubernetes.io/docs/tasks/administer-cluster/enabling-service-topology/#enable-service-topology

[lanefu]

@mikeaorlando something that I discovered.....

If you're using terraform modules, it's easy to switch from EKS managed nodegroups to "worker groups". the level of config effort is about the same in terraform whether managed or unmanaged, and you can easily pass arguments for feature flags.. EX:

workers_group_defaults = {
  instance_type = "t3a.xlarge"
  key_name = "secret_key"
  kubelet_extra_args = "--feature-gates=ExpandCSIVolumes=true,ExpandPersistentVolumes=true"
}

[DanOfir]

@mikeaorlando @lanefu i am also trying to use Service Topology running on EKS 1.17, i created the feature gates as @lanefu described, how can i make sure the feature is enabled? @mikeaorlando did you succeeded using the feature in EKS at all? cause every time i create/edit service with the TopologyKeys it seems to be deleted after saving.
any advice guys?

[chasmosis]

@mikeaorlando @DanOfir I am in the same boat as you wanting to use Service Topology on EKS 1.17 and find that since it has to be enabled at the API server label I'm unable to set that feature gate.
We are wanting to use https://kubernetes.io/docs/concepts/services-networking/service-topology/#only-node-local-endpoints for host binding a service on only the nodes running the pods for that service.

[mikeaorlando]

I have been unable to use Service Topology running on EKS 1.17. My current workaround is to have all nodes in the cluster in one availability zone. This does not constrict traffic to be node-local, but reduces ingress and egress charges from cross-az traffic. My current plan is to wait until EKS 1.18.

This was helpful in determining which feature flags were set in the api-server: https://docs.aws.amazon.com/eks/latest/userguide/api-server-flags.html

[3oris]

In our case, we would want to set --feature-gates=IAMIdentityMappingCRD

[nicolai86]

In our case we want to set --feature-gates=EvenPodsSpread on EKS 1.18...

[mbelang]

Would it be possible to have a generic way to enable any feature-gates in EKS?

I'm on 1.18 and want StartupProbes

[llamahunter]

In our case we want to set --feature-gates=EvenPodsSpread on EKS 1.18...

Wait, AWS put out release notes specifically saying that pod topologySpreadConstraints were supported, but did not enable the feature gate?
https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html

Pod Topology Spread has reached beta status. You can use topology spread constraints to control how pods are spread across your cluster among failure-domains such as Regions, zones, nodes, and other user-defined topology domains. This can help to achieve high availability as well as efficient resource utilization. For more information, see Pod Topology Spread Constraints in the Kubernetes documentation.

[nicolai86]

@llamahunter it seems beta features are enabled by default in k8s, so the feature gate doesn't need to be passed explicitly anymore.

[llamahunter]

@nicolai86 Have you successfully set pod topologySpreadConstraints using topology.kubernetes.io/zone topologyKeys in EKS v1.18?

[s2504s]

It would be nice to have CrossNamespaceVolumeDataSource, which was introduced in 1.26
Thanks in advance 😃

PS
I think this issue kubernetes-sigs/aws-ebs-csi-driver#1682 is related to this feature request

[everythings-gonna-be-alright]

Yep, CrossNamespaceVolumeDataSource is a really very useful feature. Especially in cases where you need to create multi-environments from one database snapshot.

[dims]

@anton-vylushchak @s2504s @everythings-gonna-be-alright CrossNamespaceVolumeDataSource and MaxUnavailableStatefulSet are both still in alpha and defaulted to false (even in upcoming 1.28):
https://github.com/kubernetes/kubernetes/blob/master/pkg/features/kube_features.go#L956

When the community elevates them to beta (and defaults them to true) then we'll in inherit the same in EKS.

[hontarenko]

It would be really great if we could get InPlacePodVerticalScaling

[rahul-paigavan]

I would like to disable a feature that is by default (beta version) enabled in EKS 1.23
I feel the feature is having some issues that are fixed in (stable version) EKS 1.26, and causing issues in my EKS 1.23 cluster.
There is some time for us (in production) to currently move beyond EKS 1.23+ clusters, Hence I wanted to disable a specific feature (ServiceInternalTrafficPolicy) and continue using EKS 1.23 for few more months.

[kitsirota]

InPlacePodVerticalScaling is a game changer for our workloads. Is there an ETA for when this feature gate would be available in EKS?

[dims]

@kitsirota see above #512 (comment) - the situation is the same for any feature gate in upstream.

[grglzrv]

InPlacePodVerticalScaling is a game changer for our workloads. Is there an ETA for when this feature gate would be available in EKS?

Agreed 💯

[Noksa]

4 year anniversary and we are still there where we are

[dims]

@grglzrv @Noksa if you have any specific things you are looking for, please comment which feature gate and why you need it (and for which kubernetes versions). Then we can have a healthy discussion about those specific use cases, otherwise it is not productive.

[grglzrv]

@dims, in our company(NEXO) we use AWS EKS 1.28(managed cluster), so we wanna use feature gate InPlacePodVerticalScaling but this feature is not enabled by default. Do you know how we may enable it? We wanna achieve this approach https://piotrminkowski.com/2023/08/22/resize-cpu-limit-to-speed-up-java-startup-on-kubernetes/

[dims]

@grglzrv see upstream issue kubernetes/enhancements#1287 - the feature will hit Beta in 1.30, that is typically when community enables it automatically (without needing to switch on a feature gate). We follow the community strictly, so we will be doing the same. You can check the upstream code on the feature status here.

You can see at least 3 fixes going intoInPlacePodVerticalScaling feature since it first landed:
https://github.com/kubernetes/kubernetes/pulls?q=is%3Apr+InPlacePodVerticalScaling+is%3Amerged+sort%3Acreated-desc+author%3Avinaykul

You can see the list of open issues against this feature:
https://github.com/kubernetes/kubernetes/issues?q=is%3Aissue+is%3Aopen+InPlacePodVerticalScaling

Before you ask what it would take to make this go from alpha to beta, please see here:
https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/1287-in-place-update-pod-resources#beta

Presence of a feature gate does not mean it is ready by any means... you can also read about progression of feature gates in the community here:
https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/feature-gates.md

Hope this helps!

[grglzrv]

@grglzrv see upstream issue kubernetes/enhancements#1287 - the feature will hit Beta in 1.30, that is typically when community enables it automatically (without needing to switch on a feature gate). We follow the community strictly, so we will be doing the same. You can check the upstream code on the feature status here.

You can see at least 3 fixes going intoInPlacePodVerticalScaling feature since it first landed: https://github.com/kubernetes/kubernetes/pulls?q=is%3Apr+InPlacePodVerticalScaling+is%3Amerged+sort%3Acreated-desc+author%3Avinaykul

You can see the list of open issues against this feature: https://github.com/kubernetes/kubernetes/issues?q=is%3Aissue+is%3Aopen+InPlacePodVerticalScaling

Before you ask what it would take to make this go from alpha to beta, please see here: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/1287-in-place-update-pod-resources#beta

Presence of a feature gate does not mean it is ready by any means... you can also read about progression of feature gates in the community here: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/feature-gates.md

Hope this helps!

Thanks, I appreciate your reply 🙏

[Revolution1]

any update?

[sidewinder12s]

@dims Does EKS have a more generic feature ask to support Alpha clusters similar to GKE Alpha Clusters, where all alpha feature flags are enabled/support is best effort/no guarantees?

[dims]

@dims Does EKS have a more generic feature ask to support Alpha clusters similar to GKE Alpha Clusters, where all alpha feature flags are enabled/support is best effort/no guarantees?

I don't remember seeing one @sidewinder12s

[dmarkhas]

We are interested in ValidatingAdmissionPolicy feature gate for 1.28, in which it was promoted to Beta.

[sedatsencan]

"--feature-gates=InPlacePodVerticalScaling=true"

I found myself here while trying to define the above parameter. Any update for EKS?

[WarpRat]

Can I please get a clarification on the documentation? According to https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html

Which Kubernetes features are supported by Amazon EKS?
Amazon EKS supports all generally available (GA) features of the Kubernetes API. Starting with Kubernetes version 1.24, new beta APIs aren't enabled in clusters by default. However, previously existing beta APIs and new versions of existing beta APIs continue to be enabled by default. Alpha features aren't supported.

Does this mean that any feature gate added after 1.24 is disabled until it's GA? Or that EKS follows the upstream kubernetes default for whether a beta feature gate is enabled or not and before 1.24 all beta gates were enabled regardless of the default?

[faymard]

Hi, we have high interest in the ServiceTrafficDistribution feature gate, introduced as an Alpha feature in Kubernetes 1.30. This will help us in reducing our cross-AZ traffic costs drastically and we are very much willing to help the Kubernetes team and the EKS team in stabilizing the feature.

[dejwsz]

I think docs are quite clear - GA are supported and enabled, beta and thier updates - only those before 1.24 (later ones are not supported), alpha are not supported and not enabled - that is the general policy. InPlacePodVerticalScaling is denoted as alpha one so you can forget about it for now in EKS, it can be checked here: https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/

[jcogilvie]

@dejwsz the docs are clear on how it currently works. This feature request is asking for it to work differently.

[realhuseyn]

Hello. I want to use InPlacePodVerticalScaling feature on my EKS cluster. EKS version is v1.27. Which version needs for it?

[lukasmrtvy]

@realhuseyn bruh, it is not even in beta yet, currently deferred to 1.32 ( kubernetes/enhancements#1287 ), proly 1.34 stable, thus available for EKS

[eugenestarchenko]

I'm here for
https://kubernetes.io/blog/2023/01/06/unhealthy-pod-eviction-policy-for-pdbs/#how-do-i-use-it
Looks like a solution aws/karpenter-provider-aws#3875 (comment)