Make the EBS CSI driver configurable

[hardikdr]

How to categorize this issue?

/area usability
/kind enhancement
/priority normal

What would you like to be added: The EBS CSI driver v0.6.0 has been released recently. It contains the changes, allowing users to overwrite the max-attachable-volumes per node for the cluster.

We need to enable both a new version of CSI-driver and users to be able to configure the volume-attach-limit flag of the csi-driver via shoot-resource.

To mention, this is part of the issue gardener/gardener#2354 , and a prominent need for the MemoryOne integration to work at its best with k8s 1.18+.

Why is this needed: The environment variable KUBE_MAX_PD_VOLS is ineffective with CSI migration, which leaves no way for a user to configure the max-attachable-volumes.

[hardikdr]

cc @rfranzke

[ialidzhikov]

/assign @ialidzhikov

[rfranzke]

/topology shoot
/in-progress

The idea is that the end-user can configure the volume attach limit by the following annotation on the shoot: aws.provider.extensions.gardener.cloud/volume-attach-limit: "42".
The AWS control plane extension can read this information from the Cluster resource and configure the CSI driver in the shoot properly.

[ialidzhikov]

The idea is that the end-user can configure the volume attach limit by the following annotation on the shoot: aws.provider.extensions.gardener.cloud/volume-attach-limit: "42".
The AWS control plane extension can read this information from the Cluster resource and configure the CSI driver in the shoot properly.

Currently the gardener-apiserver won't increase the metadata.generation hence Shoot won't be triggered for reconciliation when the annotation is newly added. Any ideas how to circumvent that? Annotation key matching in gardener-apiserver and increasing the generation when the annotation contains extensions.gardener.cloud? 😅

I will now propose PR for the v0.6.0 upgrade and a draft PR for the --volume-attach-limit handling.

[rfranzke]

Well, I wouldn't do anything for now and simply state that the user can trigger the reconciliation using gardener.cloud/operation=reconcile annotation.