Applies zone labels to newly created vsphere volumes

[subramanian-neelakantan]

What type of PR is this?
/kind bug

What this PR does / why we need it:
When a volume is created on vSphere, the zone label does not get applied on it today. Without this information on the volume, a pod that uses this volume could get scheduled to a different zone where this volume is not visible/accessible. This issue can be solved by just labeling the volume with the zone information. This PR computes the zone of a volume in the PersistentVolumeLabel admission controller since vsphere is still an in-tree cloud provider and the out-of-tree plugin in still the works.

Which issue(s) this PR fixes:
Fixes #73301
Part of issue #67703

Special notes for your reviewer:
This part does not guarantee that allowedTopologies specified in a StorageClass will be honoured. That would be done as part of a different PR.

Does this PR introduce a user-facing change?:
Yes

This change applies zone labels to vSphere Volumes automatically. The zone labels are visible on the PV:
$ kubectl get pv --show-labels
NAME           CAPACITY   ACCESSMODES   STATUS    CLAIM            REASON    AGE       LABELS
pv-abc                5Gi        RWO                   Bound     default/claim1                    46s       failure-domain.beta.kubernetes.io/region=VC1,failure-domain.beta.kubernetes.io/zone=cluster-1

[k8s-ci-robot]

Hi @subramanian-neelakantan. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[subramanian-neelakantan]

/test all

[k8s-ci-robot]

@subramanian-neelakantan: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test all

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[subramanian-neelakantan]

/cc @abrarshivani @SandeepPissay @divyenpatel

[subramanian-neelakantan]

@SandeepPissay Thanks for the review and both good suggestions.

The main concern I have with this PR is performance -

• datastore to zone/region mapping is static but we seem to be computing the zone for the same datastore for every volume created on it.

Let us track this performance improvement as a separate task since it fits in easily as an incremental change. Here is the issue I just created to track this - #73955

• computing volume labels logic is time consuming as it involves multiple remote calls. This is done even if zones feature is not enabled in k8s cluster.

There is no way to disable the 'zones' feature in a k8s cluster AFAICT. The 'Labels' section of the vsphere.conf should not be treated as a feature gate. It only over-rides the tag category name used for zone and region.

[SandeepPissay]

@subramanian-neelakantan I would like to see the content of vsphere.conf with this code change? Could you paste it here?

I'm wondering if we can have a flag in vSphere.conf to explicitly enable multi zones support in VCP.

[frapposelli]

My understanding is that zones support is enabled only when the zones parameters are set in vsphere.conf otherwise support is disabled.

[SandeepPissay]

If zones feature is enabled by explicitly setting zones params in vsphere.conf, then VCP/vsphere volume plugin should not be invoking tag service APIs at all if the zones param is not present(zones features disabled). But that is not the case in this PR, which is what I'm highlighting here.

If there is a way to determine if zones feature is disabled, I would suggest to skip tag service API calls into Virtual Center. Else, this PR will introduce performance regression.

[subramanian-neelakantan]

With this change, zone support is enabled if the vCenter inventory (Datacenter/Cluster/Host) is tagged with zone/region tags. VCP looks for tags with the category name "zone" and "region" to identify this. The vsphere.conf is only used to over-ride the default category names of "zone" and "region". So Tag API calls are used to determine whether zones support is enabled or not.

My opinion is that we do not need another back door feature enablement (through vsphere.conf) for the zones support. It should be present once VCP has implemented this k8s feature. The performance implication of such a feature support, if any, should be addressed separately (see issue #73955).

OTOH, if we feel strongly against supporting zone feature by default in VCP, I can revert the part of code that makes "zone" and "region" as the default category names. That way, zones will be discovered from VC only when the vsphere.conf has such an entry:
[Labels]
zone = <vc_zone_tag_category_name>
region = <vc_region_tag_category_name>

[SandeepPissay]

Most of VMware customers are not using zones and may not need this feature, so this change will introduce performance regression for them which I'm suggesting to avoid. This can be a simple flag that gets set during VCP init.

[frapposelli]

/cc @dougm @jiatongw PTAL, I don't remember the context/details on why the zones feature was made entirely optional

[dougm]

We chose to make zones/tags optional to avoid breaking existing user configs on upgrades of K8s.

[subramanian-neelakantan]

Most of VMware customers are not using zones and may not need this feature, so this change will introduce performance regression for them which I'm suggesting to avoid. This can be a simple flag that gets set during VCP init.

Since I have not heard any objections to this approach, I have made this change in commit ba9a9cf to treat the "zone and region [Labels]" entries in vsphere.conf file to indicate whether zone support is enabled for VCP. When these entries are not present in vsphere.conf, VCP avoid zone based computation. In fact, it avoids discovering the tags from VC entirely.

@SandeepPissay PTAL.

[frapposelli]

thanks @dougm for the context

@subramanian-neelakantan the approach you took with the latest commit seems to follow the original intent of the Zones feature so

/lgtm

leaving approval to @SandeepPissay

[SandeepPissay]

/lgtm

[SandeepPissay]

Move this code to after zones check as this is redundant if zones is not enabled.

[vladimirvivien]

@SandeepPissay don't forget to mark this PR as /approved so it'll get merged.

[vladimirvivien]

Will this be cherry picked to an earlier version?

[SandeepPissay]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andrewsykim, frapposelli, SandeepPissay, subramanian-neelakantan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/cloudprovider/providers/vsphere/OWNERS [SandeepPissay,andrewsykim,frapposelli]
• pkg/volume/vsphere_volume/OWNERS [SandeepPissay]
• plugin/pkg/admission/storage/persistentvolume/label/OWNERS [andrewsykim]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment