Add note in Azure autoscaler documentation about adding accelerator node label on VMSS when GPU nodes are involved.

[Weihrauch]

Which component this PR applies to?

cluster-autoscaler

What type of PR is this?

/kind documentation

What this PR does / why we need it:

I struggled for a long running issue arround GPU node scaling in a Rancher RKE Kubernetes Cluster using Azure VM Scale Set. The scale up process freeze after node count back to zero. After digging the source code I discover that label accelerator (hardcoded in azure cloud provider) should be setted on kubelet node to make cluster-autoscaler handle correctly GPU ressources.
I guess it would help further user to deal with it.

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

For more information on release notes see: https://git.k8s.io/community/contributors/guide/release-notes.md
-->

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: Weihrauch / name: Valentin Pauchet (ce31d95)

[k8s-ci-robot]

Welcome @Weihrauch!

It looks like this is your first PR to kubernetes/autoscaler 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/autoscaler has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[Weihrauch]

/assign @marwanad

[marwanad]

The GPU label isn't really an Azure-specific thing (nor VMSS for that matter) and it is part of the cloudprovider API contract so every cloudprovider has its notion of defining what the label on GPU nodes is so that CA can identify those.

autoscaler/cluster-autoscaler/cloudprovider/cloud_provider.go

Lines 112 to 113 in a13c59c

	// GPULabel returns the label added to nodes with GPU resource.
	GPULabel() string

I wonder if we should put a note in the main FAQ instead.

https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md

[mwielgus]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mwielgus, Weihrauch

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• cluster-autoscaler/cloudprovider/azure/OWNERS [mwielgus]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mweibel]

thanks for this change @Weihrauch

is it correct that you'd need to set the label accelerator with value nvidia-tesla-v100?

[Weihrauch]

Hello mweibel,
On my side I only had labl accelerator. This requirement trigger GPU instance handling in autoscaler algorithm as you can see in NodeHasGpu function. Setting value to this label may configure other things I have no ideas by now.
func NodeHasGpu(GPULabel string, node *apiv1.Node) bool { _, hasGpuLabel := node.Labels[GPULabel] gpuAllocatable, hasGpuAllocatable := node.Status.Allocatable[ResourceNvidiaGPU] return hasGpuLabel || (hasGpuAllocatable && !gpuAllocatable.IsZero()) }