Disable accelerated network for AvSet clusters

[vpnachev]

How to categorize this PR?

/area networking
/kind bug
/priority critical
/platform azure

What this PR does / why we need it:
Disable accelerated networking for AvSet clusters. This is necessary because for legacy clusters it is not possible to create new VMs. The creation calls are failing with

ExistingAvailabilitySetWasNotDeployedOnAcceleratedNetworkingEnabledCluster" Message="Cannot add a virtual machine (/subscriptions/...) with accelerated-networking-enabled network interface(s) on an existing availability set (/subscriptions/...) with VM(s) without accelerated-networking-enabled network interfaces.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

See MicrosoftDocs/azure-docs#10536 for more details.

Release note:

Accelerated Networking feature has been disabled for AvSet clusters. It is still available for zoned clusters. 

[rfranzke]

Very nice PR, lgtm, I would just enhance our user-facing docs. :)

[rfranzke]

/lgtm

[dkistner]

Oh thats a pity. Once more on case which was not part of the test matrix... To many deployment scenarios on Azure...

So we can reenable it when we have an own avset for each workerpool. These avset contain then only machines of same type and configuration (e.g. operating system).

[vpnachev]

So we can reenable it when we have an own avset for each workerpool. These avset contain then only machines of same type and configuration (e.g. operating system).

I am not sure - if the worker pool is rolled from disabled accelerated network to enabled, we will face the same issue. So we have to keep status of the AvSet and disallow accelerated networking if it still has NICs with disabled accelerated networking. Which means we will have to list all NICs in the AvSet which is one more operation contributing to the Azure API calls.

[dkistner]

Hmm. I rather thinking in the direction of creating a new avset in case the machineclass spec is changing. The rolled machines (with acc networking) would then be part of a new avset and the not rolled ones would be part of the old avset.

[vpnachev]

Valid point, I've opened #101 to summarize the topic and keep track on it there.