`azurerm_kubernetes_cluster_node_pool` - `os_sku` supports update #26139

ms-henglu · 2024-05-29T08:33:41Z

Community Note

Please vote on this PR by adding a 👍 reaction to the original PR to help the community and maintainers prioritize for review
Please do not leave "+1" or "me too" comments, they generate extra noise for PR followers and do not help prioritize for review

Description

PR Checklist

I have followed the guidelines in our Contributing Documentation.
I have checked to ensure there aren't other open Pull Requests for the same update/change.
I have checked if my changes close any open issues. If so please include appropriate closing keywords below.
I have updated/added Documentation as required written in a helpful and kind way to assist users that may be unfamiliar with the resource / data source.
I have used a meaningful PR title to help maintainers and other users understand this change and help prevent duplicate work.
For example: “resource_name_here - description of change e.g. adding property new_property_name_here”

Changes to existing Resource / Data Source

I have added an explanation of what my changes do and why I'd like you to include them (This may be covered by linking to an issue above, but may benefit from additional explanation).
I have written new tests for my resource or datasource changes & updated any relevent documentation.
I have successfully run tests with my changes locally. If not, please provide details on testing challenges that prevented you running the tests.
(For changes that include a state migration only). I have manually tested the migration path between relevant versions of the provider.

Testing

My submission includes Test coverage as described in the Contribution Guide and the tests pass. (if this is not possible for any reason, please include details of why you did or could not add test coverage)

=== RUN   TestAccKubernetesClusterNodePool_osSkuMigration
=== PAUSE TestAccKubernetesClusterNodePool_osSkuMigration
=== CONT  TestAccKubernetesClusterNodePool_osSkuMigration
--- PASS: TestAccKubernetesClusterNodePool_osSkuMigration (1066.68s)
PASS

Change Log

Below please provide what should go into the changelog (if anything) conforming to the Changelog Format documented here.

azurerm_resource - support for the thing1 property [GH-00000]

This is a (please select all that apply):

Bug Fix
New Feature (ie adding a service, resource, or data source)
Enhancement
Breaking Change

Related Issue(s)

Fixes #0000

Note

If this PR changes meaningfully during the course of review please update the title and description as required.

stephybun

Thanks @ms-henglu but given the ongoing discussions about whether to support preview features in the AKS resource or not we can't progress with this since this may potentially need to be removed again in the next major version of the provider.

Could we close this PR for the moment, we're happy to reopen once we have a resolution on the discussion.

ms-henglu · 2024-05-29T09:56:48Z

Agreed! I'll close this task.

stephybun · 2024-06-05T07:28:56Z

website/docs/r/kubernetes_cluster_node_pool.html.markdown

-* `os_sku` - (Optional) Specifies the OS SKU used by the agent pool. Possible values are `AzureLinux`, `Ubuntu`, `Windows2019` and `Windows2022`. If not specified, the default is `Ubuntu` if OSType=Linux or `Windows2019` if OSType=Windows. And the default Windows OSSKU will be changed to `Windows2022` after Windows2019 is deprecated. Changing this forces a new resource to be created.
+* `os_sku` - (Optional) Specifies the OS SKU used by the agent pool. Possible values are `AzureLinux`, `Ubuntu`, `Windows2019` and `Windows2022`. If not specified, the default is `Ubuntu` if OSType=Linux or `Windows2019` if OSType=Windows. And the default Windows OSSKU will be changed to `Windows2022` after Windows2019 is deprecated. Changing this from `AzureLinux` or `Ubuntu` to `AzureLinux` or `Ubuntu` will not replace the resource, otherwise it forces a new resource to be created.
+
+-> **Note:**  This requires that the Preview Feature `Microsoft.ContainerService/OSSKUMigrationPreview` is enabled and the Resource Provider is re-registered, see [the documentation](https://learn.microsoft.com/en-us/azure/azure-linux/tutorial-azure-linux-migration) for more information.


This should be removed if it's not really a preview feature anymore

Suggested change

-> **Note:** This requires that the Preview Feature `Microsoft.ContainerService/OSSKUMigrationPreview` is enabled and the Resource Provider is re-registered, see [the documentation](https://learn.microsoft.com/en-us/azure/azure-linux/tutorial-azure-linux-migration) for more information.

I'll remove it when it's GA, currently we still need this feature registered.

Does it though? That flag isn't registered on our testing subscription

$ az feature show --namespace "Microsoft.ContainerService" --name "OSSKUMigrationPreview" { "id": "/subscriptions/*/providers/Microsoft.Features/providers/microsoft.ContainerService/features/OSSKUMigrationPreview", "name": "microsoft.ContainerService/OSSKUMigrationPreview", "properties": { "state": "NotRegistered" }, "type": "Microsoft.Features/providers/features" }

And the test passes

Got it, I'll remove the notice.

Co-authored-by: stephybun <[email protected]>

hbeberman

Left a few comments on things that need to change.

Additionally the function resourceKubernetesClusterNodePoolUpdate in kubernetes_cluster_node_pool_resource.go will need to apply the os_sku if it's changed.

	if d.HasChange("os_sku") {
		if osSku := d.Get("os_sku").(string); osSku != "" {
			props.OsSKU = pointer.To(agentpools.OSSKU(osSku))
		}
	}

hbeberman · 2024-06-24T19:52:27Z

internal/services/containers/kubernetes_cluster_node_pool_resource.go

@@ -67,6 +67,18 @@ func resourceKubernetesClusterNodePool() *pluginsdk.Resource {
 		Schema: resourceKubernetesClusterNodePoolSchema(),

 		CustomizeDiff: pluginsdk.CustomDiffInSequence(
+			pluginsdk.ForceNewIfChange("os_sku", func(ctx context.Context, old, new, meta interface{}) bool {
+				// Ubuntu and AzureLinux are currently the only allowed Linux OSSKU Migration targets.
+				if old != agentpools.OSSKUUbuntu && old != agentpools.OSSKUAzureLinux {


These comparisons are comparing a string object to an agentpools.OSSKU object, so it'll always return true from the function since they'll never be equal. The agentpools consts need to be typecast for the comparison.

Suggested change

if old != agentpools.OSSKUUbuntu && old != agentpools.OSSKUAzureLinux {

if old != string(agentpools.OSSKUUbuntu) && old != string(agentpools.OSSKUAzureLinux) {

hbeberman · 2024-06-24T19:57:19Z

internal/services/containers/kubernetes_cluster_node_pool_resource.go

@@ -67,6 +67,18 @@ func resourceKubernetesClusterNodePool() *pluginsdk.Resource {
 		Schema: resourceKubernetesClusterNodePoolSchema(),

 		CustomizeDiff: pluginsdk.CustomDiffInSequence(
+			pluginsdk.ForceNewIfChange("os_sku", func(ctx context.Context, old, new, meta interface{}) bool {


This CustomDiff only applies to azurerm_kubernetes_cluster_node_pool resources.

We need a copy of it in kubernetes_cluster_resource.go too so that the logic applies when validating default_node_pool in azurerm_kubernetes_cluster.

hbeberman · 2024-06-24T19:59:00Z

internal/services/containers/kubernetes_cluster_node_pool_resource.go

+					return true
+				}
+
+				if new != agentpools.OSSKUUbuntu && new != agentpools.OSSKUAzureLinux {


Suggested change

if new != agentpools.OSSKUUbuntu && new != agentpools.OSSKUAzureLinux {

if new != string(agentpools.OSSKUUbuntu) && new != string(agentpools.OSSKUAzureLinux) {

hbeberman · 2024-06-24T20:42:31Z

internal/services/containers/kubernetes_cluster_node_pool_resource.go

@@ -269,7 +281,6 @@ func resourceKubernetesClusterNodePoolSchema() map[string]*pluginsdk.Schema {
 		"os_sku": {
 			Type:     pluginsdk.TypeString,
 			Optional: true,
-			ForceNew: true,


Need to remove "default_node_pool.0.os_sku", from cycleNodePoolProperties in kubernetes_cluster_resource.go because a temporary_name_for_rotation is no longer required since we can in-place upgrade.

Yes, I opened a separate PR for kubernetes_cluster_resource

ms-henglu · 2024-06-25T07:02:49Z

Thanks @hbeberman for the code review. I've updated this PR as suggested.

ms-henglu · 2024-06-29T06:05:18Z

Hi @stephybun , the tests were passing because there's a bug in my PR that the forceNew:true is not removed, so the node pool is replaced.

I've fixed it, and the tests are failing because the preview feature flag is not enabled:

=== RUN   TestAccKubernetesClusterNodePool_osSkuMigration
=== PAUSE TestAccKubernetesClusterNodePool_osSkuMigration
=== CONT  TestAccKubernetesClusterNodePool_osSkuMigration
    testcase.go:113: Step 3/6 error: Error running apply: exit status 1
        Error: updating Node Pool Agent Pool (Subscription: "*******"
        Resource Group Name: "acctestRG-aks-240628111457303318"
        Managed Cluster Name: "acctestaks240628111457303318"
        Agent Pool Name: "internal"): performing CreateOrUpdate: unexpected status 400 (400 Bad Request) with response: {
          "code": "PropertyChangeNotAllowed",
          "details": null,
          "message": "Changing property 'agentPoolProfile.OSSKU' is not allowed.",
          "subcode": "",
          "target": "agentPoolProfile.OSSKU"
         }
          with azurerm_kubernetes_cluster_node_pool.test,
          on terraform_plugin_test.tf line 45, in resource "azurerm_kubernetes_cluster_node_pool" "test":
          45: resource "azurerm_kubernetes_cluster_node_pool" "test" {

…own" This reverts commit 504d4a7.

ms-henglu · 2024-06-29T06:06:49Z

Hi @stephybun - I also confirmed with the service team that this feature flag is required. I've reverted the previous commit which removes the notice.

stephybun

Thanks @ms-henglu LGTM 👍

<Actions> <action id="f410411e63aff4bb73a81c2aec1d373cf8a903e63b30dee2006b0030d8a94cc8"> <h3>Bump Terraform `azurerm` provider version</h3> <details id="1d9343c012f5434ac9fe8a98135bae3667b399259be16d9b14302ea3bd424a24"> <summary>Update Terraform lock file</summary> <p>changes detected:
	"hashicorp/azurerm" updated from "3.110.0" to "3.111.0" in file ".terraform.lock.hcl"</p> <details> <summary>3.111.0</summary> <pre>Changelog retrieved from:
	https://github.com/hashicorp/terraform-provider-azurerm/releases/tag/v3.111.0
FEATURES:

* **New Resource:** `azurerm_restore_point_collection` ([#26518](https://github.com/hashicorp/terraform-provider-azurerm/issues/26518))

ENHANCEMENTS:

* dependencies: updating to `v0.20240701.1082110` of `github.com/hashicorp/go-azure-sdk` ([#26502](hashicorp/terraform-provider-azurerm#26502 `azurerm_disk_encryption_set` - support for the `managed_hsm_key_id` property ([#26201](hashicorp/terraform-provider-azurerm#26201 `azurerm_firewall_policy` - remove Computed from the `sku` property and add a default of `Standard` in 4.0 ([#26499](hashicorp/terraform-provider-azurerm#26499 `azurerm_kubernetes_cluster` - support updating `default_node_pool.os_sku` between `Ubuntu` and `AzureLinux` ([#26262](hashicorp/terraform-provider-azurerm#26262 `azurerm_kubernetes_cluster_node_pool` - support updating `os_sku` between `Ubuntu` and `AzureLinux` ([#26139](hashicorp/terraform-provider-azurerm#26139 `azurerm_service_plan` - support for new the Flex Consumption plan ([#26351](https://github.com/hashicorp/terraform-provider-azurerm/issues/26351))

BUG FIXES:

* `azurerm_kubernetes_cluster` - prevent a panic ([#26478](hashicorp/terraform-provider-azurerm#26478 `azurerm_kubernetes_cluster` - prevent a diff in `upgrade_settings` when the API returns an empty object ([#26541](hashicorp/terraform-provider-azurerm#26541 `azurerm_kubernetes_cluster_node_pool` - prevent a diff in `upgrade_settings` when the API returns an empty object ([#26541](hashicorp/terraform-provider-azurerm#26541 `azurerm_virtual_network_gateway` - split create and update function to fix lifecycle - ignore ([#26451](hashicorp/terraform-provider-azurerm#26451 `azurerm_virtual_network_gateway_connection` - split create and update function to fix lifecycle - ignore ([#26431](https://github.com/hashicorp/terraform-provider-azurerm/issues/26431))


</pre> </details> </details> <a href="https://infra.ci.jenkins.io/job/updatecli/job/azure/job/main/306/">Jenkins pipeline link</a> </action> </Actions> --- <table> <tr> <td width="77"> <img src="https://www.updatecli.io/images/updatecli.png" alt="Updatecli logo" width="50" height="50"> </td> <td> <p> Created automatically by <a href="https://www.updatecli.io/">Updatecli</a> </p> <details><summary>Options:</summary> <br /> <p>Most of Updatecli configuration is done via <a href="https://www.updatecli.io/docs/prologue/quick-start/">its manifest(s)</a>.</p> <ul> <li>If you close this pull request, Updatecli will automatically reopen it, the next time it runs.</li> <li>If you close this pull request and delete the base branch, Updatecli will automatically recreate it, erasing all previous commits made.</li> </ul> <p> Feel free to report any issues at <a href="https://github.com/updatecli/updatecli/issues">github.com/updatecli/updatecli</a>.<br /> If you find this tool useful, do not hesitate to star <a href="https://github.com/updatecli/updatecli/stargazers">our GitHub repository</a> as a sign of appreciation, and/or to tell us directly on our <a href="https://matrix.to/#/#Updatecli_community:gitter.im">chat</a>! </p> </details> </td> </tr> </table> Co-authored-by: Jenkins Infra Bot (updatecli) <[email protected]>

github-actions · 2024-08-03T02:08:30Z

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

azurerm_kubernetes_cluster_node_pool - os_sku supports update

8a9713a

github-actions bot added documentation size/S labels May 29, 2024

stephybun added upstream/microsoft Indicates that there's an upstream issue blocking this issue/PR preview blocked labels May 29, 2024

stephybun reviewed May 29, 2024

View reviewed changes

github-actions bot added the waiting-response label May 29, 2024

ms-henglu closed this May 29, 2024

github-actions bot removed the waiting-response label May 29, 2024

ms-henglu reopened this Jun 4, 2024

Merge branch 'main' into ticket-28167061-aks-pool-supports-update-ossku

d51e253

ms-henglu marked this pull request as ready for review June 4, 2024 02:09

github-actions bot added the waiting-response label Jun 4, 2024

make fmt

e0a2f95

github-actions bot removed the waiting-response label Jun 4, 2024

stephybun requested changes Jun 5, 2024

View reviewed changes

stephybun added enhancement service/kubernetes-cluster and removed upstream/microsoft Indicates that there's an upstream issue blocking this issue/PR preview blocked labels Jun 5, 2024

github-actions bot added the waiting-response label Jun 5, 2024

Update website/docs/r/kubernetes_cluster_node_pool.html.markdown

504d4a7

Co-authored-by: stephybun <[email protected]>

github-actions bot removed the waiting-response label Jun 5, 2024

stephybun self-assigned this Jun 7, 2024

hbeberman suggested changes Jun 24, 2024

View reviewed changes

update

13e34bf

hbeberman approved these changes Jun 25, 2024

View reviewed changes

Revert "Update website/docs/r/kubernetes_cluster_node_pool.html.markd…

82a4d8a

…own" This reverts commit 504d4a7.

ms-henglu requested a review from stephybun June 29, 2024 06:06

stephybun approved these changes Jul 2, 2024

View reviewed changes

stephybun merged commit 6ef7041 into hashicorp:main Jul 2, 2024
34 checks passed

github-actions bot added this to the v3.111.0 milestone Jul 2, 2024

stephybun added a commit that referenced this pull request Jul 4, 2024

Update for #26139 #26262 #26541

57d98af

github-actions bot locked as resolved and limited conversation to collaborators Aug 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`azurerm_kubernetes_cluster_node_pool` - `os_sku` supports update #26139

`azurerm_kubernetes_cluster_node_pool` - `os_sku` supports update #26139

ms-henglu commented May 29, 2024

stephybun left a comment

ms-henglu commented May 29, 2024

stephybun Jun 5, 2024

ms-henglu Jun 5, 2024

stephybun Jun 5, 2024

ms-henglu Jun 5, 2024

hbeberman left a comment

hbeberman Jun 24, 2024

hbeberman Jun 24, 2024

hbeberman Jun 24, 2024

hbeberman Jun 24, 2024

ms-henglu Jun 25, 2024

ms-henglu commented Jun 25, 2024

ms-henglu commented Jun 29, 2024

ms-henglu commented Jun 29, 2024

stephybun left a comment

github-actions bot commented Aug 3, 2024

	if old != agentpools.OSSKUUbuntu && old != agentpools.OSSKUAzureLinux {
	if old != string(agentpools.OSSKUUbuntu) && old != string(agentpools.OSSKUAzureLinux) {

	if new != agentpools.OSSKUUbuntu && new != agentpools.OSSKUAzureLinux {
	if new != string(agentpools.OSSKUUbuntu) && new != string(agentpools.OSSKUAzureLinux) {

azurerm_kubernetes_cluster_node_pool - os_sku supports update #26139

azurerm_kubernetes_cluster_node_pool - os_sku supports update #26139

Conversation

ms-henglu commented May 29, 2024

Community Note

Description

PR Checklist

Changes to existing Resource / Data Source

Testing

Change Log

Related Issue(s)

stephybun left a comment

Choose a reason for hiding this comment

ms-henglu commented May 29, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hbeberman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ms-henglu commented Jun 25, 2024

ms-henglu commented Jun 29, 2024

ms-henglu commented Jun 29, 2024

stephybun left a comment

Choose a reason for hiding this comment

github-actions bot commented Aug 3, 2024

`azurerm_kubernetes_cluster_node_pool` - `os_sku` supports update #26139

`azurerm_kubernetes_cluster_node_pool` - `os_sku` supports update #26139