Update go-azure-sdk for latest metadata fixes, and configure resource identifier for storage data plane clients at runtime #25546

manicminer · 2024-04-09T14:28:40Z

Community Note

Please vote on this PR by adding a 👍 reaction to the original PR to help the community and maintainers prioritize for review
Please do not leave "+1" or "me too" comments, they generate extra noise for PR followers and do not help prioritize for review

Description

storage: configure resource identifier for storage data plane clients at runtime, so that tokens are scoped to a particular storage account in the configured cloud

Caution

This PR depends on hashicorp/go-azure-sdk#960 and needs rebasing before merging.

PR Checklist

I have followed the guidelines in our Contributing Documentation.
I have checked to ensure there aren't other open Pull Requests for the same update/change.
I have checked if my changes close any open issues. If so please include appropriate closing keywords below.
I have updated/added Documentation as required written in a helpful and kind way to assist users that may be unfamiliar with the resource / data source.
I have used a meaningful PR title to help maintainers and other users understand this change and help prevent duplicate work.
For example: “resource_name_here - description of change e.g. adding property new_property_name_here”

Changes to existing Resource / Data Source

I have added an explanation of what my changes do and why I'd like you to include them (This may be covered by linking to an issue above, but may benefit from additional explanation).
I have written new tests for my resource or datasource changes & updated any relevent documentation.
I have successfully run tests with my changes locally. If not, please provide details on testing challenges that prevented you running the tests.
(For changes that include a state migration only). I have manually tested the migration path between relevant versions of the provider.

Testing

My submission includes Test coverage as described in the Contribution Guide and the tests pass. (if this is not possible for any reason, please include details of why you did or could not add test coverage)

Change Log

Below please provide what should go into the changelog (if anything) conforming to the Changelog Format documented here.

ENHANCEMENTS:

dependencies: updating to v0.20240411.1104331 of github.com/hashicorp/go-azure-sdk/resourcemanager and github.com/hashicorp/go-azure-sdk/sdk

BUG FIXES:

provider: fix an issue where the provider was not correctly configured when using a custom metadata host

This is a (please select all that apply):

Bug Fix
New Feature (ie adding a service, resource, or data source)
Enhancement
Breaking Change

Related Issue(s)

N/A

Note

If this PR changes meaningfully during the course of review please update the title and description as required.

manicminer · 2024-04-10T20:27:03Z

Deprecated Usage check can be ignored due to false positive on internal/acceptance/testclient/client.go

tombuildsstuff

Left a few minor comments but this otherwise LGTM 👍

tombuildsstuff · 2024-04-11T08:14:32Z

website/docs/index.html.markdown

@@ -107,7 +107,7 @@ The following arguments are supported:

 * `client_id_file_path` (Optional) The path to a file containing the Client ID which should be used. This can also be sourced from the `ARM_CLIENT_ID_FILE_PATH` Environment Variable.

-* `environment` - (Optional) The Cloud Environment which should be used. Possible values are `public`, `usgovernment`, `german`, and `china`. Defaults to `public`. This can also be sourced from the `ARM_ENVIRONMENT` Environment Variable.
+* `environment` - (Optional) The Cloud Environment which should be used. Possible values are `public`, `usgovernment`, `german`, and `china`. Defaults to `public`. This can also be sourced from the `ARM_ENVIRONMENT` Environment Variable. Not used when `metadata_host` is specified.


Suggested change

* `environment` - (Optional) The Cloud Environment which should be used. Possible values are `public`, `usgovernment`, `german`, and `china`. Defaults to `public`. This can also be sourced from the `ARM_ENVIRONMENT` Environment Variable. Not used when `metadata_host` is specified.

* `environment` - (Optional) The Cloud Environment which should be used. Possible values are `public`, `usgovernment`, `german`, and `china`. Defaults to `public`. This can also be sourced from the `ARM_ENVIRONMENT` Environment Variable. This is not used and should not be specified when `metadata_host` is specified.

tombuildsstuff · 2024-04-11T08:15:30Z

internal/provider/provider.go

@@ -207,7 +207,7 @@ func azureProvider(supportLegacyTestSuite bool) *schema.Provider {
 				Type:        schema.TypeString,
 				Required:    true,
 				DefaultFunc: schema.EnvDefaultFunc("ARM_ENVIRONMENT", "public"),
-				Description: "The Cloud Environment which should be used. Possible values are public, usgovernment, and china. Defaults to public.",
+				Description: "The Cloud Environment which should be used. Possible values are public, usgovernment, and china. Defaults to public. Not used when `metadata_host` is specified.",


Suggested change

Description: "The Cloud Environment which should be used. Possible values are public, usgovernment, and china. Defaults to public. Not used when `metadata_host` is specified.",

Description: "The Cloud Environment which should be used. Possible values are public, usgovernment, and china. Defaults to public. This is not used and should not be specified when `metadata_host` is specified.",

tombuildsstuff · 2024-04-11T08:17:13Z

internal/services/storage/client/client.go

@@ -34,7 +34,7 @@ type Client struct {
 	BlobServicesClient *storage.BlobServicesClient
 	FileServicesClient *storage.FileServicesClient

-	authorizerForAad auth.Authorizer
+	authConfig *auth.Credentials


I think it'd be worth calling this out as AzureAD/Entra specific credentials, since there's different authentication mechanisms here and auth.Credentials is unspecific?

Suggested change

authConfig *auth.Credentials

azureADAuthConfig *auth.Credentials

tombuildsstuff · 2024-04-11T08:17:46Z

internal/services/storage/client/data_plane.go

-		baseClient.SetAuthorizer(c.authorizerForAad)
+func (c Client) configureDataPlane(ctx context.Context, clientName, resourceIdentifier string, baseClient client.BaseClient, account accountDetails, operation DataPlaneOperation) error {
+	if operation.SupportsAadAuthentication && c.authConfig != nil {
+		api := c.authConfig.Environment.Storage.WithResourceIdentifier(resourceIdentifier)


I suspect this might be problematic since we're mutating the original Storage object - see hashicorp/go-azure-sdk#960 (comment)

…orp/go-azure-sdk/resourcemanager` and `github.com/hashicorp/go-azure-sdk/sdk`

…t name

…t tokens are scoped to a particular storage account in the configured cloud

… plane client

…MinimumTLSVersion`, as it fails with custom/private clouds and has no effect in built-in clouds

…k `environments.Environment`

…dability

manicminer · 2024-04-11T11:05:37Z

Tested locally and confirmed that

Storage Accounts and containers
Key Vaults and keys
Application Registrations (AzureAD)

all are working using AAD authentication, with prebuilt cloud environment config for public, and configuring entirely from metadata at https://management.azure.com/metadata/endpoints?api-version=2022-09-01.

<Actions> <action id="f410411e63aff4bb73a81c2aec1d373cf8a903e63b30dee2006b0030d8a94cc8"> <h3>Bump Terraform `azurerm` provider version</h3> <details id="1d9343c012f5434ac9fe8a98135bae3667b399259be16d9b14302ea3bd424a24"> <summary>Update Terraform lock file</summary> <p>changes detected:
	"hashicorp/azurerm" updated from "3.98.0" to "3.99.0" in file ".terraform.lock.hcl"</p> <details> <summary>3.99.0</summary> <pre>Changelog retrieved from:
	https://github.com/hashicorp/terraform-provider-azurerm/releases/tag/v3.99.0
BREAKING CHANGE: 

* `azurerm_linux_web_app` - `site_config.0.application_stack.0.java_version` must be specified with `java_server` and `java_server_version` ([#25553](https://github.com/hashicorp/terraform-provider-azurerm/issues/25553))

ENHANCEMENTS:

* dependencies: updating to `v0.20240411.1104331` of `github.com/hashicorp/go-azure-sdk/resourcemanager` and `github.com/hashicorp/go-azure-sdk/sdk` ([#25546](hashicorp/terraform-provider-azurerm#25546 dependencies: updating to `v0.26.1` of `github.com/tombuildsstuff/giovanni` ([#25551](hashicorp/terraform-provider-azurerm#25551 `azurerm_key_vault` - deprecate the `contact` property from v3.x provider and update properties to Computed & Optional ([#25552](hashicorp/terraform-provider-azurerm#25552 `azurerm_key_vault_certificate_contacts` - in v4.0 make the `contact` property optional to allow for deletion of contacts from the key vault ([#25552](hashicorp/terraform-provider-azurerm#25552 `azurerm_signalr_service` - support for setting the `sku` property to `Premium_P2` ([#25578](hashicorp/terraform-provider-azurerm#25578 `azurerm_snapshot` - support for the `network_access_policy` and `public_network_access_enabled` properties ([#25421](hashicorp/terraform-provider-azurerm#25421 `azurerm_storage_account` - extend the support level of `(blob|queue|share)_properties` for Storage kind ([#25427](hashicorp/terraform-provider-azurerm#25427 `azurerm_storage_blob` - support for the `encryption_scope` property ([#25551](hashicorp/terraform-provider-azurerm#25551 `azurerm_storage_container` - support for the `default_encryption_scope` and `encryption_scope_override_enabled` properties ([#25551](hashicorp/terraform-provider-azurerm#25551 `azurerm_storage_data_lake_gen2_filesystem` - support for the `default_encryption_scope` property ([#25551](hashicorp/terraform-provider-azurerm#25551 `azurerm_subnet` - the `delegation.x.service_delegation.x.name` property now supports `Oracle.Database/networkAttachments` ([#25571](hashicorp/terraform-provider-azurerm#25571 `azurerm_web_pubsub` - support setting the `sku` property to `Premium_P2` ([#25578](https://github.com/hashicorp/terraform-provider-azurerm/issues/25578))

BUG FIXES:

* provider: fix an issue where the provider was not correctly configured when using a custom metadata host ([#25546](hashicorp/terraform-provider-azurerm#25546 storage: fix a number of potential crashes during plan/apply with resources using the Storage data plane API ([#25525](hashicorp/terraform-provider-azurerm#25525 `azurerm_application_insights` - fix issue where the wrong Application ID was set into the property `app_id` ([#25520](hashicorp/terraform-provider-azurerm#25520 `azurerm_application_insights_api_key` - add a state migration to re-case static segments of the resource ID ([#25567](hashicorp/terraform-provider-azurerm#25567 `azurerm_container_app_environment_certificate` - the `subject_name` attribute is now correctly populated ([#25516](hashicorp/terraform-provider-azurerm#25516 `azurerm_function_app_slot` - will now taint the resource when partially created ([#24520](hashicorp/terraform-provider-azurerm#24520 `azurerm_linux_function_app` - will now taint the resource when partially created ([#24520](hashicorp/terraform-provider-azurerm#24520 `azurerm_managed_disk` - filtering the Resource SKUs response to reduce the memory overhead, when determining whether a Managed Disk can be online resized or not ([#25549](hashicorp/terraform-provider-azurerm#25549 `azurerm_monitor_alert_prometheus_rule_group` - the `severity` property is now set correctly when `0` ([#25408](hashicorp/terraform-provider-azurerm#25408 `azurerm_monitor_smart_detector_alert_rule` - normalising the value for `id` within the `action_group` block ([#25559](hashicorp/terraform-provider-azurerm#25559 `azurerm_redis_cache_access_policy_assignment` - the `object_id_alias` property now allows usernames ([#25523](hashicorp/terraform-provider-azurerm#25523 `azurerm_windows_function_app` - will not taint the resource when partially created ([#24520](hashicorp/terraform-provider-azurerm#24520 `azurerm_windows_function_app` - will not taint the resource when partially created ([#24520](https://github.com/hashicorp/terraform-provider-azurerm/issues/24520))

DEPRECATIONS:

* `azurerm_cosmosdb_account` - the `connection_strings` property has been superseded by the primary and secondary connection strings for sql, mongodb and readonly ([#25510](hashicorp/terraform-provider-azurerm#25510 `azurerm_cosmosdb_account` - the `enable_free_tier` property has been superseded by `free_tier_enabled` ([#25510](hashicorp/terraform-provider-azurerm#25510 `azurerm_cosmosdb_account` - the `enable_multiple_write_locations` property has been superseded by `multiple_write_locations_enabled` ([#25510](hashicorp/terraform-provider-azurerm#25510 `azurerm_cosmosdb_account` - the `enable_automatic_failover` property has been superseded by `automatic_failover_enabled` ([#25510](https://github.com/hashicorp/terraform-provider-azurerm/issues/25510))


</pre> </details> </details> <a href="https://infra.ci.jenkins.io/job/updatecli/job/azure/job/main/105/">Jenkins pipeline link</a> </action> </Actions> --- <table> <tr> <td width="77"> <img src="https://www.updatecli.io/images/updatecli.png" alt="Updatecli logo" width="50" height="50"> </td> <td> <p> Created automatically by <a href="https://www.updatecli.io/">Updatecli</a> </p> <details><summary>Options:</summary> <br /> <p>Most of Updatecli configuration is done via <a href="https://www.updatecli.io/docs/prologue/quick-start/">its manifest(s)</a>.</p> <ul> <li>If you close this pull request, Updatecli will automatically reopen it, the next time it runs.</li> <li>If you close this pull request and delete the base branch, Updatecli will automatically recreate it, erasing all previous commits made.</li> </ul> <p> Feel free to report any issues at <a href="https://github.com/updatecli/updatecli/issues">github.com/updatecli/updatecli</a>.<br /> If you find this tool useful, do not hesitate to star <a href="https://github.com/updatecli/updatecli/stargazers">our GitHub repository</a> as a sign of appreciation, and/or to tell us directly on our <a href="https://matrix.to/#/#Updatecli_community:gitter.im">chat</a>! </p> </details> </td> </tr> </table> Co-authored-by: Jenkins Infra Bot (updatecli) <[email protected]>

github-actions · 2024-05-12T02:10:33Z

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

manicminer mentioned this pull request Apr 9, 2024

Metadata fixes hashicorp/go-azure-sdk#960

Merged

github-actions bot added dependencies service/storage size/XL labels Apr 9, 2024

manicminer added bug authentication provider and removed dependencies service/storage size/XL labels Apr 9, 2024

github-actions bot added waiting-response dependencies service/storage service/attestation size/XL and removed waiting-response labels Apr 9, 2024

manicminer force-pushed the metadata-fixes branch from 9d8a7a5 to eb93d1c Compare April 10, 2024 18:56

github-actions bot added documentation waiting-response and removed waiting-response labels Apr 10, 2024

manicminer force-pushed the metadata-fixes branch from 9b2d738 to d948150 Compare April 10, 2024 19:36

github-actions bot added waiting-response and removed waiting-response labels Apr 10, 2024

manicminer marked this pull request as ready for review April 10, 2024 20:14

manicminer requested a review from a team April 10, 2024 20:20

github-actions bot removed the waiting-response label Apr 10, 2024

tombuildsstuff approved these changes Apr 11, 2024

View reviewed changes

dependencies: updating to v0.20240411.1104331 of `github.com/hashic…

1c4f370

…orp/go-azure-sdk/resourcemanager` and `github.com/hashicorp/go-azure-sdk/sdk`

manicminer added 7 commits April 11, 2024 11:54

provider: environments.FromEndpoint() no longer needs an environmen…

4517fed

…t name

storage: configure resource identifier for storage at runtime, so tha…

0ef3af0

…t tokens are scoped to a particular storage account in the configured cloud

attestation: use domain suffix from environment when configuring data…

325ff76

… plane client

storage: remove legacy config logic for AllowBlobPublicAccess and `…

03072ba

…MinimumTLSVersion`, as it fails with custom/private clouds and has no effect in built-in clouds

containers: update use of autorest azure.Environment to go-azure-sd…

c088cef

…k `environments.Environment`

provider: log the metdata configuration mode, tidy up provider tests

f8849bc

storage: rename authConfig to authConfigForAzureAD for better rea…

db61b1a

…dability

manicminer force-pushed the metadata-fixes branch from d948150 to db61b1a Compare April 11, 2024 10:54

github-actions bot added the waiting-response label Apr 11, 2024

manicminer merged commit aade0fd into main Apr 11, 2024
31 of 32 checks passed

manicminer deleted the metadata-fixes branch April 11, 2024 11:05

github-actions bot removed the waiting-response label Apr 11, 2024

github-actions bot added this to the v3.99.0 milestone Apr 11, 2024

manicminer added a commit that referenced this pull request Apr 11, 2024

Changelog for #25546

8d9c8db

crw mentioned this pull request Apr 29, 2024

azurerm backend ignores metadata_host value for custom cloud environments hashicorp/terraform#35095

Open

github-actions bot locked as resolved and limited conversation to collaborators May 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update go-azure-sdk for latest metadata fixes, and configure resource identifier for storage data plane clients at runtime #25546

Update go-azure-sdk for latest metadata fixes, and configure resource identifier for storage data plane clients at runtime #25546

manicminer commented Apr 9, 2024 •

edited

Loading

manicminer commented Apr 10, 2024

tombuildsstuff left a comment

tombuildsstuff Apr 11, 2024

tombuildsstuff Apr 11, 2024

tombuildsstuff Apr 11, 2024

tombuildsstuff Apr 11, 2024

manicminer commented Apr 11, 2024

github-actions bot commented May 12, 2024

	* `environment` - (Optional) The Cloud Environment which should be used. Possible values are `public`, `usgovernment`, `german`, and `china`. Defaults to `public`. This can also be sourced from the `ARM_ENVIRONMENT` Environment Variable. Not used when `metadata_host` is specified.
	* `environment` - (Optional) The Cloud Environment which should be used. Possible values are `public`, `usgovernment`, `german`, and `china`. Defaults to `public`. This can also be sourced from the `ARM_ENVIRONMENT` Environment Variable. This is not used and should not be specified when `metadata_host` is specified.

	Description: "The Cloud Environment which should be used. Possible values are public, usgovernment, and china. Defaults to public. Not used when `metadata_host` is specified.",
	Description: "The Cloud Environment which should be used. Possible values are public, usgovernment, and china. Defaults to public. This is not used and should not be specified when `metadata_host` is specified.",

	authConfig *auth.Credentials
	azureADAuthConfig *auth.Credentials

Update go-azure-sdk for latest metadata fixes, and configure resource identifier for storage data plane clients at runtime #25546

Update go-azure-sdk for latest metadata fixes, and configure resource identifier for storage data plane clients at runtime #25546

Conversation

manicminer commented Apr 9, 2024 • edited Loading

Community Note

Description

PR Checklist

Changes to existing Resource / Data Source

Testing

Change Log

Related Issue(s)

manicminer commented Apr 10, 2024

tombuildsstuff left a comment

Choose a reason for hiding this comment

tombuildsstuff Apr 11, 2024

Choose a reason for hiding this comment

tombuildsstuff Apr 11, 2024

Choose a reason for hiding this comment

tombuildsstuff Apr 11, 2024

Choose a reason for hiding this comment

tombuildsstuff Apr 11, 2024

Choose a reason for hiding this comment

manicminer commented Apr 11, 2024

github-actions bot commented May 12, 2024

manicminer commented Apr 9, 2024 •

edited

Loading