azurerm backend ignores metadata_host value for custom cloud environments

[daveinci]

Terraform Version

2024-04-29T10:17:54.072-0500 [INFO]  Terraform version: 1.8.2
2024-04-29T10:17:54.072-0500 [DEBUG] using github.com/hashicorp/go-tfe v1.51.0
2024-04-29T10:17:54.072-0500 [DEBUG] using github.com/hashicorp/hcl/v2 v2.20.0
2024-04-29T10:17:54.072-0500 [DEBUG] using github.com/hashicorp/terraform-svchost v0.1.1
2024-04-29T10:17:54.072-0500 [DEBUG] using github.com/zclconf/go-cty v1.14.3
2024-04-29T10:17:54.072-0500 [INFO]  Go runtime version: go1.22.1
...
Terraform v1.8.2
on linux_amd64
+ provider registry.terraform.io/hashicorp/azurerm v3.101.0
+ provider registry.terraform.io/hashicorp/null v3.2.2
+ provider registry.terraform.io/hashicorp/random v3.6.1

Terraform Configuration Files

terraform {
  required_version = ">= 0.14"
  backend "azurerm" {
    resource_group_name  = "rg-tfstate"  # Can be passed via `-backend-config=`"resource_group_name=<resource group name>"` in the `init` command.
    storage_account_name = "<storag-account-name"                      # Can be passed via `-backend-config=`"storage_account_name=<storage account name>"` in the `init` command.
    container_name       = "tfstate"                       # Can be passed via `-backend-config=`"container_name=<container name>"` in the `init` command.
    key                  = "prod.terraform.tfstate"        # Can be passed via `-backend-config=`"key=<blob key name>"` in the `init` command.
    use_azuread_auth     = true                            # Can also be set via `ARM_USE_AZUREAD` environment variable.
    metadata_host="management.usgovcloudapi.net"
  }  
} 

provider "azurerm" {
  metadata_host="management.usgovcloudapi.net"
  skip_provider_registration = true
  features {}
}

Debug Output

https://gist.github.com/daveinci/de6cc4e24c4fe6f2998e3ce6ccf9ce8e

Expected Behavior

Terraform should be utilizing the endpoints associated to the metadata_host value which indicates a custom cloud environment is being used. In this case, should be using login.microsoftonline.us and blob.core.usgovcloudapi.net

Actual Behavior

Terraform defaulted to the public endpoints for login and storage, ignoring the metadata_host value. This is similar to behavior seen in the azurerm and azuread providers that @manicminer fixed in v3.99.0

hashicorp/terraform-provider-azurerm#25546
hashicorp/terraform-provider-azuread#1353

Steps to Reproduce

az cloud show -n AzureUSGovernment > myCustomCloud.json

## Edit myCustomCloud.json replacing the name:
Before: "name": "AzureUSGovernment"
After: "name": "myCustomCloud"

az cloud register -n MyCustomCloud --cloud-config @<path to file>/myCustomCloud.json
az cloud set -n MyCustomCloud
az login <--use-device-code>

Configure azurerm provider and backend block for azure using metadata_host="management.usgovcloudapi.net" as above
terraform init

Additional Context

No response

References

hashicorp/terraform-provider-azurerm#25546
hashicorp/terraform-provider-azuread#1353

[crw]

Thanks for this submission! Changes to the AzureRM backend are managed by the AzureRM Provider maintainer team, who have been alerted.

If you are viewing this issue and would like to indicate your interest, please use the 👍 reaction on the issue description to upvote this issue. We also welcome additional use case descriptions. Thanks again!

[manicminer]

@daveinci Thanks for reporting this. The Azure backend needs to be updated to the latest SDKs to take advantage of various bugfixes including those for custom clouds. I'm working on this at the moment and this should be fixed in an upcoming minor Terraform release.

[daveinci]

Thanks, @manicminer. Any update on which release we can expect this in?

[jazcto86]

@daveinci would this also resolve the backend being unable to find "ussec" as an environment? At this point we are able to terraform resources in an USSec (microsoft.scloud) environment without any issues, except for being unable to set a backend for our states in an storage account within this sub... We've tried all methods but the backend always throws an "environment not found: ussec" when running an init... it's as if the backend expects only whatever cloud environments it currently has hardcoded (public, china, germany (I believe), azusgovernment)....

Would love to know if this SDK update would also cover adding ussec to the list of envs!

[manicminer]

@jazcto86 Yes it would resolve that too. The backend is currently using a now-outdated SDK and, once updated/refactored, it will support non-public clouds just as the provider now does (i.e. via the metadata_host property).

At that time - and also currently for both AzureRM and AzureAD providers - you only need to specify the metadata_host property / ARM_METADATA_HOSTNAME environment variable. The environment property / ARM_ENVIRONMENT env var is ignored, so you can remove that from your config when using a custom metadata service.

[CoreyCook8]

👋 @manicminer Is there any update on this issue? Has this been fixed?

[crw]

No updates, as far as I know it has not been fixed (it would be linked to a pull request that fixes the issue in that case).