data source `azurerm_resources` can no longer query the existence of resource group #24154

herquan-docusign · 2023-12-07T21:58:48Z

Is there an existing issue for this?

I have searched the existing issues

Community Note

Since this change, we no longer put resource group in the filter but we always assume resource group should exist if it is specified. This data source is supposed "to access information about existing resources." and I think whether resource group exists or not should also be part of the job of this data source.
If we do want to assume resource group always exists, we should also provide a data source to check if a resource group exists or not. It could be named azurerm_resource_groups

Terraform Version

1.5.5

AzureRM Provider Version

3.56.0

Affected Resource(s)/Data Source(s)

azurerm_resources

Terraform Configuration Files

data "azurerm_resources" "example" {
  name = "someresource"
  resource_group_name = "a_non_exist_resource_group"
}

Debug Output/Panic Output

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: getting resources by resource group: resources.Client#ListByResourceGroup: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceGroupNotFound" Message="Resource group 'a_non_exist_resource_group' could not be found."
│
│   with data.azurerm_resources.example,
│   on main.tf line 28, in data "azurerm_resources" "example":
│   28: data "azurerm_resources" "example" {
│
╵

Expected Behaviour

Data source should return 
{
      + id                  = "resource-db0937ac-5c9a-442e-919c-1d8c8efbebdf"
      + name                = "someresource"
      + required_tags       = null
      + resource_group_name = "a_non_exist_resource_group"
      + resources           = []
      + timeouts            = null
      + type                = null
    }

Actual Behaviour

No response

Steps to Reproduce

No response

Important Factoids

No response

References

No response

The text was updated successfully, but these errors were encountered:

wuxu92 · 2023-12-11T02:04:15Z

hi @herquan-docusign, could you please make sure that you are using the coccret authentication and the right subscription? This is not a scenario I can reproduce locally. If the subscription is right, could you please grab and share the debug log from terraform with command: TF_LOG=debug terraform apply?

herquan-docusign · 2023-12-11T05:18:24Z

@wuxu92 there are too much information dumped and it may contain sensitive information. What do you need to check?
Here is the full source code

provider "azurerm" {
  features {
  }
  subscription_id = "someid"
  tenant_id       = "someidtoo"
}

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "3.55.0" # or "3.56.0"
    }
  }
}

data "azurerm_resources" "example" {
  name = "someresource"
  resource_group_name = "a_non_exist_resource_group"
}

You can see the difference between version 3.55.0 and 3.56.0

# 3.55.0
M:\a>terraform init -upgrade
2023-12-10T21:12:56.599-0800 [INFO]  Terraform version: 1.5.5
2023-12-10T21:12:56.600-0800 [DEBUG] using github.com/hashicorp/go-tfe v1.26.0
2023-12-10T21:12:56.600-0800 [DEBUG] using github.com/hashicorp/hcl/v2 v2.16.2
2023-12-10T21:12:56.600-0800 [DEBUG] using github.com/hashicorp/terraform-svchost v0.1.0
2023-12-10T21:12:56.600-0800 [DEBUG] using github.com/zclconf/go-cty v1.12.2
2023-12-10T21:12:56.600-0800 [INFO]  Go runtime version: go1.20.7
2023-12-10T21:12:56.600-0800 [INFO]  CLI args: []string{"terraform", "init", "-upgrade"}
2023-12-10T21:12:56.605-0800 [DEBUG] Attempting to open CLI config file: somepath\terraform.rc
2023-12-10T21:12:56.605-0800 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2023-12-10T21:12:56.605-0800 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2023-12-10T21:12:56.605-0800 [DEBUG] ignoring non-existing provider search directory somepath\terraform.d\plugins
2023-12-10T21:12:56.606-0800 [DEBUG] ignoring non-existing provider search directory somepath\HashiCorp\Terraform\plugins
2023-12-10T21:12:56.606-0800 [INFO]  CLI command args: []string{"init", "-upgrade"}

Initializing the backend...
2023-12-10T21:12:56.608-0800 [DEBUG] New state was assigned lineage "some id"
2023-12-10T21:12:57.028-0800 [DEBUG] checking for provisioner in "."
2023-12-10T21:12:57.028-0800 [DEBUG] checking for provisioner in "my local path"

Initializing provider plugins...
- Finding hashicorp/azurerm versions matching "3.55.0"...
2023-12-10T21:12:57.034-0800 [DEBUG] Service discovery for registry.terraform.io at https://registry.terraform.io/.well-known/terraform.json
2023-12-10T21:12:57.169-0800 [DEBUG] GET https://registry.terraform.io/v1/providers/hashicorp/azurerm/versions
- Using previously-installed hashicorp/azurerm v3.55.0

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

M:\a>rem terraform plan

M:\a>set TF_LOG=

M:\a>terraform apply -auto-approve
data.azurerm_resources.example: Reading...
data.azurerm_resources.example: Read complete after 1s [id=resource-000000000000000000]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

# 3.56.0
M:\a>terraform init -upgrade

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/azurerm versions matching "3.56.0"...
- Installing hashicorp/azurerm v3.56.0...
- Installed hashicorp/azurerm v3.56.0 (signed by HashiCorp)

Terraform has made some changes to the provider dependency selections recorded
in the .terraform.lock.hcl file. Review those changes and commit them to your
version control system if they represent changes you intended to make.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

M:\a>rem terraform plan

M:\a>set TF_LOG=

M:\a>terraform apply -auto-approve
data.azurerm_resources.example: Reading...
╷
│ Error: getting resources by resource group: resources.Client#ListByResourceGroup: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceGroupNotFound" Message="Resource group 'a_non_exist_resource_group' could not be found."
│
│   with data.azurerm_resources.example,
│   on main.tf line 17, in data "azurerm_resources" "example":
│   17: data "azurerm_resources" "example" {
│
╵

# 3.84.0
M:\a>terraform init -upgrade

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/azurerm versions matching "3.84.0"...
- Installing hashicorp/azurerm v3.84.0...
- Installed hashicorp/azurerm v3.84.0 (signed by HashiCorp)

Terraform has made some changes to the provider dependency selections recorded
in the .terraform.lock.hcl file. Review those changes and commit them to your
version control system if they represent changes you intended to make.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

M:\a>rem terraform plan

M:\a>set TF_LOG=

M:\a>terraform apply -auto-approve
data.azurerm_resources.example: Reading...
╷
│ Error: getting resources by resource group: resources.Client#ListByResourceGroup: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceGroupNotFound" Message="Resource group 'a_non_exist_resource_group' could not be found."
│
│   with data.azurerm_resources.example,
│   on main.tf line 17, in data "azurerm_resources" "example":
│   17: data "azurerm_resources" "example" {
│
╵

What do you see from your local environment?

wuxu92 · 2023-12-11T08:31:09Z

@herquan-docusign, I apologize for my previous misunderstanding. It appears that the behavior changed with version 3.56.0, as noted in this pull request: #21661. In my opinion, we should not rely on undefined behavior by querying a resource from a non-existent resource group to check if the resource group exists.

herquan-docusign · 2023-12-11T18:24:23Z

If we want to let the data source azurerm_resources assume the resource group always exists, can we add a backlog item to create a new data source azurerm_resource_groups to query resource groups?

tombuildsstuff · 2023-12-12T09:32:35Z

@wuxu92

In my opinion, we should not rely on undefined behavior by querying a resource from a non-existent resource group to check if the resource group exists.

Indeed, FWIW if a resource_group_name (or any other field) is being specified as a part of the Data Source then it should be assumed to exist, else we should be raising an error (as we're doing) - so whilst the error message isn't great here ultimately the behaviour looks correct (and matches other data sources both within this Provider, but also across other Providers/as expected by Terraform Core).

Since the error message in question should be clearer however, can we check whether the Resource Group exists prior to listing Resources within it? That'd allow us to output an error along the lines of the Resource Group "SomeGroup" was not found) rather than this generic error.

@herquan-docusign

We could look to introduce a new data source to list the Resource Groups (e.g. to cover "find all resource groups with a name matching search-*, or with these tags) - however as with other Data Sources this would need to raise an error should no matching Resource Groups be found - as such would you mind elaborating on your intended use-case for that Data Source?

Thanks!

herquan-docusign · 2023-12-12T21:22:52Z

@tombuildsstuff we do not have any data source to "check whether the Resource Group exists prior to..." but if we all agree to have one we should add a feature request somewhere.
My use case is to check if a resource exists or not where the resource group is not guaranteed to exist.

github-actions bot added the v/3.x label Dec 7, 2023

rcskosir added service/resources new-data-source labels Dec 8, 2023

rcskosir added the waiting-response label Dec 11, 2023

github-actions bot removed the waiting-response label Dec 11, 2023

tombuildsstuff added question waiting-response and removed new-data-source labels Dec 12, 2023

github-actions bot removed the waiting-response label Dec 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

data source `azurerm_resources` can no longer query the existence of resource group #24154

data source `azurerm_resources` can no longer query the existence of resource group #24154

herquan-docusign commented Dec 7, 2023 •

edited

Loading

wuxu92 commented Dec 11, 2023

herquan-docusign commented Dec 11, 2023 •

edited

Loading

wuxu92 commented Dec 11, 2023

herquan-docusign commented Dec 11, 2023

tombuildsstuff commented Dec 12, 2023

herquan-docusign commented Dec 12, 2023

data source azurerm_resources can no longer query the existence of resource group #24154

data source azurerm_resources can no longer query the existence of resource group #24154

Comments

herquan-docusign commented Dec 7, 2023 • edited Loading

Is there an existing issue for this?

Community Note

Terraform Version

AzureRM Provider Version

Affected Resource(s)/Data Source(s)

Terraform Configuration Files

Debug Output/Panic Output

Expected Behaviour

Actual Behaviour

Steps to Reproduce

Important Factoids

References

wuxu92 commented Dec 11, 2023

herquan-docusign commented Dec 11, 2023 • edited Loading

wuxu92 commented Dec 11, 2023

herquan-docusign commented Dec 11, 2023

tombuildsstuff commented Dec 12, 2023

herquan-docusign commented Dec 12, 2023

data source `azurerm_resources` can no longer query the existence of resource group #24154

data source `azurerm_resources` can no longer query the existence of resource group #24154

herquan-docusign commented Dec 7, 2023 •

edited

Loading

herquan-docusign commented Dec 11, 2023 •

edited

Loading