azurerm_monitor_diagnostic_setting - retention_policy deprecated in favor of azurerm_storage_management_policy

[BillCoateARS]

Description

I was trying to do a terraform apply on mlz (plan had no errors) and it eventually failed with the following output (kept a couple of lines before the error for context):

module.firewall.azurerm_storage_account.loganalytics: Creation complete after 20s [id=/subscriptions/067672d1-dedc-404c-97b9-2027c5706580/resourceGroups/arkloudDemo-1568c300e7a7a1ed-arkloud-demo-rg-hub/providers/Microsoft.Storage/storageAccounts/arklouddemohubfirewalllo]
module.firewall.azurerm_monitor_diagnostic_setting.firewall-diagnostics: Creating...
module.firewall.azurerm_monitor_diagnostic_setting.publicip-diagnostics: Creating...
╷
│ Error: creating Monitor Diagnostics Setting "arkloud-demo-hub-firewall-fw-diagnostics" for Resource "/subscriptions/067672d1-dedc-404c-97b9-2027c5706580/resourceGroups/arkloudDemo-1568c300e7a7a1ed-arkloud-demo-rg-hub/providers/Microsoft.Network/azureFirewalls/arkloud-demo-hub-firewall": diagnosticsettings.DiagnosticSettingsClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="BadRequest" Message="Diagnostic settings does not support retention for new diagnostic settings."
│ 
│   with module.firewall.azurerm_monitor_diagnostic_setting.firewall-diagnostics,
│   on ../modules/firewall/main.tf line 90, in resource "azurerm_monitor_diagnostic_setting" "firewall-diagnostics":
│   90: resource "azurerm_monitor_diagnostic_setting" "firewall-diagnostics" {
│ 
╵
╷
│ Error: creating Monitor Diagnostics Setting "arkloud-demo-hub-firewall-client-public-ip-pip-diagnostics" for Resource "/subscriptions/067672d1-dedc-404c-97b9-2027c5706580/resourceGroups/arkloudDemo-1568c300e7a7a1ed-arkloud-demo-rg-hub/providers/Microsoft.Network/publicIPAddresses/arkloud-demo-hub-firewall-client-public-ip": diagnosticsettings.DiagnosticSettingsClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="BadRequest" Message="Diagnostic settings does not support retention for new diagnostic settings."
│ 
│   with module.firewall.azurerm_monitor_diagnostic_setting.publicip-diagnostics,
│   on ../modules/firewall/main.tf line 134, in resource "azurerm_monitor_diagnostic_setting" "publicip-diagnostics":
│  134: resource "azurerm_monitor_diagnostic_setting" "publicip-diagnostics" {
│ 
╵

I tried updating azurerm version to 3.77.0 in main.tf and this time got some failures on the terraform plan some of which may be expected, but it also contained the following warning (which I overlooked at first):

╷
│ Warning: Argument is deprecated
│ 
│   with azurerm_monitor_diagnostic_setting.hub-central,
│   on main.tf line 221, in resource "azurerm_monitor_diagnostic_setting" "hub-central":
│  221: resource "azurerm_monitor_diagnostic_setting" "hub-central" {
│ 
│ `retention_policy` has been deprecated in favor of `azurerm_storage_management_policy` resource - to learn more https://aka.ms/diagnostic_settings_log_retention
│ 
│ (and 57 more similar warnings elsewhere)
╵

Eventually noticed in the changelog for azurerm 3.74.0 the above and it gave a link to pull request hashicorp/terraform-provider-azurerm#23260 where I noticed the above link which expands to https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/migrate-to-azure-storage-lifecycle-policy?tabs=portal , which includes a note that on 31 Mar 2023 it should no longer be valid to configure retention rules for log data.

I searched issues in this project for azurerm_monitor_diagnostic_setting and found two issues. One closed, the other approved & merged. So filling this out.

Steps to Reproduce

Steps to reproduce the behavior:

1. Have a fresh environment for a US gov subscription type (ours is named "Azure Government Pay-As-You-Go").
2. Download mlz
3. Set variables.tf file
4. Run terraform init
5. Run terraform apply

Expected behavior

It works/deploys to completion

Actual behavior

It stops on the above error due to two resources of resource type azurerm_monitor_diagnostic_setting have child log entries which have retention_policy entries below them enabled. This does show up in at least one other place in MLZ, as the deprecated warning I got was for a different spot and there are another 55 potential candidates that might have it.

Screenshots

Additional context

Operating System:
Terraform Version: v1.4.6
Cloud (public, Azure Government, etc.): Azure Government

[jamasten]

We will not be updating the terraform code and will be deprecating the terraform code on June 30, 2024.