Retention Policy of Azure Activity Logs Diagnostic Settings are being discarded

[apsi15]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v1.0.7
Azurerm v2.74.0

Affected Resource(s)

"azurerm_monitor_diagnostic_setting" "activity_logs"

Terraform Configuration Files

# Resource
resource "azurerm_monitor_diagnostic_setting" "activity_logs" {
 
  name               = "acitivity_logs"
  target_resource_id = "subscriptions/****"
 
  log_analytics_workspace_id     = local.log_analytics_workspace_id
 
  storage_account_id =local.storage_account_id
 
  dynamic "log" {
    for_each = local.diagnostic_settings.log
    content {
      category = log.value[0]
      enabled  = log.value[1]
      # Storage Account retention policy
      retention_policy {
        enabled = log.value[2]
        days    = log.value[3]
      }
    }
  }

# Locals
storage_account_id         = "id"
log_analytics_workspace_id = "id"
 
      diagnostic_settings= {
        log = [
          #["Category name",  "Diagnostics Enabled(true/false)", "Retention Enabled(true/false)", Retention_period] 
          ["Administrative", true, true, 90],
          ["Security", true, true, 90],
          ["ServiceHealth", true, true, 90],
          ["Alert", true, true, 90],
          ["Recommendation", true, true, 90],
          ["Policy", true, true, 90],
          ["Autoscale", true, true, 90],
          ["ResourceHealth", true, true, 90],
        ]
      }

Debug Output

**# module.diagnostics_subscription_activity_logs.azurerm_monitor_diagnostic_setting.activity_logs will be updated in-place**
  ~ resource "azurerm_monitor_diagnostic_setting" "activity_logs" {
        id                             = "****"
        name                           = "***"
        # (6 unchanged attributes hidden)
 
      + log {
          + category = "Administrative"
          + enabled  = true
 
          + retention_policy {
              + days    = 90
              + enabled = true
            }
        }
      - log {
          - category = "Administrative" -> null
          - enabled  = true -> null
        }
      + log {
          + category = "Alert"
          + enabled  = true
 
          + retention_policy {
              + days    = 90
              + enabled = true
            }
        }
      - log {
          - category = "Alert" -> null
          - enabled  = true -> null
        }
      + log {
          + category = "Autoscale"
          + enabled  = true
 
          + retention_policy {
              + days    = 90
              + enabled = true
            }
        }
      - log {
          - category = "Autoscale" -> null
          - enabled  = true -> null
        }
      + log {
          + category = "Policy"
          + enabled  = true
 
          + retention_policy {
              + days    = 90
              + enabled = true
            }
        }
      - log {
          - category = "Policy" -> null
          - enabled  = true -> null
        }
      + log {
          + category = "Recommendation"
          + enabled  = true
 
          + retention_policy {
              + days    = 90
              + enabled = true
            }
        }
      - log {
          - category = "Recommendation" -> null
          - enabled  = true -> null
        }
      + log {
          + category = "ResourceHealth"
          + enabled  = true
 
          + retention_policy {
              + days    = 90
              + enabled = true
            }
        }
      - log {
          - category = "ResourceHealth" -> null
          - enabled  = true -> null
        }
      + log {
          + category = "Security"
          + enabled  = true
 
          + retention_policy {
              + days    = 90
              + enabled = true
            }
        }
      - log {
          - category = "Security" -> null
          - enabled  = true -> null
        }
      + log {
          + category = "ServiceHealth"
          + enabled  = true
 
          + retention_policy {
              + days    = 90
              + enabled = true
            }
        }
      - log {
          - category = "ServiceHealth" -> null
          - enabled  = true -> null
        }
    }

Expected Behaviour

Retention Policy of activity logs must not be reprovisioned or updated if the settings hat not being changed.

Actual Behaviour

When first provisioning the activity logs of the subscription to be sent to the log analytics and archived in a storage account, no errors or issues are shown. Everything is working as expected. The next time we try to perform terraform plan or deploy, terraform inform us that the all diagnostic settings of the activity logs must be updated in place (delete --> deploy). After troubleshooting, we found out that following:

• Using Get-AzDiagnosticSetting -ResourceId "/subscriptions/***", showed us that terraform is not configuring the retention policy at all. The attribute "RetentionPolicy" under logs is completely empty.
• In Azure Portal, once you enter the Diagnostic Settings of the Activity Logs unter AZure monitor, you can see that there is way to configure there the Retention Policy for the Storage Account. However it can be configured using Azure Powershell or CLI.
• In Terraform tfstate, the Retention Policy is still there, even that Terraform didn't really configure it and in Azure it's still empty.

Steps to Reproduce

1- write diagnostic settings for the Azure subscription activity logs with retention policy for storage account enabled
2- deploy terraform code: terraform apply
3- rerun terraform plan or apply

Impact

This behavior is affecting the storage account retention time of all activity_logs for the subscription, specially for auditing. As well as having unnecessary delete and apply operations with each apply, which can affect the approvals in the pipeliens.

Important Factoids

N/A

References

N/A

• #0000

[kensykora]

What's interesting is that you can't set it in the portal either, I wonder if this is a limitation of Azure for Activity Log exports. I don't understand why it would be different than anything else.

[kensykora]

btw this ONLY applies to subscription scopes (For exporting Activity Log) -- other resources seem to still support this, and it's working as expected.

I'm working around this by putting a blob lifecycle policy on the account I'm exporting it to

resource "azurerm_storage_management_policy" "ops_security" {
  storage_account_id = azurerm_storage_account.ops_security.id

  rule {
    name    = "DeleteAfter1Year"
    enabled = true

    filters {
    }

    actions {
      base_blob {
        delete_after_days_since_modification_greater_than = 365
      }

      snapshot {
        delete_after_days_since_creation_greater_than = 365
      }

      version {
        delete_after_days_since_creation = 365
      }
    }
  }
}

[alxndr13]

The problem still persists with terraform v1.0.11. Thanks for the workaround @kensykora. Unfortunately this affects the assessment results of Regulatory Compliance Checks.

[daniel-anova]

Duplicate or related to #10388 ?

[alxndr13]

Duplicate or related to #10388 ?

Definetly related. #10388 just has a different scope. While we're talking about a subscription wide setting here, the issue #10388 is Azure SQL DB and Synapse Pool related. Could be the same issue though.

[tombuildsstuff]

Duplicate of #10388

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.