Support for configuring Tables (basic logs, retention time) in Log Analytics

[kensykora]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Basic logs is now in preview and supports greatly reduced ingest cost for things like raw container output which doesn't need to be frequently queried.

Azure CLI also shows there is a retention time setting you can manage

Details

PS C:\Users\KenSykora> az monitor log-analytics workspace table update --help

Command
    az monitor log-analytics workspace table update : Update the properties of a Log Analytics
    workspace table, currently only support updating retention time.

Arguments
    --name -n           [Required] : Name of the table.
    --resource-group -g [Required] : Name of resource group. You can configure the default group
                                     using `az configure --defaults group=`.
    --retention-time    [Required] : The data table data retention in days, between 30 and 730.
                                     Setting this property to null will default to the workspace.
    --workspace-name    [Required] : Name of the Log Analytics Workspace.

Global Arguments
    --debug                        : Increase logging verbosity to show all debug logs.
    --help -h                      : Show this help message and exit.
    --only-show-errors             : Only show errors, suppressing warnings.
    --output -o                    : Output format.  Allowed values: json, jsonc, none, table, tsv,
                                     yaml, yamlc.  Default: table.
    --query                        : JMESPath query string. See http://jmespath.org/ for more
                                     information and examples.
    --subscription                 : Name or ID of subscription. You can configure the default
                                     subscription using `az account set -s NAME_OR_ID`.
    --verbose                      : Increase logging verbosity. Use --debug for full debug logs.

Examples
    Update the retention time of a Log Analytics workspace table
        az monitor log-analytics workspace table update --resource-group MyResourceGroup
        --workspace-name MyWorkspace -n MyTable --retention-time 30

New or Affected Resource(s)

• azurerm_log_analytics_workspace
• azurerm_log_analytics_workspace_table

Potential Terraform Configuration

resource "azurerm_resource_group" "example" {
  name     = "example-resources"
  location = "West Europe"
}

resource "azurerm_log_analytics_workspace" "example" {
  name                = "acctest-01"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
  sku                 = "PerGB2018"
  retention_in_days   = 30
}

resource "azurerm_log_analytics_workspace_table" "container_logs" {
  name         = "ContainerLog"
  workspace_id = azurerm_log_analytics_workspace.example.id

  plan              = "Basic" # or "Analytics"
  retention_in_days = 7 # per docs, setting to null defaults to workspace default
}

References

• https://docs.microsoft.com/en-us/azure/azure-monitor/logs/basic-logs-configure?tabs=api-1%2Cportal-1
• https://docs.microsoft.com/en-us/rest/api/loganalytics/tables/update
• https://azure.microsoft.com/en-us/pricing/details/monitor/

[cgraf-spiria]

Like you mentioned, currently you can set retention per table with the Azure CLI.
As a workaround (until official support) I've implemented it like this:

locals {
    workspace_name = ""
    resource_group_name= ""
    retention_days = -1 # Use default (30 days or 90 days for app. insights tables)
    retention_total_days = 183 # 6 months
    
    # Note: Value beyond two years is restricted to full years. 
    # Allowed values are: [4-730], 1095, 1460, 1826, 2191, 2556 days
}

resource "null_resource" "log_analytics_table_retention" {
    # when changed, will force the null resource to be replaced
    triggers = {
        retention_days = local.retention_days
        retention_total_days = local.retention_total_days
    }

    # Put the name of all the tables you want to change here:
    for_each = toset([
        "TableName"
    ])
    
    provisioner "local-exec" {
        command = "az monitor log-analytics workspace table update --resource-group ${local.resource_group_name} --workspace-name ${local.workspace_name } --name ${each.key} --retention-time ${local.retention_days} --total-retention-time ${local.retention_total_days}"
    }
}

[jcontti-axa]

Is there any update about this?

[wiperpaul]

I Achieved table creation using azapi provider. Sharing to help others.

main.tf

resource "azapi_resource" "table_creation" {
  for_each  = var.tables
  type      = "Microsoft.OperationalInsights/workspaces/tables@2022-10-01"
  name      = each.key
  parent_id = azurerm_log_analytics_workspace.this.id

  body = jsonencode(
    {
      "properties" : {
        "schema" : {
          "name" : each.key,
          "columns" : each.value.columns
        },
        "retentionInDays" : each.value.retentionInDays,
        "totalRetentionInDays" : each.value.totalRetentionInDays
      }
    }
  )
  response_export_values = ["id"]

  depends_on = [
    azurerm_log_analytics_workspace.this
  ]
}

versions.tf

terraform {
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
    }
    azapi = {
      source = "azure/azapi"
    }
  }
}

partial vars.json

          "tables": {
            "test_CL": {
              "retentionInDays": 90,
              "totalRetentionInDays": 0,
              "columns": [
                {
                "name": "TimeGenerated",
                "type": "DateTime"
                },
                {
                "name": "TestColumn",
                "type": "String"
                }

[pregress]

Based on @cgraf-spiria anwser. If you want to change it for all application insight tables and also support for resources in multiple subscriptions:

resource "null_resource" "log_analytics_table_retention" {
  # when changed, will force the null resource to be replaced
  triggers = {
    retention_days      = #fill me in
    resource_group_name = #fill me in
    workspace_name      = #fill me in
    subscription_id     = #fill me in
  }

  # Put the name of all the tables you want to change here:
  for_each = toset([
    "AppAvailabilityResults",
    "AppBrowserTimings",
    "AppDependencies",
    "AppEvents", 
    "AppExceptions",
    "AppMetrics",
    "AppPageViews",
    "AppPerformanceCounters",
    "AppRequests",
    "AppSystemEvents",
    "AppTraces",
  ])

  provisioner "local-exec" {
    command = "az monitor log-analytics workspace table update --resource-group ${self.triggers.resource_group_name} --workspace-name ${self.triggers.workspace_name} --name ${each.key} --retention-time ${self.triggers.retention_days} --total-retention-time ${self.triggers.retention_days} --subscription ${self.triggers.subscription_id}"
  }
}

[Galileo1]

Just a question community folks. Has anyone tried updating the table plan to basic logs. I have been trying the following and keep getting the error from api.

This is the list of tables and these are allowed to be modified as per azure documentation here

log_analytics_workspace_tables_with_basic_logs = ["StorageBlobLogs", "StorageFileLogs", "StorageQueueLogs", "StorageTableLogs", "AKSAuditAdmin", "AKSControlPlane"]

resource "azapi_update_resource" "update_log_analytics_workspace_table_plan" {
  for_each = toset(var.log_analytics_workspace_tables_with_basic_logs)

  type      = "Microsoft.OperationalInsights/workspaces/tables@2022-10-01"
  name      = each.key
  parent_id = azurerm_log_analytics_workspace.primary.id
  body = jsonencode({
    properties = {
      plan = "Basic"
    }
  })
}

RESPONSE 400: 400 Bad Request
│ ERROR CODE: InvalidParameter
│ --------------------------------------------------------------------------------
│ {
│ "error": {
│ "code": "InvalidParameter",
│ "message": "Update retention is not supported on Basic Logs plan. Operation Id: 'blah'"
│ }
│ }

[mattedavi]

I'm trying to use azapi provider to update AzureDiagnostics table, changing the properties retentionInDays and totalRetentionInDays, but I receive a Bad Request error

`resource "azapi_update_resource" "table_retention" {
for_each = local.local_data.tables
type = "Microsoft.OperationalInsights/workspaces/tables@2022-10-01"
name = each.key
parent_id = azurerm_log_analytics_workspace.this.id

body = jsonencode(
{
"properties" : {
"retentionInDays" : each.value.retentionInDays,
"totalRetentionInDays" : each.value.totalRetentionInDays
}
}
)
response_export_values = ["id"]
}`

---

│ RESPONSE 400: 400 Bad Request
│ ERROR CODE: InvalidParameter
│ --------------------------------------------------------------------------------
│ {
│ "error": {│ "code": "InvalidParameter",
│ "message": "Changing Classic table AzureDiagnostics schema by using DataCollectionRuleBased tables api is forbidden, please migrate the table first or use the appropriate classic table api. Operation Id: '*************'"
│ }
│ }
│ --------------------------------------------------------------------------------

Using Azure CLI command I can set retention withuout problems.

[wiperpaul]

@mattedavi

Changing Classic table AzureDiagnostics schema by using DataCollectionRuleBased tables api is forbidden, please migrate the table first or use the appropriate classic table api. Operation Id:

Seems like you are using new log ingestion API and you have not converted your tables or done migration. https://learn.microsoft.com/en-us/azure/azure-monitor/logs/custom-logs-migrate#migration-procedure

I used the old datacollector api

[mgattei]

@mattedavi

Changing Classic table AzureDiagnostics schema by using DataCollectionRuleBased tables api is forbidden, please migrate the table first or use the appropriate classic table api. Operation Id:

I think that this is due to how the azapi_update_resource works.
I noticed that it performs a get of the resource, then the resource is patched with the things in the body of the azapi_update_resource and then a PUT is sent with the result of the merge.

Sending a PUT with the schema populated (even if the schema is unchanged) causes this kind of error.

So by setting the schema property to null you will solve the issue and keep the schema unchanged.

Example:

resource "azapi_update_resource" "table_retention" {
  type      = "Microsoft.OperationalInsights/workspaces/tables@2022-10-01"
  name      = "AzureDiagnostics"
  parent_id = azurerm_log_analytics_workspace.this.id

  body = jsonencode(
    {
      "properties" : {
        "schema" : null,
        "totalRetentionInDays" : 90,
        "retentionInDays" : -1,//Default Workspace retention
      }
    }
  )
}

EDIT: Seems that this would alter the schema, need more investigation.

[mattedavi]

Thank you @mgattei !
Setting the property schema to null it works properly.

[gcbikram]

@mattedavi @mgattei i have tried that approach with azapi, however my terraform plan always shows that its going to change the schema, is it the same for you?

[asolanki22]

@gcbikram did you found something I am getting the change in schema.

[mgattei]

@mattedavi @mgattei i have tried that approach with azapi, however my terraform plan always shows that its going to change the schema, is it the same for you?

Does the 'AzureDiagnostics' table exist in your Log Analytics Workspace?

[gcbikram]

@mgattei yes 'AzureDiagnostics' table exist in my workspace, how did you solve yours?

[mgattei]

@mgattei yes 'AzureDiagnostics' table exist in my workspace, how did you solve yours?

I just tried running a new plan on the sample used before and noticed that the behavior is the same as yours. I need to investigate it more..

[Lu-Ka]

Hi everyone,

Did you successfully to find a solution ? I've got the same error as you, schema of the table is altered.. 😔

[marcindulak]

The functionality requested in the original request is becoming more important due to #23051

[Annesars90]

Yes please, since the issue with the retention policy in diagnostic settings seems to not only be deprecated for storage accounts, but also for log analytics workspaces.

[ireznykov-s]

I use azapi v.1.9.0 and azurerm v.3.42.0. After series of experiments, I ended with the following solution:

• don't use "schema" property, put it to ignore_body_changes section
• at the first run use retentionInDays = -1 to avoid the error "Retention is immutable, not supported on Basic Logs plan."
• at the second run to remove false positive distinctions use retentionInDays = 8

locals {
  log_analytics_retention_days = 60
  log_analytics_basic_tables = [
    "StorageBlobLogs",
    "StorageFileLogs",
    "StorageQueueLogs",
    "StorageTableLogs",
    # add others depending on your services
  ]
}

resource "azapi_update_resource" "basic_log_analytics_workspace_tables" {
  for_each                = toset(local.log_analytics_basic_tables)
  type                    = "Microsoft.OperationalInsights/workspaces/tables@2022-10-01"
  name                    = each.key
  parent_id               = azurerm_log_analytics_workspace.workspace.id
  ignore_missing_property = true

  body = jsonencode(
    {
      # when resource is added, `retentionInDays` should equals -1,
      # at the next run it gives distinction 8 -> -1, so the value should be
      # updated to 8 (default retention period for Basic logs) and changes
      # should be applied
      properties = {
        plan                 = "Basic",
        retentionInDays      = 8
        totalRetentionInDays = local.log_analytics_retention_days
      }
    }
  )

  ignore_body_changes = [
    "schema"
  ]
}

[varismii]

Any news on this?
We have a project requirement to set LAW interactive and archive retention period rules per table via IAC (Terraform is used in our project). Customer requires a retention period tunable solution.

[felixstorm]

Just for reference: I could not get the workarounds based on the azapi provider to work for my case, but probably due to an issue in the provider: Azure/terraform-provider-azapi#385

[ms-henglu]

Hi all,

I'd like to share another workaround by azapi_resource_action if you just want to change the retention in days, hope it could help you:

data "azapi_resource_id" "table" {
  type      = "Microsoft.OperationalInsights/workspaces/tables@2022-10-01"
  name      = "StorageBlobLogs"
  parent_id = azurerm_log_analytics_workspace.main_log.id
}

resource "azapi_resource_action" "log_analytics_workspace_audit_table" {
  type        = "Microsoft.OperationalInsights/workspaces/tables@2022-10-01"
  resource_id = data.azapi_resource_id.table.id
  method      = "PUT"
  // the PUT request body only contains the content in the config, and there'll be no plan-diff for the action resource.
  body = jsonencode({
    properties = {
      schema               = null
      retentionInDays      = 730
      totalRetentionInDays = 1826
    }
  })
}

But above config won't monitor the retentionInDays and totalRetentionInDays, if these fields are changed by other client tools, this resource action won't be triggered automatically.

[qmalik111]

I Achieved table creation using azapi provider. Sharing to help others.

main.tf

resource "azapi_resource" "table_creation" {
  for_each  = var.tables
  type      = "Microsoft.OperationalInsights/workspaces/tables@2022-10-01"
  name      = each.key
  parent_id = azurerm_log_analytics_workspace.this.id

  body = jsonencode(
    {
      "properties" : {
        "schema" : {
          "name" : each.key,
          "columns" : each.value.columns
        },
        "retentionInDays" : each.value.retentionInDays,
        "totalRetentionInDays" : each.value.totalRetentionInDays
      }
    }
  )
  response_export_values = ["id"]

  depends_on = [
    azurerm_log_analytics_workspace.this
  ]
}

versions.tf

terraform {
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
    }
    azapi = {
      source = "azure/azapi"
    }
  }
}

partial vars.json

          "tables": {
            "test_CL": {
              "retentionInDays": 90,
              "totalRetentionInDays": 0,
              "columns": [
                {
                "name": "TimeGenerated",
                "type": "DateTime"
                },
                {
                "name": "TestColumn",
                "type": "String"
                }

Using this initial method, I am trying to update the retention but being faced with, "A resource with the ID "/subscriptions/XXX/... already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for "azapi_resource" for more information.

in order to import the tables into state file, I have hundreds of tables across many tenants. Is there a better way to support this, or any advancements on official documents for this please?

[vanny96]

I think this should be closed as addressed by this other ticket: #6199

[mybayern1974]

Close this issue for, as vanny96 suggested, the expected resource has been supported in AzureRM provider.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.