Diagnostic Settings v2 - What's Missing

[Springstone]

Diagnostic Settings have had a lot of attention here in ALZ land, but with the transition to using the new built-in initiatives and policies, there may be some gaps. This is the place to share those gaps, with people that can potentially drive improvements regularly reviewing.

If you find a service not covered, please added it to this discussion thread so we can track it.

ALZ no longer support diagnostic settings policies. We've completely transitioned to using built-in policies.

[alperkar]

There is no built in policy for:
Function Apps - resource type: Microsoft.Web/sites kind: functionapp
Standard Logic Apps - resource type: Microsoft.Web/sites, kind: functionapp,workflowapp
Web Apps - resource type: Microsoft.Web/sites, kind: NOT contains functionapp

[MikaelJcSoderberg]

I dont see any of the 4 Storage Account Diagnostic settings
blob
queue
table
file

I assume that they are all in this resource type:
Type=Microsoft.Storage/storageAccounts

[MikaelJcSoderberg]

Microsoft.Sql/servers/elasticPools is missing AllLogs, should that be listed as well?

[Springstone]

Storage Accounts are not included due to the very high costs associated with Diagnostic Settings enabled on Storage Accounts, and as such we don't assign this by default (and PG hasn't included Storage Accounts in the initiative). You can selectively configure Diag Settings for critical storage accounts, but do track this carefully, as this can generate a lot of logs.

[NucLabs]

I am a little confused. At first I thought the builtin inititiative Enable allLogs category group resource logging for supported resources to Log Analytics (0884adba-2312-4468-abeb-5422caed1038) is the complete initiative for all diagnostic logging on all resource types, but it is not. Is it a goal for this initiative to be complete in that respect? Or should I create my own initiative with policies that I miss The initiative lacks policies for web applications, function apps and storage accounts, for instance.

[alperkar]

I am just continuing to use the old, deprecated policies and modify them as needed since the built in policies are a-not complete, b-do not include metrics

[NucLabs]

We are doing something like that. We use the builtin alllogs policies as much as possible and I created an initiative myself using the deprecated CAF policies to collect the metrics.
If MS/CAF will not address this issue we will consider to make an initiative with new policies, based on the builtin alllogs policies that will have parameters/deployments for metrics as well.

[mundayn]

Nothing to add - but I'm so disappointed that it took so long to get the new initiatives out and they're not even complete. This is making is so difficult for partners/customers to configure logging on mass.

[jtracey93]

The goal is for this to cover all resources over time, yes

[Springstone]

For the metrics collection topic, please note that metrics through Diag Settings is considered legacy (hence not implemented), this is the way forward: https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/data-collection-metrics?tabs=log-analytics-workspaces

[mwildenb]

The new All Logs policy do not configure any METRICS and there is no built in policy for configuring additional metrics. Problem is that metrics are required for proper defender / threat analysis. Any suggestions on how to tackle?

MORE INFO:

Screenshot on the KeyVault policy to enable "AllLogs" (you can see the empty Metrics)

For Dutch public sector there is a Built In BIO policy urging to enable metric logging on several resources :

And a blog post with a recommendation to enable metrics on storage, eventhub, keyvault, apps and networking
https://techcommunity.microsoft.com/t5/microsoft-defender-for-cloud/security-control-enable-audit-and-logging/ba-p/2079589

[Springstone]

Just a note to everyone on METRICS. METRICS will not be added to Diag Settings policies or initiatives as the control plane is changing and Diag Settings Metrics are considered legacy (and very expensive). Going forward the way to handle this is to use metric streaming based metric export via DCRs which is currently in public preview for some resources. As more resources include support for this paradigm, PG may consider releasing a dedicated initiative for metrics.

See more here: https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/data-collection-metrics?tabs=log-analytics-workspaces

[mundayn]

Why is there such a difference between 'allLogs' and 'audit'?

There are 69 policies for audit, but 140 for allLogs.

I checked a couple of the missing ones, and they have the ability for 'audit'.

For example:
Traffic Manager profiles
Analysis Services

[Springstone]

The difference is by what the resource providers (each resource owner needs to update support for this) have implemented. Category groups are relatively new, and nearly all support allLogs, but only a subset support the auditOnly log category - specifically for audit -> Sentinel. We (ALZ) don't have any influence here, we work with the product groups and owners to improve coverage.

[gr-gh-nexia]

Not sure if it is the right place to post, but I'll give it a shot... I agree with previous posts that (1) there should definitely be a parameter in the built-in initiative to allow sending metrics to LAWS. Further, (2) I think there should be a third Category Group to allow sending the "non-Audit" logs to one destination and the audit logs to another (separating security and operational, more or less). Right now the only choice you have is Audit or AllLogs (which include Audit: double ingestion costs). In order to support, this, we ended up rewriting everything as custom policies (and initiative) to use Categories instead of Category Groups and send Metrics.

[Springstone]

It is possible to send allLogs or auditLogs to one workspace, and metrics to another today. However, all resources are moving to using category groups for logging (not metrics). There shouldn't be double ingestion as you can only use one sink per category.

[gr-gh-nexia]

I probably was not clear in my comment for point # 2 : from what I see, currently, the CategoryGroup "allLogs" contains the "audit" CategoryGroups. So if you send audit logs to a security LAWS and allLogs to an operational LAWS, you would have double ingestion for the audit logs, wouldn't you?

[RS-MPersson]

Microsoft.ContainerService/managedClusters
Microsoft.Compute/virtualMachines
Microsoft.Compute/virtualMachineScaleSets
Microsoft.Network/networkInterfaces