[Metricbeat] Azure storage blob metricset

[andresrc]

Metricbeat Module / Dataset release checklist

Metricset inside the Azure module for blob storage service.

This checklist is intended for Devs which create or update a module to make sure modules are consistent.

Modules

For a metricset to go GA, the following criterias should be met:

• Supported versions are documented
• Supported operating systems are documented (if applicable)
• Integration tests exist
• System tests exist
• Automated checks that all fields are documented
• Documentation
• Fields follow ECS and naming conventions
• Dashboards exists (if applicable)
• Kibana Home Tutorial (if applicable)
  • Open issue in EUI repo to add icon for module if not already exists.
  • Open PR against Kibana repo with tutorial. Examples can be found here.

Metricbeat module

• Example data.json exists and an automated way to generate it exists (go test -data)
• Test environment in Docker exist for integration tests

[narph]

@andresrc , @sorantis , I wonder if we can merge these metricsets (also #14549) into one.
One metricset called storage_account and have a similar configuration as the rest of the metricsets but offer the option to the user which services they want to monitor on (blob, file, queue, table).

- module: azure
  metricsets:
  - storage_account
  enabled: true
  period: 300s
  client_id: '${AZURE_CLIENT_ID:""}'
  client_secret: '${AZURE_CLIENT_SECRET:""}'
  tenant_id: '${AZURE_TENANT_ID:""}'
  subscription_id: '${AZURE_SUBSCRIPTION_ID:""}'
  refresh_list_interval: 600s
  resources:
  - resource_group: [""]
    service_type: ["blob", "file"]
  - resource_id: [""]  

Users can enter one or more resource groups or resource id's to filter on (as we have done with the other metricsets).
Under each resource option they can specify the service type, blob, or file, queue, table. They can ignore that option so we will check for all services inside the storage account.

If no resource option is configured we will have to look for all storage accounts inside the sub (as we have done with the other metricsets).
We will separate each event by service type, ex, a storage account has a container and a table, an event will be generated for each of them containing their specific metrics.
Let me know what you think.

[sorantis]

I'm in favor having one metricset for all storage types. Makes it simple to configure.
Ultimately we want to have the following prefixes:

azure.storage.blob.*
azure.storage.file.*
azure.storage.queue.*
azure.storage.table.*

Can we rename the metricset from storage_account to storage?

[narph]

I don't think that renaming it to storage would be confusing, there are other %storage% resource types there but less popular. So yes, I think we can do that.

also cc @exekias

[narph]

closed by #15342