Common Fields for Cloud Inventory Schema

[kaiyan-sheng]

This issue is to track the work for adding common fields among different cloud metrics. These proposed common fields will become required fields to collect when adding support for new cloud platforms.

Proposed Common Fields:

• cloud.account.id
• cloud.availability_zone
• cloud.instance.id
• cloud.instance.name
• cloud.machine.type
• cloud.provider
• cloud.region

TODO:

• 1. Add cloud.availability_zone into azure: Add cloud.availability_zone into azure module #21363
• 2. Add cloud.account.id into azure: Add cloud.account.id into azure module #21381

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[kaiyan-sheng]

For cloud.account.id, the definition in ECS is:

The cloud account or organization id used to identify different entities in a multi-tenant environment.

Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.

AWS and GCP are both clear what to use for cloud.account.id. For Azure, we are looking at using subscription ID for cloud.account.id. I think we should update the ECS document on cloud.account.id to include what is used for Azure.

[kaiyan-sheng]

Question from @narph:
If we map it to the subscription id, how do we envision this to look in the UI? Will users be able to filter using the account.id? If so, can we have a more clear naming in the UI that would make sure azure users will understand? Just having Account ID in the list could make them think of the billing account or other types of accounts when it comes to Azure?

@sorantis @exekias WDYT?

[exekias]

@narph, correct me if I'm wrong here, this is how I understand it:

All resources (ie VMs) are tied to the subscription id. Users know it well, as they use it to configure Azure API clients (like our own agent), you cannot move a resource to another subscription.

Billing accounts group subscriptions for billing purposes. You can move subscriptions from some billing accounts to another.

Having all this into account (pun intended 😛), it would look like subscription id is a better fit for account.id. For these reasons:

• Resources are always tied to the same subscription id
• This id is closer to users, so they know it better
• If we store billing account as account.id, I cannot filter to a single subscription id, which is something I may want? If we store subscription id, you can still jump from a billing account to all the subscription ids on it, using that as a filter if needed.

[narph]

@exekias , I thought we previously agreed that we would align the sub ID to the cloud.account.id field (and still keep the value inside azure.subscriptionId) , my question was more related to the UI, how will it look and if we can accommodate a naming (in the UI) that would help azure users understand we are talking about the subscription ID., for example Account ID/Subscription ID.

[exekias]

Understood, any opinions @sorantis? Something that comes to my mind is that UI can be smart enough to detect the cloud.provider and show the correct name based on it

[sorantis]

Today the UI doesn't provide account id as a default field to group by. Here the default option for Hosts:

Users are free to group by any field, e.g. cloud.account.id, but right now it's not explicitly provided. If we decide to have it by default then I'd use Account ID as the preferred naming even for Azure.

cc @simianhacker

[kaiyan-sheng]

We discussed this question during meeting and the conclusion is: we will use subscription ID as cloud.account.id and once we start changing metrics UI, we can start discuss how to clarify the definition of cloud.account.id between different cloud providers.

[kaiyan-sheng]

I'm closing this issue because the only one task left is to add cloud.availability_zone into azure, which is tracked by #21363. This issue is depends on a fix in Azure SDK so we don't know when will be resolved.