mdatagen support for slice and map attributes

[fredthomsen]

Component(s)

cmd/mdatagen

Is your feature request related to a problem? Please describe.

I would like for mdatagen to support generation of code supporting attributes that are maps or slices. I am attempting to add a resource attribute which is a list of strings containing some information that should be acted on by a downstream processor (I realize that #16811 makes this unfeasible at the moment).

Describe the solution you'd like

In the metadata.yml file used by mdatagen, I would like to be able to use a configuration like the following:

resource_attributes:
  sl.app.classifications:
    description: A list of classifications for this data
    type: strings

And this would generate a function like this for setting the resource attribute:

func WithSlAppClassifications(vals []string) ResourceMetricsOption {
       return func(rm pmetric.ResourceMetrics) {
               c := rm.Resource().Attributes().PutEmptySlice("sl.app.classifications")
               for _, val := range vals {
                       c.AppendEmpty().SetStr(val)
        }
 }

Similar approach should be taken for metric attributes by providing their RecordXDataPoint functions.

Describe alternatives you've considered

Just use a attribute string that is split by a known deliminator such as |.

Additional context

The above just assumes a single-level of nesting. With multiple levels specifying the types at each level requires a different way to specify the type.

[github-actions]

Pinging code owners:

• cmd/mdatagen: @dmitryax

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dmitryax]

Thanks for reporting this @fredthomsen. Which receiver do you want to use this functionality in?

[fredthomsen]

We have a custom receiver that we have written to scrape infrastructure data from our monitoring system (this is still pretty alpha) and we were planning on using these resource tags to make some downstream storage routing decisions. Also, I am happy to handle this work if you agree this functionality is desirable.

[dmitryax]

Yes, I think this functionality is desirable. Fill free to submit a PR

[dmitryax]

Couple of comments:

Let's stick to go type definitions in metadata.yaml:

resource_attributes:
  sl.app.classifications:
    description: A list of classifications for this data
    type: []string

I believe we can make the generated code simple and have the following:

func WithSlAppClassifications(vals []string) ResourceMetricsOption {
       return func(rm pmetric.ResourceMetrics) {
               rm.Resource().Attributes().FromRaw(vals)
        }
 }

Setting å resource attribute is one operation per scrape, so it's not a performance critical part

[fredthomsen]

Setting å resource attribute is one operation per scrape, so it's not a performance critical part

I used resource attributes as an example, but was planning on doing this for metric attributes as well.

[dmitryax]

FromRaw for maps and slices should be as performant as your example. Let's use it and keep assignments for other types as is for now

[fredthomsen]

@dmitryax PR is out, but I am having to deal with some conversions from []string to []any and map[string]string to map[string]any to leverage FromRaw. I am sure those implications are worth discussing.

[Aneurysm9]

I used resource attributes as an example, but was planning on doing this for metric attributes as well.

Note that maps and heterogeneous arrays are not valid attribute values for metrics or resource attributes. This may change in the (maybe near) future, but I'm not sure we should get out ahead of the spec here.

/cc @open-telemetry/collector-approvers

[dmitryax]

@Aneurysm9 Thanks for pointing to the PR. Looks like it's close to being merged. Given that it's supported in the OTLP, I think we should be good to add the mdatagen support once the spec PR is merged.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• cmd/mdatagen: @dmitryax

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dmitryax]

Resolved by #18487