mdatagen: add wildcard name matching for configs #10065

braydonk · 2024-05-01T19:26:23Z

Description

This PR adds support for wildcard name matching.

Generated MetricsConfig and ResourceAttributesConfig will now support providing names not just as full names, but also using * wildcards and multimatching with {x,y,etc}. This allows you to apply configs to groups of metrics and resource attributes, simplifying configs.

Link to tracking issue

Fixes #10074

Testing

I go installed mdatagen from this PR and ran it on a receiver in opentelemetry-collector-contrib and was able to use wildcard matching to enable/disable groups of metrics as expected.

Documentation

I have written a specification in a gist: https://gist.github.com/braydonk/ccb6775331fdd5dca91a650330b9839f

I am not sure where this documentation should live.

codecov · 2024-05-01T20:15:53Z

Codecov Report

Attention: Patch coverage is 33.33333% with 8 lines in your changes missing coverage. Please review.

Project coverage is 92.43%. Comparing base (c7c3401) to head (d82df38).
Report is 119 commits behind head on main.

❗ Current head d82df38 differs from pull request most recent head a522523

Please upload reports for the commit a522523 to get more accurate results.

Files	Patch %	Lines
cmd/mdatagen/main.go	33.33%	4 Missing and 4 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #10065      +/-   ##
==========================================
- Coverage   92.59%   92.43%   -0.16%     
==========================================
  Files         387      387              
  Lines       18198    18264      +66     
==========================================
+ Hits        16850    16883      +33     
- Misses       1007     1031      +24     
- Partials      341      350       +9

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

braydonk · 2024-05-01T20:19:55Z

I think that code coverage change is fine here because none of the return err cases for template file generation are covered anyway.

braydonk · 2024-05-02T12:20:23Z

The changelog issue can only be resolved once the issue is moved from contrib to here.

cmd/mdatagen/templates/match.go.tmpl

github-actions · 2024-05-19T03:16:01Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

dmitryax · 2024-05-21T19:16:01Z

@braydonk, are you still working on this? Just checking. Let me know if you have any concerns with my comments

braydonk · 2024-05-21T19:40:13Z

Hi @dmitryax yes I am. I realized that I forgot to mention that I am writing a full spec to explain how it works, and while I was doing that found a number of edge cases that prompted me to rewrite the parser. Taking longer than expected. I am still working on this though and will keep it unstale.

This PR adds support for wildcard name matching. Generated MetricsConfig and ResourceAttributesConfig will now support providing names not just as full names, but also using `*` wildcards and multimatching with `{x,y,etc}`. This allows you to apply configs to groups of metrics and resource attributes, simplifying configs. Signed-off-by: braydonk <[email protected]>

braydonk · 2024-05-28T16:17:54Z

What the actual errors look like from collector output is not super obvious from the PR, so here are some examples. I used the hostmetrics process scraper in my example.

Not matching any names with the patterns

hostmetrics:
  collection_interval: 10s
  scrapers:
    process:
      metrics:
        "nothing.*":
          enabled: false

$ ./otelcontribcol --config config.yaml 
Error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:

* error decoding 'receivers': error reading configuration for "hostmetrics": error reading settings for scraper type "process": 1 error(s) decoding:

* error decoding 'metrics': specified patterns don't match any names
2024/05/28 16:06:44 collector server run finished with error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:

* error decoding 'receivers': error reading configuration for "hostmetrics": error reading settings for scraper type "process": 1 error(s) decoding:

* error decoding 'metrics': specified patterns don't match any names

Parsing errors in a pattern

hostmetrics:
  collection_interval: 10s
  scrapers:
    process:
      metrics:
        "a_.*":
          enabled: false

$ ./otelcontribcol --config config.yaml
Error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:

* error decoding 'receivers': error reading configuration for "hostmetrics": error reading settings for scraper type "process": 1 error(s) decoding:

* error decoding 'metrics': pattern parsing error:
a_.*
  ^
invalid pattern: unexpected token
2024/05/28 16:06:52 collector server run finished with error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:

* error decoding 'receivers': error reading configuration for "hostmetrics": error reading settings for scraper type "process": 1 error(s) decoding:

* error decoding 'metrics': pattern parsing error:
a_.*
  ^
invalid pattern: unexpected token

braydonk · 2024-05-28T16:38:25Z

I'm gonna wait to address the conflicts, cause this is just going to keep coming up continuously throughout the review.

github-actions · 2024-06-12T03:16:18Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

braydonk · 2024-06-12T12:18:34Z

Not stale

dmitryax · 2024-06-13T16:51:11Z

cmd/mdatagen/templates/config.go.tmpl

@@ -10,6 +10,12 @@ import (
 )
 {{ if .Metrics -}}

+var MetricNames = []string{


We don't need to expose this variable if it's not intended to be used outside. It can be rendered even right in expandPatternMap

Good catch thanks, this was an accident

Done in a758241

I couldn't render directly in expandPatternMap because there could be either metrics or resource attributes passed in. I render both arrays more locally to their usage though instead of as globals.

dmitryax · 2024-06-13T17:19:25Z

cmd/mdatagen/templates/match.go.tmpl

+		// or end of input are valid.
+		if !p.isFinished() {
+			curr, _ := p.current()
+			if curr != '.' {


If we allow a group as a whole to be either an identifier, a group, or a wildcard, can we use strings.Split(pattern, '.') and just read each part instead of having this loop? I think it should simplify the code.

I like the approach you took. However, even if we do the parsing ourselves, I'd like to keep the code simpler if possible.

I'll give that a shot. When I first tried that I was having trouble making nice error messages; currently the error messages can track the exact token that is causing a problem. I think I can come up with a way to preserve that though, will give it a try soon

I tried this but I found it was becoming equally as complex. The complexity saved by this approach is:

Shared scanner object, so I'm not needing to pass around indexes for context to different spots

I can share identifier parsing code between an identifier as a root group, and an identifier as part of a multimatch

How strongly do you feel on this? I can try to make it work if you think it's the way to go.

dmitryax · 2024-06-13T17:20:06Z

cmd/mdatagen/main.go

@@ -124,6 +124,23 @@ func run(ymlPath string) error {
 		toGenerate[filepath.Join(tmplDir, "resource.go.tmpl")] = filepath.Join(codeDir, "generated_resource.go")
 		toGenerate[filepath.Join(tmplDir, "resource_test.go.tmpl")] = filepath.Join(codeDir, "generated_resource_test.go")
 	}
+	if err = generateFile(filepath.Join(tmplDir, "match.go.tmpl"),


This will be generated for every component in contrib. We need to generate these files only for scrapers len(md.Metrics) > 0.

Done in f416166

Correction in a522523

github-actions · 2024-07-09T03:16:30Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

github-actions · 2024-07-23T03:16:36Z

Closed as inactive. Feel free to reopen if this PR is still being worked on.

braydonk requested review from a team and Aneurysm9 May 1, 2024 19:26

braydonk force-pushed the wildcard_names branch 2 times, most recently from 026ad18 to 2ed7ec4 Compare May 1, 2024 20:07

braydonk mentioned this pull request May 1, 2024

[cmd/mdatagen] Ability to disable/enable all metrics #10074

Open

dmitryax reviewed May 4, 2024

View reviewed changes

cmd/mdatagen/templates/match.go.tmpl Outdated Show resolved Hide resolved

cmd/mdatagen/templates/match.go.tmpl Show resolved Hide resolved

cmd/mdatagen/templates/match.go.tmpl Outdated Show resolved Hide resolved

github-actions bot added the Stale label May 19, 2024

dmitryax removed the Stale label May 21, 2024

braydonk added 4 commits May 28, 2024 15:59

fix issue number

5c9fb7e

matching code rewritten

19f3d88

fail parsing if there are no matches

d82df38

braydonk force-pushed the wildcard_names branch from 3f0a4e8 to d82df38 Compare May 28, 2024 16:02

rerun gogenerate

08cf72d

github-actions bot added the Stale label Jun 12, 2024

dmitryax removed the Stale label Jun 13, 2024

dmitryax reviewed Jun 13, 2024

View reviewed changes

braydonk added 3 commits June 24, 2024 10:13

don't export metric and resattr names

a758241

only generate match.go when necessary

f416166

also generate match_test.go

a522523

github-actions bot added the Stale label Jul 9, 2024

github-actions bot closed this Jul 23, 2024

braydonk mentioned this pull request Sep 20, 2024

hostmetricsreceiver semantic conventions transition open-telemetry/opentelemetry-collector-contrib#35325

Open

12 tasks

dmitryax mentioned this pull request Sep 20, 2024

Scraper feedback, and what to do next? #11238

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

mdatagen: add wildcard name matching for configs #10065

mdatagen: add wildcard name matching for configs #10065

braydonk commented May 1, 2024 •

edited

Loading

codecov bot commented May 1, 2024 •

edited

Loading

braydonk commented May 1, 2024

braydonk commented May 2, 2024

github-actions bot commented May 19, 2024

dmitryax commented May 21, 2024

braydonk commented May 21, 2024

braydonk commented May 28, 2024 •

edited

Loading

braydonk commented May 28, 2024

github-actions bot commented Jun 12, 2024

braydonk commented Jun 12, 2024

dmitryax Jun 13, 2024

braydonk Jun 20, 2024

braydonk Jun 24, 2024

dmitryax Jun 13, 2024

braydonk Jun 20, 2024 •

edited

Loading

braydonk Jun 24, 2024

dmitryax Jun 13, 2024

braydonk Jun 24, 2024

braydonk Jun 24, 2024

github-actions bot commented Jul 9, 2024

github-actions bot commented Jul 23, 2024

mdatagen: add wildcard name matching for configs #10065

mdatagen: add wildcard name matching for configs #10065

Conversation

braydonk commented May 1, 2024 • edited Loading

Description

Link to tracking issue

Testing

Documentation

codecov bot commented May 1, 2024 • edited Loading

Codecov Report

braydonk commented May 1, 2024

braydonk commented May 2, 2024

github-actions bot commented May 19, 2024

dmitryax commented May 21, 2024

braydonk commented May 21, 2024

braydonk commented May 28, 2024 • edited Loading

braydonk commented May 28, 2024

github-actions bot commented Jun 12, 2024

braydonk commented Jun 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

braydonk Jun 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jul 9, 2024

github-actions bot commented Jul 23, 2024

braydonk commented May 1, 2024 •

edited

Loading

codecov bot commented May 1, 2024 •

edited

Loading

braydonk commented May 28, 2024 •

edited

Loading

braydonk Jun 20, 2024 •

edited

Loading