Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove dynamic templates from otel-plugin that set index:false #113409

Merged
merged 5 commits into from
Sep 26, 2024
Merged

Remove dynamic templates from otel-plugin that set index:false #113409

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
6f4ed6d
Remove dynamic templates from otel-plugin that set `index:false`
gregkalapos Sep 23, 2024
c2fae90
Update x-pack/plugin/otel-data/src/yamlRestTest/resources/rest-api-sp…
gregkalapos Sep 24, 2024
ee89dde
Remove unused dynamic templates
gregkalapos Sep 24, 2024
c49f374
Merge branch 'main' into otel_fix_dynamic_templates_with_index_false
elasticmachine Sep 25, 2024
1c90154
Update [email protected]
gregkalapos Sep 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 0 additions & 44 deletions x-pack/plugin/core/template-resources/src/main/resources/[email protected]
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,50 +43,6 @@
"default_metric": "value_count"
}
}
},
{
"histogram": {
"mapping": {
"type": "histogram",
"ignore_malformed": true
}
}
},
{
"counter_long": {
"mapping": {
"type": "long",
"time_series_metric": "counter",
"ignore_malformed": true
}
}
},
{
"gauge_long": {
"mapping": {
"type": "long",
"time_series_metric": "gauge",
"ignore_malformed": true
}
}
},
{
"counter_double": {
"mapping": {
"type": "double",
"time_series_metric": "counter",
"ignore_malformed": true
}
}
},
{
"gauge_double": {
"mapping": {
"type": "double",
"time_series_metric": "gauge",
"ignore_malformed": true
}
}
}
],
"properties": {
Expand Down
43 changes: 43 additions & 0 deletions x-pack/plugin/otel-data/src/main/resources/component-templates/[email protected]
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,46 @@ template:
type: keyword
time_series_dimension: true
ignore_above: 1024
dynamic_templates:
- histogram:
mapping:
type: histogram
ignore_malformed: true
- counter_long:
mapping:
type: long
time_series_metric: counter
ignore_malformed: true
- gauge_long:
mapping:
type: long
time_series_metric: gauge
ignore_malformed: true
- counter_double:
mapping:
type: double
time_series_metric: counter
ignore_malformed: true
- gauge_double:
mapping:
type: double
time_series_metric: gauge
ignore_malformed: true
- summary_gauge:
mapping:
type: aggregate_metric_double
time_series_metric: gauge
ignore_malformed: true
- summary_counter:
mapping:
type: aggregate_metric_double
time_series_metric: counter
ignore_malformed: true
- histogram_metrics:
mapping:
type: histogram
- summary_metrics:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we take the opportunity to make this one's name consistent with the others?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me. We'd then just call it summary, right? This means that there will need to be changes in the ES exporter, though.

cc @carsonip

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I was thinking summary too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I renamed it to summary. @carsonip I'll wait for you to acknowledge this before I push merge, just to make sure we don't miss the change in ES exporter.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering if rolling out that change is easier by temporarily adding both summary and summary_metrics in this PR, then making the change in the ES exporter, then removing the summary_metrics dynamic template in another ES PR.

That way, there's never a situation where ES exporter@main is incompatible with ES@main, without having to coordinate to merge the ES and ES exporter PRs at the same time (which is somewhat unpredictable for the ES exporter).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ack. Will rename it to summary.

I'm wondering if rolling out that change is easier by temporarily adding both summary and summary_metrics in this PR, then making the change in the ES exporter, then removing the summary_metrics dynamic template in another ES PR.

I'm not too concerned. e2e poc pins ES version and es exporter version anyway. The mains don't have to be compatible at every point of time.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR: open-telemetry/opentelemetry-collector-contrib#35424

mapping:
type: aggregate_metric_double
metrics: sum, value_count
default_metric: value_count
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These 2 are duplicates from [email protected] - they were already there before, so I also left them in [email protected]. Do we actually need these 2 here for OTel?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/elasticsearchexporter/model.go, I could not find usage of the following dynamic templates. Thus, I think they can be removed.

  • summary_gauge
  • summary_counter
  • histogram_metrics

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will they be useful in any scenarios? If there is a chance that we need it, might be useful to keep these if a future version of es exporter need to work with 8.16 ES. Just asking if see how likely we need them.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After looking into this, I'd say it's safe to remove all 3.

  • histogram_metrics: That is covered by our histogram template, which is the same plus it adds ignore_malformed: true. Without ignore_malformed: true, I doubt we'd want to re-use this. histogram_metrics comes from the old core templates.
  • summary_counter: This is covered by counter_long and counter_double, the difference is that this one sets type: aggregate_metric_double instead of double and long. By looking at docs for counters in OTel I doubt we'd want to use aggregate_metric_double - that does not match the OTel counters.
  • summary_gauge: same as above, and this is covered by gauge_long and gauge_double.

So with that, I don't think we'd want to use those dynamic templates for OTel later. -> I'd be for just removing.

Any thoughts? Maybe something I'm overlooking?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sgtm

2 changes: 0 additions & 2 deletions x-pack/plugin/otel-data/src/main/resources/index-templates/[email protected]
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@ _meta:
description: default OpenTelemetry metrics template installed by x-pack
managed: true
composed_of:
- metrics@mappings
- metrics@tsdb-settings
- otel@mappings
- metrics-otel@mappings
Expand Down Expand Up @@ -38,4 +37,3 @@ template:
ignore_above: 1024
type: keyword
match_mapping_type: string

18 changes: 18 additions & 0 deletions x-pack/plugin/otel-data/src/yamlRestTest/resources/rest-api-spec/test/20_metrics_tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,3 +178,21 @@ IP dimensions:
expand_wildcards: hidden
- match: { .$idx0name.mappings.properties.resource.properties.attributes.properties.host\.ip.type: 'ip' }
- match: { .$idx0name.mappings.properties.attributes.properties.philip.type: "keyword" }
---
"Long data type in attributes must be accepted":
- do:
indices.get_index_template:
name: metrics-otel@template
- length: {index_templates: 1}
- do:
bulk:
index: metrics-generic.otel-default
refresh: true
body:
- create: {}
- '{"@timestamp":"2024-07-18T14:48:33.467654000Z","data_stream":{"dataset":"generic.otel","namespace":"default"},"attributes":{"processor.pid": 17, "transaction.root":false},"resource":{"attributes":{"service.name":"OtelSample","telemetry.sdk.language":"dotnet","telemetry.sdk.name":"opentelemetry"}}}'
gregkalapos marked this conversation as resolved.
Show resolved Hide resolved
- is_false: errors
- do:
search:
index: metrics-generic.otel-default
- length: { hits.hits: 1 }