Add support for consuming OTLP/gRPC metrics #4722

axw · 2021-02-11T05:19:05Z

Motivation/summary

Add support for consuming basic OTLP/gRPC metrics,
like we support in the OpenTelemetry Collector exporter.

We do not yet support distributions or summaries; these
will be added once the Elasticsearch enhancements we
are dependent on are available. If these are received, the
server will ignore them and increment a monitoring counter
to indicate that an unsupported metric was received and
dropped.

Checklist

Update CHANGELOG.asciidoc
~~- [ ] Documentation has been updated~~ [OTLP] Document support for OTLP/gRPC on the standard port #4678

How to test these changes

Start apm-server with monitoring enabled
Run an application that is instrumented with OpenTelemetry to collect metrics (including some distributions and summaries), and configured to export OTLP to APM Server's standard host:port.
Repeat step 2, exporting to an OpenTelemetry Collector with the elastic exporter.
Compare the data received directly by APM Server to the data sent by the exporter.
Check the beats monitoring doc for apm-server, and check the value for apm-server.otlp.grpc.metrics.consumer.unsupported_dropped corresponds to the number of distribution and summary metrics sent

Related issues

Closes #4503

apmmachine · 2021-02-11T05:22:58Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #4722 updated
Start Time: 2021-02-16T01:42:34.255+0000
Duration: 56 min 20 sec
Commit: 53a9a74

Test stats 🧪

Test	Results
Failed	0
Passed	4733
Skipped	124
Total	4857

Trends 🧪

Steps errors

Expand to view the steps failures

`Run Window tests`

Took 10 min 6 sec . View more details on here

`Compress`

Took 0 min 0 sec . View more details on here
Description: tar --exclude=coverage-files.tgz -czf coverage-files.tgz coverage

`Compress`

Took 0 min 0 sec . View more details on here
Description: tar --exclude=system-tests-linux-files.tgz -czf system-tests-linux-files.tgz system-tests

`Test Sync`

Took 3 min 26 sec . View more details on here
Description: ./.ci/scripts/sync.sh

Add support for consuming basic OTLP/gRPC metrics, like we support in the OpenTelemetry Collector exporter. This does not cover distributions or summaries; these will be added once the Elasticsearch enhancements we are dependent on are available.

codecov-io · 2021-02-11T07:49:38Z

Codecov Report

Merging #4722 (53a9a74) into master (a25b110) will increase coverage by 0.26%.
The diff coverage is 97.72%.

@@            Coverage Diff             @@
##           master    #4722      +/-   ##
==========================================
+ Coverage   76.43%   76.70%   +0.26%     
==========================================
  Files         165      166       +1     
  Lines       10083    10211     +128     
==========================================
+ Hits         7707     7832     +125     
- Misses       2376     2379       +3

Impacted Files	Coverage Δ
beater/otlp/grpc.go	`87.87% <91.30%> (+2.16%)`	⬆️
processor/otel/metrics.go	`99.06% <99.06%> (ø)`
processor/otel/consumer.go	`94.89% <100.00%> (+0.02%)`	⬆️

jalvz · 2021-02-11T11:07:00Z

processor/otel/consumer.go

+	UnsupportedMetricsDropped int64
+}
+
+type consumerStats struct {


I am confused about this, why both consumerStats and ConsumerStats are needed?

consumerStats might change at any time, ConsumerStats is a snapshot. I'll add a comment.

jalvz · 2021-02-11T11:11:34Z

processor/otel/metrics.go

+	otelMetrics := in.Metrics()
+	var unsupported int64
+	for i := 0; i < otelMetrics.Len(); i++ {
+		if !c.addMetric(otelMetrics.At(i), &ms) {


instead of having a method that both mutates and argument and returns a value, can't we say unsupported += otelMetrics.Len() - ms.Len()?

No, because two metrics may end up in the same metricset. (ms is a collection of metricsets)

jalvz · 2021-02-11T11:16:24Z

processor/otel/metrics.go

+func (c *Consumer) addMetric(metric pdata.Metric, ms *metricsets) bool {
+	switch metric.DataType() {
+	case pdata.MetricDataTypeIntGauge:
+		dps := metric.IntGauge().DataPoints()


Maybe dumb question, but can't we make all these data types conform an interface defining LabelsMap(), Timestamp() and Value()?
All these cases are the same

Yes, but:

That won't work when we add support for distributions and summaries

Ideally we would also support distinguishing doubles from ints, storing the latter in a long field type

I'd prefer to leave it as it is for now (this is also how it looks in opentelemetry-collector-contrib), and refactor it later if it's a pain.

axw · 2021-02-15T02:50:10Z

jenkins run the tests please

simitt · 2021-03-04T13:17:38Z

Tested with BC 2; metrics are accepted and ingested as expected.

Regarding testing of distributions or summaries - how did you test that when implementing? I was instrumenting a go app, but the otel go metrics implementation only supports counters so far.

axw · 2021-03-05T01:57:41Z

@simitt it does support histograms and summaries. You need to define an aggregator in the metrics controller, and then use a ValueRecorder meter. Frankly speaking their metrics API is obtuse, so you can't be faulted for missing that :)

Here's how I tested histograms:

apm-server/systemtest/otlp_test.go

Lines 174 to 193 in d0624d4

    
           controller := controller.New( 
        
           	processor.New( 
        
           		simple.NewWithHistogramDistribution([]float64{1, 100, 1000, 10000}), 
        
           		exporter, 
        
           	), 
        
           	controller.WithPusher(exporter), 
        
           	controller.WithCollectPeriod(time.Minute), 
        
           ) 
        
           if err := controller.Start(context.Background()); err != nil { 
        
           	return err 
        
           } 
        
           meterProvider := controller.MeterProvider() 
        
           meter := meterProvider.Meter("test-meter") 
        
           float64Counter := metric.Must(meter).NewFloat64Counter("float64_counter") 
        
           float64Counter.Add(context.Background(), 1) 
        
           // This will be dropped, as we do not support consuming histograms yet. 
        
           int64Recorder := metric.Must(meter).NewInt64ValueRecorder("int64_recorder") 
        
           int64Recorder.Record(context.Background(), 123)

I believe you need to change NewWithHistogramDistribution to NewWithInexpensiveDistribution to aggregate the recordings as a summary instead.

simitt · 2021-03-05T08:52:30Z

I was missing that one could use simple.NewWithInexpensiveDistribution() - thanks.

Tested with BC3:

compared metrics for data sent via otel exporter and elastic exporter - no differences found
works as expected when using the otel exporter to directly send data to APM Server
when using the elastic exporter, no beats_stats.metrics.apm-server.otlp.grpc.metrics.consumer.unsupported_dropped are recorded (probably expected that this is not supported, but mentioning it due to the testing instructions above)
otlp data are not indexed, as they are missing in the template definition. This is probably fine as no aggregations are currently done on the data.

axw added test-plan v7.12.0 labels Feb 11, 2021

axw marked this pull request as draft February 11, 2021 05:19

axw force-pushed the otel-metrics branch from e0fbf7b to ec30095 Compare February 11, 2021 05:21

axw force-pushed the otel-metrics branch from ec30095 to 4da9e41 Compare February 11, 2021 07:29

axw marked this pull request as ready for review February 11, 2021 08:50

axw requested a review from a team February 11, 2021 08:50

Merge branch 'master' into otel-metrics

8cd5920

jalvz reviewed Feb 11, 2021

View reviewed changes

processor/otel: add comment about [Cc]onsumerStats

fd54544

axw requested a review from jalvz February 11, 2021 12:57

jalvz approved these changes Feb 15, 2021

View reviewed changes

Merge branch 'master' into otel-metrics

53a9a74

axw merged commit 6d8ad81 into elastic:master Feb 16, 2021

axw deleted the otel-metrics branch February 16, 2021 02:56

This was referenced Feb 19, 2021

[7.x] Add support for consuming OTLP/gRPC metrics (#4722) #4825

Merged

[7.12] Add support for consuming OTLP/gRPC metrics (#4722) #4826

Merged

simitt self-assigned this Mar 2, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for consuming OTLP/gRPC metrics #4722

Add support for consuming OTLP/gRPC metrics #4722

axw commented Feb 11, 2021 •

edited

Loading

apmmachine commented Feb 11, 2021 •

edited

Loading

Build stats

Test stats 🧪

Trends 🧪

`Run Window tests`

`Compress`

`Compress`

`Test Sync`

codecov-io commented Feb 11, 2021 •

edited

Loading

jalvz Feb 11, 2021

axw Feb 11, 2021

jalvz Feb 11, 2021

axw Feb 11, 2021

jalvz Feb 11, 2021

axw Feb 11, 2021

axw commented Feb 15, 2021

simitt commented Mar 4, 2021

axw commented Mar 5, 2021

simitt commented Mar 5, 2021

Add support for consuming OTLP/gRPC metrics #4722

Add support for consuming OTLP/gRPC metrics #4722

Conversation

axw commented Feb 11, 2021 • edited Loading

Motivation/summary

Checklist

How to test these changes

Related issues

apmmachine commented Feb 11, 2021 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

Steps errors

Run Window tests

Compress

Compress

Test Sync

codecov-io commented Feb 11, 2021 • edited Loading

Codecov Report

jalvz Feb 11, 2021

Choose a reason for hiding this comment

axw Feb 11, 2021

Choose a reason for hiding this comment

jalvz Feb 11, 2021

Choose a reason for hiding this comment

axw Feb 11, 2021

Choose a reason for hiding this comment

jalvz Feb 11, 2021

Choose a reason for hiding this comment

axw Feb 11, 2021

Choose a reason for hiding this comment

axw commented Feb 15, 2021

simitt commented Mar 4, 2021

axw commented Mar 5, 2021

simitt commented Mar 5, 2021

axw commented Feb 11, 2021 •

edited

Loading

apmmachine commented Feb 11, 2021 •

edited

Loading

`Run Window tests`

`Compress`

`Compress`

`Test Sync`

codecov-io commented Feb 11, 2021 •

edited

Loading