[OpenTelemetry] OTLP import of counters and histograms don't work correctly #6185

vchirikov · 2021-09-16T10:44:48Z

APM Server version (apm-server version): apm-server version 7.14.0 (amd64), libbeat 7.14.0 [af26b5a744b6af9bcccb04fade6435241ab91f43 built 2021-07-29 19:55:24 +0000 UTC]

Description of the problem including expected versus actual behavior:

I tried to setup the following schema otel-collector => apm-server => es and ran into the problems with metrics.

Although the official doc says that otlp metric should work (except summary (#3195)), it doesn't work correctly. The counter is recorded, but always with zero value, histogram produces errors.

using (var meter = new Meter(ActivitySource.Name, _assemblyName.Version.ToString()))
{
    
    var counter = meter.CreateCounter<long>("mycounter");
    var histogram = meter.CreateHistogram<int>("my.histogram");
    for(int i = 0; i < 5; ++i){
        counter.Add(5);
        histogram.Record(5);
        await Task.Delay(300);
    }
}

collector config:

exporters:
logging:
    loglevel: debug
    sampling_initial: 2
    sampling_thereafter: 300
...
otlp/apm:
    endpoint: "apm-server:8200"
    insecure: true
    sending_queue:
    num_consumers: 4
    queue_size: 100
    retry_on_failure:
    enabled: true
...
pipelines:
    logs:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        # exporters: [elasticsearchexporter, logging]
        exporters: [logging]
    metrics:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [otlp/apm, logging]
    traces:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [otlp/apm, logging]

Example of record in index:

{
  "_index": "apm-7.14.0-metric-000001",
  "_type": "_doc",
  "_id": "YwP_7XsBVurXYf4s4HcT",
  "_score": 1,
  "_source": {
    "agent": {
      "name": "otlp",
      "version": "unknown"
    },
    "processor": {
      "name": "metric",
      "event": "metric"
    },
    "metricset.name": "app",
    "observer": {
      "hostname": "apm-server-5bd6d6484c-bjhq5",
      "id": "6d3d9ff9-bc84-49ec-bf94-f219339f83d0",
      "ephemeral_id": "c34c0a5f-baec-4b6c-a0d1-31148ed3cef1",
      "type": "apm-server",
      "version": "7.14.0",
      "version_major": 7
    },
    "@timestamp": "2021-09-16T09:46:10.527Z",
    "ecs": {
      "version": "1.10.0"
    },
    "service": {
      "node": {
        "name": "a0dab156-5d00-407f-afa4-51af92dcb37e"
      },
      "name": "linqpad_test",
      "language": {
        "name": "unknown"
      }
    },
    "mycounter": 0,
    "event": {
      "ingested": "2021-09-16T09:46:14.418773330Z"
    }
  },
  "fields": {
    "service.name": [
      "linqpad_test"
    ],
    "processor.name": [
      "metric"
    ],
    "service.node.name": [
      "a0dab156-5d00-407f-afa4-51af92dcb37e"
    ],
    "observer.version_major": [
      7
    ],
    "service.language.name": [
      "unknown"
    ],
    "observer.hostname": [
      "apm-server-5bd6d6484c-bjhq5"
    ],
    "metricset.name": [
      "app"
    ],
    "event.ingested": [
      "2021-09-16T09:46:14.418Z"
    ],
    "observer.id": [
      "6d3d9ff9-bc84-49ec-bf94-f219339f83d0"
    ],
    "@timestamp": [
      "2021-09-16T09:46:10.527Z"
    ],
    "observer.ephemeral_id": [
      "c34c0a5f-baec-4b6c-a0d1-31148ed3cef1"
    ],
    "observer.version": [
      "7.14.0"
    ],
    "ecs.version": [
      "1.10.0"
    ],
    "observer.type": [
      "apm-server"
    ],
    "processor.event": [
      "metric"
    ],
    "mycounter": [
      0
    ],
    "agent.name": [
      "otlp"
    ],
    "agent.version": [
      "unknown"
    ]
  }
}

And Otel-collector logs: (to be sure that it's collected correctly):

2021-09-16T09:46:13.412Z	INFO	loggingexporter/logging_exporter.go:56	MetricsExporter	{"#metrics": 18}
2021-09-16T09:46:13.412Z	DEBUG	loggingexporter/logging_exporter.go:66	ResourceMetrics    #0
Resource labels:
     -> service.name: STRING(linqpad_test)
     -> service.instance.id: STRING(a0dab156-5d00-407f-afa4-51af92dcb37e)
InstrumentationLibraryMetrics    #0
InstrumentationLibrary LINQPadQuery 1.0.0.520
Metric    #0
Descriptor:
     -> Name: mycounter
     -> Description: 
     -> Unit: 
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: AGGREGATION_TEMPORALITY_CUMULATIVE
NumberDataPoints    #0
StartTimestamp: 2021-09-16 09:46:07.2873705 +0000 UTC
Timestamp: 2021-09-16 09:46:07.3982794 +0000 UTC
Value: 5
Metric    #1
Descriptor:
     -> Name: my.histogram
     -> Description: 
     -> Unit: 
     -> DataType: Histogram
     -> AggregationTemporality: AGGREGATION_TEMPORALITY_CUMULATIVE
HistogramDataPoints    #0
StartTimestamp: 2021-09-16 09:46:07.2874285 +0000 UTC
Timestamp: 2021-09-16 09:46:07.3982813 +0000 UTC
Count: 1
Sum: 5.000000
ExplicitBounds    #0: 0.000000
ExplicitBounds    #1: 5.000000
ExplicitBounds    #2: 10.000000
ExplicitBounds    #3: 25.000000
ExplicitBounds    #4: 50.000000
ExplicitBounds    #5: 75.000000
ExplicitBounds    #6: 100.000000
ExplicitBounds    #7: 250.000000
ExplicitBounds    #8: 500.000000
ExplicitBounds    #9: 1000.000000
Buckets    #0, Count: 0
Buckets    #1, Count: 1
Buckets    #2, Count: 0
Buckets    #3, Count: 0
Buckets    #4, Count: 0
Buckets    #5, Count: 0
Buckets    #6, Count: 0
Buckets    #7, Count: 0
Buckets    #8, Count: 0
Buckets    #9, Count: 0
Buckets    #10, Count: 0
ResourceMetrics    #1
Resource labels:
     -> service.name: STRING(linqpad_test)
     -> service.instance.id: STRING(a0dab156-5d00-407f-afa4-51af92dcb37e)
InstrumentationLibraryMetrics    #0
InstrumentationLibrary LINQPadQuery 1.0.0.520
Metric    #0
Descriptor:
     -> Name: mycounter
     -> Description: 
     -> Unit: 
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: AGGREGATION_TEMPORALITY_CUMULATIVE
NumberDataPoints    #0
StartTimestamp: 2021-09-16 09:46:07.2873705 +0000 UTC
Timestamp: 2021-09-16 09:46:08.3275436 +0000 UTC
Value: 20
Metric    #1
Descriptor:
     -> Name: my.histogram
     -> Description: 
     -> Unit: 
     -> DataType: Histogram
     -> AggregationTemporality: AGGREGATION_TEMPORALITY_CUMULATIVE
HistogramDataPoints    #0
StartTimestamp: 2021-09-16 09:46:07.2874285 +0000 UTC
Timestamp: 2021-09-16 09:46:08.3275455 +0000 UTC
Count: 4
Sum: 20.000000
ExplicitBounds    #0: 0.000000
ExplicitBounds    #1: 5.000000
ExplicitBounds    #2: 10.000000
ExplicitBounds    #3: 25.000000
ExplicitBounds    #4: 50.000000
ExplicitBounds    #5: 75.000000
ExplicitBounds    #6: 100.000000
ExplicitBounds    #7: 250.000000
ExplicitBounds    #8: 500.000000
ExplicitBounds    #9: 1000.000000
Buckets    #0, Count: 0
Buckets    #1, Count: 4
Buckets    #2, Count: 0
Buckets    #3, Count: 0
Buckets    #4, Count: 0
Buckets    #5, Count: 0
Buckets    #6, Count: 0
Buckets    #7, Count: 0
Buckets    #8, Count: 0
Buckets    #9, Count: 0
Buckets    #10, Count: 0
ResourceMetrics    #2
Resource labels:
     -> service.name: STRING(linqpad_test)
     -> service.instance.id: STRING(a0dab156-5d00-407f-afa4-51af92dcb37e)
InstrumentationLibraryMetrics    #0
InstrumentationLibrary LINQPadQuery 1.0.0.520
Metric    #0
Descriptor:
     -> Name: mycounter
     -> Description: 
     -> Unit: 
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: AGGREGATION_TEMPORALITY_CUMULATIVE
NumberDataPoints    #0
StartTimestamp: 2021-09-16 09:46:07.2873705 +0000 UTC
Timestamp: 2021-09-16 09:46:08.6412742 +0000 UTC
Value: 25
Metric    #1
Descriptor:
     -> Name: my.histogram
     -> Description: 
     -> Unit: 
     -> DataType: Histogram
     -> AggregationTemporality: AGGREGATION_TEMPORALITY_CUMULATIVE
HistogramDataPoints    #0
StartTimestamp: 2021-09-16 09:46:07.2874285 +0000 UTC
Timestamp: 2021-09-16 09:46:08.6412753 +0000 UTC
Count: 5
Sum: 25.000000
ExplicitBounds    #0: 0.000000
ExplicitBounds    #1: 5.000000
ExplicitBounds    #2: 10.000000
ExplicitBounds    #3: 25.000000
ExplicitBounds    #4: 50.000000
ExplicitBounds    #5: 75.000000
ExplicitBounds    #6: 100.000000
ExplicitBounds    #7: 250.000000
ExplicitBounds    #8: 500.000000
ExplicitBounds    #9: 1000.000000
Buckets    #0, Count: 0
Buckets    #1, Count: 5
Buckets    #2, Count: 0
Buckets    #3, Count: 0
Buckets    #4, Count: 0
Buckets    #5, Count: 0
Buckets    #6, Count: 0
Buckets    #7, Count: 0
Buckets    #8, Count: 0
Buckets    #9, Count: 0
Buckets    #10, Count: 0
ResourceMetrics    #3
....

Steps to reproduce:

Send counter or histogram metric to the apm-server
Check apm the indices / logs.

logs:

I didn't find any apm logs for the counter. But found apm logs for the histogram:

{"log.level":"info","@timestamp":"2021-09-16T09:46:13.412Z","log.logger":"beater.grpc","log.origin":{"file.name":"interceptors/logging.go","file.line":65},"message":"","service.name":"apm-server","event.dataset":"apm-server","source.address":"10.23.201.39","grpc.request.method":"/opentelemetry.proto.collector.trace.v1.TraceService/Export","event.duration":111802,"grpc.response.status_code":"OK","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-09-16T09:46:13.413Z","log.logger":"beater.grpc","log.origin":{"file.name":"interceptors/logging.go","file.line":65},"message":"","service.name":"apm-server","event.dataset":"apm-server","source.address":"10.23.201.39","grpc.request.method":"/opentelemetry.proto.collector.metrics.v1.MetricsService/Export","event.duration":148402,"grpc.response.status_code":"OK","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2021-09-16T09:46:14.464Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":405},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x17bd4a54, ext:63767382367, loc:(*time.Location)(nil)}, Meta:{\"pipeline\":\"apm\"}, Fields:{\"_doc_count\":1,\"_metric_descriptions\":{\"my.histogram\":{\"type\":\"histogram\"}},\"agent\":{\"name\":\"otlp\",\"version\":\"unknown\"},\"ecs\":{\"version\":\"1.10.0\"},\"metricset.name\":\"app\",\"my\":{\"histogram\":{\"counts\":[1],\"values\":[2.5]}},\"observer\":{\"ephemeral_id\":\"c34c0a5f-baec-4b6c-a0d1-31148ed3cef1\",\"hostname\":\"apm-server-5bd6d6484c-bjhq5\",\"id\":\"6d3d9ff9-bc84-49ec-bf94-f219339f83d0\",\"type\":\"apm-server\",\"version\":\"7.14.0\",\"version_major\":7},\"processor\":{\"event\":\"metric\",\"name\":\"metric\"},\"service\":{\"language\":{\"name\":\"unknown\"},\"name\":\"linqpad_test\",\"node\":{\"name\":\"a0dab156-5d00-407f-afa4-51af92dcb37e\"}}}, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {\"type\":\"mapper_parsing_exception\",\"reason\":\"Can't find dynamic template for dynamic template name [histogram] of field [my.histogram]\"}","service.name":"apm-server","event.dataset":"apm-server","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2021-09-16T09:46:14.464Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":405},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x1385f29c, ext:63767382368, loc:(*time.Location)(nil)}, Meta:{\"pipeline\":\"apm\"}, Fields:{\"_doc_count\":4,\"_metric_descriptions\":{\"my.histogram\":{\"type\":\"histogram\"}},\"agent\":{\"name\":\"otlp\",\"version\":\"unknown\"},\"ecs\":{\"version\":\"1.10.0\"},\"metricset.name\":\"app\",\"my\":{\"histogram\":{\"counts\":[4],\"values\":[2.5]}},\"observer\":{\"ephemeral_id\":\"c34c0a5f-baec-4b6c-a0d1-31148ed3cef1\",\"hostname\":\"apm-server-5bd6d6484c-bjhq5\",\"id\":\"6d3d9ff9-bc84-49ec-bf94-f219339f83d0\",\"type\":\"apm-server\",\"version\":\"7.14.0\",\"version_major\":7},\"processor\":{\"event\":\"metric\",\"name\":\"metric\"},\"service\":{\"language\":{\"name\":\"unknown\"},\"name\":\"linqpad_test\",\"node\":{\"name\":\"a0dab156-5d00-407f-afa4-51af92dcb37e\"}}}, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {\"type\":\"mapper_parsing_exception\",\"reason\":\"Can't find dynamic template for dynamic template name [histogram] of field [my.histogram]\"}","service.name":"apm-server","event.dataset":"apm-server","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2021-09-16T09:46:14.464Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":405},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x263915a4, ext:63767382368, loc:(*time.Location)(nil)}, Meta:{\"pipeline\":\"apm\"}, Fields:{\"_doc_count\":5,\"_metric_descriptions\":{\"my.histogram\":{\"type\":\"histogram\"}},\"agent\":{\"name\":\"otlp\",\"version\":\"unknown\"},\"ecs\":{\"version\":\"1.10.0\"},\"metricset.name\":\"app\",\"my\":{\"histogram\":{\"counts\":[5],\"values\":[2.5]}},\"observer\":{\"ephemeral_id\":\"c34c0a5f-baec-4b6c-a0d1-31148ed3cef1\",\"hostname\":\"apm-server-5bd6d6484c-bjhq5\",\"id\":\"6d3d9ff9-bc84-49ec-bf94-f219339f83d0\",\"type\":\"apm-server\",\"version\":\"7.14.0\",\"version_major\":7},\"processor\":{\"event\":\"metric\",\"name\":\"metric\"},\"service\":{\"language\":{\"name\":\"unknown\"},\"name\":\"linqpad_test\",\"node\":{\"name\":\"a0dab156-5d00-407f-afa4-51af92dcb37e\"}}}, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {\"type\":\"mapper_parsing_exception\",\"reason\":\"Can't find dynamic template for dynamic template name [histogram] of field [my.histogram]\"}","service.name":"apm-server","event.dataset":"apm-server","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2021-09-16T09:46:14.464Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":405},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x38e73ce4, ext:63767382368, loc:(*time.Location)(nil)}, Meta:{\"pipeline\":\"apm\"}, Fields:{\"_doc_count\":5,\"_metric_descriptions\":{\"my.histogram\":{\"type\":\"histogram\"}},\"agent\":{\"name\":\"otlp\",\"version\":\"unknown\"},\"ecs\":{\"version\":\"1.10.0\"},\"metricset.name\":\"app\",\"my\":{\"histogram\":{\"counts\":[5],\"values\":[2.5]}},\"observer\":{\"ephemeral_id\":\"c34c0a5f-baec-4b6c-a0d1-31148ed3cef1\",\"hostname\":\"apm-server-5bd6d6484c-bjhq5\",\"id\":\"6d3d9ff9-bc84-49ec-bf94-f219339f83d0\",\"type\":\"apm-server\",\"version\":\"7.14.0\",\"version_major\":7},\"processor\":{\"event\":\"metric\",\"name\":\"metric\"},\"service\":{\"language\":{\"name\":\"unknown\"},\"name\":\"linqpad_test\",\"node\":{\"name\":\"a0dab156-5d00-407f-afa4-51af92dcb37e\"}}}, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {\"type\":\"mapper_parsing_exception\",\"reason\":\"Can't find dynamic template for dynamic template name [histogram] of field [my.histogram]\"}","service.name":"apm-server","event.dataset":"apm-server","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2021-09-16T09:46:14.464Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":405},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x10222788, ext:63767382369, loc:(*time.Location)(nil)}, Meta:{\"pipeline\":\"apm\"}, Fields:{\"_doc_count\":5,\"_metric_descriptions\":{\"my.histogram\":{\"type\":\"histogram\"}},\"agent\":{\"name\":\"otlp\",\"version\":\"unknown\"},\"ecs\":{\"version\":\"1.10.0\"},\"metricset.name\":\"app\",\"my\":{\"histogram\":{\"counts\":[5],\"values\":[2.5]}},\"observer\":{\"ephemeral_id\":\"c34c0a5f-baec-4b6c-a0d1-31148ed3cef1\",\"hostname\":\"apm-server-5bd6d6484c-bjhq5\",\"id\":\"6d3d9ff9-bc84-49ec-bf94-f219339f83d0\",\"type\":\"apm-server\",\"version\":\"7.14.0\",\"version_major\":7},\"processor\":{\"event\":\"metric\",\"name\":\"metric\"},\"service\":{\"language\":{\"name\":\"unknown\"},\"name\":\"linqpad_test\",\"node\":{\"name\":\"a0dab156-5d00-407f-afa4-51af92dcb37e\"}}}, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {\"type\":\"mapper_parsing_exception\",\"reason\":\"Can't find dynamic template for dynamic template name [histogram] of field [my.histogram]\"}","service.name":"apm-server","event.dataset":"apm-server","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2021-09-16T09:46:14.464Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":405},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x22ca6328, ext:63767382369, loc:(*time.Location)(nil)}, Meta:{\"pipeline\":\"apm\"}, Fields:{\"_doc_count\":5,\"_metric_descriptions\":{\"my.histogram\":{\"type\":\"histogram\"}},\"agent\":{\"name\":\"otlp\",\"version\":\"unknown\"},\"ecs\":{\"version\":\"1.10.0\"},\"metricset.name\":\"app\",\"my\":{\"histogram\":{\"counts\":[5],\"values\":[2.5]}},\"observer\":{\"ephemeral_id\":\"c34c0a5f-baec-4b6c-a0d1-31148ed3cef1\",\"hostname\":\"apm-server-5bd6d6484c-bjhq5\",\"id\":\"6d3d9ff9-bc84-49ec-bf94-f219339f83d0\",\"type\":\"apm-server\",\"version\":\"7.14.0\",\"version_major\":7},\"processor\":{\"event\":\"metric\",\"name\":\"metric\"},\"service\":{\"language\":{\"name\":\"unknown\"},\"name\":\"linqpad_test\",\"node\":{\"name\":\"a0dab156-5d00-407f-afa4-51af92dcb37e\"}}}, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {\"type\":\"mapper_parsing_exception\",\"reason\":\"Can't find dynamic template for dynamic template name [histogram] of field [my.histogram]\"}","service.name":"apm-server","event.dataset":"apm-server","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2021-09-16T09:46:14.464Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":405},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x358c9cb0, ext:63767382369, loc:(*time.Location)(nil)}, Meta:{\"pipeline\":\"apm\"}, Fields:{\"_doc_count\":5,\"_metric_descriptions\":{\"my.histogram\":{\"type\":\"histogram\"}},\"agent\":{\"name\":\"otlp\",\"version\":\"unknown\"},\"ecs\":{\"version\":\"1.10.0\"},\"metricset.name\":\"app\",\"my\":{\"histogram\":{\"counts\":[5],\"values\":[2.5]}},\"observer\":{\"ephemeral_id\":\"c34c0a5f-baec-4b6c-a0d1-31148ed3cef1\",\"hostname\":\"apm-server-5bd6d6484c-bjhq5\",\"id\":\"6d3d9ff9-bc84-49ec-bf94-f219339f83d0\",\"type\":\"apm-server\",\"version\":\"7.14.0\",\"version_major\":7},\"processor\":{\"event\":\"metric\",\"name\":\"metric\"},\"service\":{\"language\":{\"name\":\"unknown\"},\"name\":\"linqpad_test\",\"node\":{\"name\":\"a0dab156-5d00-407f-afa4-51af92dcb37e\"}}}, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {\"type\":\"mapper_parsing_exception\",\"reason\":\"Can't find dynamic template for dynamic template name [histogram] of field [my.histogram]\"}","service.name":"apm-server","event.dataset":"apm-server","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2021-09-16T09:46:14.465Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":405},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xca09d38, ext:63767382370, loc:(*time.Location)(nil)}, Meta:{\"pipeline\":\"apm\"}, Fields:{\"_doc_count\":5,\"_metric_descriptions\":{\"my.histogram\":{\"type\":\"histogram\"}},\"agent\":{\"name\":\"otlp\",\"version\":\"unknown\"},\"ecs\":{\"version\":\"1.10.0\"},\"metricset.name\":\"app\",\"my\":{\"histogram\":{\"counts\":[5],\"values\":[2.5]}},\"observer\":{\"ephemeral_id\":\"c34c0a5f-baec-4b6c-a0d1-31148ed3cef1\",\"hostname\":\"apm-server-5bd6d6484c-bjhq5\",\"id\":\"6d3d9ff9-bc84-49ec-bf94-f219339f83d0\",\"type\":\"apm-server\",\"version\":\"7.14.0\",\"version_major\":7},\"processor\":{\"event\":\"metric\",\"name\":\"metric\"},\"service\":{\"language\":{\"name\":\"unknown\"},\"name\":\"linqpad_test\",\"node\":{\"name\":\"a0dab156-5d00-407f-afa4-51af92dcb37e\"}}}, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {\"type\":\"mapper_parsing_exception\",\"reason\":\"Can't find dynamic template for dynamic template name [histogram] of field [my.histogram]\"}","service.name":"apm-server","event.dataset":"apm-server","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2021-09-16T09:46:14.465Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":405},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x1f6c3d00, ext:63767382370, loc:(*time.Location)(nil)}, Meta:{\"pipeline\":\"apm\"}, Fields:{\"_doc_count\":5,\"_metric_descriptions\":{\"my.histogram\":{\"type\":\"histogram\"}},\"agent\":{\"name\":\"otlp\",\"version\":\"unknown\"},\"ecs\":{\"version\":\"1.10.0\"},\"metricset.name\":\"app\",\"my\":{\"histogram\":{\"counts\":[5],\"values\":[2.5]}},\"observer\":{\"ephemeral_id\":\"c34c0a5f-baec-4b6c-a0d1-31148ed3cef1\",\"hostname\":\"apm-server-5bd6d6484c-bjhq5\",\"id\":\"6d3d9ff9-bc84-49ec-bf94-f219339f83d0\",\"type\":\"apm-server\",\"version\":\"7.14.0\",\"version_major\":7},\"processor\":{\"event\":\"metric\",\"name\":\"metric\"},\"service\":{\"language\":{\"name\":\"unknown\"},\"name\":\"linqpad_test\",\"node\":{\"name\":\"a0dab156-5d00-407f-afa4-51af92dcb37e\"}}}, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {\"type\":\"mapper_parsing_exception\",\"reason\":\"Can't find dynamic template for dynamic template name [histogram] of field [my.histogram]\"}","service.name":"apm-server","event.dataset":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-09-16T09:46:15.229Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":63},"message":"request ok","service.name":"apm-server","event.dataset":"apm-server","url.original":"/","http.request.method":"GET","user_agent.original":"kube-probe/1.19","source.address":"10.23.198.177","http.request.body.bytes":0,"http.request.id":"a2d97a65-b0ba-4163-b5b9-25a7650467f3","event.duration":106402,"http.response.status_code":200,"ecs.version":"1.6.0"}

Could you look at this?

cc: @cyrille-leclerc , @axw

The text was updated successfully, but these errors were encountered:

axw · 2021-09-16T10:51:44Z

This zero value issue looks similar to #6008, which will be fixed in 7.15.0. That should be available soon.

I'm not sure why the error Can't find dynamic template for dynamic template name [histogram] of field [my.histogram]\" would be appearing though. How did you install APM Server?

vchirikov · 2021-09-16T10:56:55Z

I installed apm with helm-chart version: 7.14.0
the apmConfig is here:

apmConfig:
      # https://www.elastic.co/guide/en/apm/server/7.14
      apm-server.yml: |
        setup.template.name: "apm-server"
        setup.template.pattern: "apm-*"
        setup.template.overwrite: false
        setup.template.settings:
          index.number_of_shards: 3
          index.number_of_replicas: 0
        apm-server:
          host: "0.0.0.0:8200"
          ilm:
            enabled: "true"
            setup:
              enabled: true
              overwrite: false
          rum:
            enabled: true
          kibana:
            enabled: true
            host: "http://kibana:5601"
        queue: {}
        output.elasticsearch:
          hosts: ["http://elasticsearch-master:9200"]

Do you need anything else?

p.s. the counter issue is similar to #5960

axw · 2021-09-20T02:12:13Z

Do you need anything else?

No, thank you. 7.15.0 should be available very soon. I'll ping when it is, and ask you to test again. I expect the issue will be resolved by the upgrade.

riferrei · 2021-09-21T22:33:58Z

I have noticed similar behavior in an OTel application that I wrote in Go—looking forward to seeing the fix for this.

axw · 2021-09-23T00:52:29Z

@vchirikov 7.15.0 has been released. Can you please try again?

leewoobin789 · 2021-09-24T10:57:53Z

open-telemetry/opentelemetry-collector-contrib#4432

Hello guys. i was the guy who opened this issue on opentelemetry side. it seems to be already resolved with the newest version of 7.15.0.

vchirikov · 2021-09-25T13:49:50Z

@vchirikov 7.15.0 has been released. Can you please try again?

I will wait for helm chart update (it's on 7.14 at the moment)

axw · 2021-09-26T06:33:16Z

@leewoobin789 thanks for the update. I'll close #5960 then.

axw · 2021-10-07T09:53:56Z

@vchirikov looks like 7.15.0 Helm charts are available now. I'll close this issue as I'm pretty confident it's fixed; please reopen it if you still have issues after upgrading to 7.15.0. Thanks!

vchirikov added the bug label Sep 16, 2021

zube bot added the [zube]: Blocked label Oct 6, 2021

leewoobin789 mentioned this issue Oct 7, 2021

Value of kafkametricsreceiver not correctly transformed on otlp/elastic open-telemetry/opentelemetry-collector-contrib#4432

Closed

axw closed this as completed Oct 7, 2021

zube bot added [zube]: Done and removed [zube]: Blocked labels Oct 7, 2021

cyrille-leclerc added the OpenTelemetry label Nov 16, 2021

zube bot removed the [zube]: Done label Jan 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[OpenTelemetry] OTLP import of counters and histograms don't work correctly #6185

[OpenTelemetry] OTLP import of counters and histograms don't work correctly #6185

vchirikov commented Sep 16, 2021 •

edited by zube bot

Loading

axw commented Sep 16, 2021

vchirikov commented Sep 16, 2021

axw commented Sep 20, 2021

riferrei commented Sep 21, 2021

axw commented Sep 23, 2021

leewoobin789 commented Sep 24, 2021

vchirikov commented Sep 25, 2021

axw commented Sep 26, 2021

axw commented Oct 7, 2021

[OpenTelemetry] OTLP import of counters and histograms don't work correctly #6185

[OpenTelemetry] OTLP import of counters and histograms don't work correctly #6185

Comments

vchirikov commented Sep 16, 2021 • edited by zube bot Loading

axw commented Sep 16, 2021

vchirikov commented Sep 16, 2021

axw commented Sep 20, 2021

riferrei commented Sep 21, 2021

axw commented Sep 23, 2021

leewoobin789 commented Sep 24, 2021

vchirikov commented Sep 25, 2021

axw commented Sep 26, 2021

axw commented Oct 7, 2021

vchirikov commented Sep 16, 2021 •

edited by zube bot

Loading