Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Metric queueSize twice fails with opentelemetry collector with a prometheus metrics exporter #18194

Closed
cmunger opened this issue Jan 31, 2023 · 9 comments
Closed

Metric queueSize twice fails with opentelemetry collector with a prometheus metrics exporter #18194

cmunger opened this issue Jan 31, 2023 · 9 comments
Labels
bug Something isn't working closed as inactive exporter/prometheus Stale

Comments

@cmunger
Copy link

cmunger commented Jan 31, 2023 •
edited
Loading

Component(s)

exporter/prometheus

What happened?

Describe the bug

When setting up an opentelemetry collector with a metrics prometheus exporter, the exporter will fail when called
if the monitored application is a java application using a opentelemetry-log4j-appender-2.17 instrumentation
since the application has 2 metrics named queueSize one for the logs and of for the spans processor

Steps to reproduce
A java application using opentelementry java agent log4j2 instrumentation :

io.opentelemetry.instrumentation
opentelemetry-log4j-appender-2.17
1.21.0-alpha
runtime

Java Application must do in loop log statements every seconds:

public static void main(String[] args) throws InterruptedException {
	while (true) {
		LoggerFactory.getLogger("root").info("TestLog");
		LockSupport.parkNanos(TimeUnit.SECONDS.toNanos(1));
	}
}

start the collector :
docker run -p 4317:4317 -p 9464:9464 -v $(pwd)/otel-collector.yaml:/etc/otelcol/config.yaml otel/opentelemetry-collector
start the java application
call http://localhost:9464/metrics will trigger the bug in the opentelemetry collector and will produce an half populated prometheus output file

What did you expect to see?
A prometheus output file with output :

# TYPE queueSize gauge
# HELP queueSize The number of spans queued,The number of logs queued
queueSize{spanProcessorType="BatchSpanProcessor"} 0
queueSize{logRecordProcessorType="BatchLogRecordProcessor"} 0

What did you see instead?
A badly generated prometheus output file with missing queueSize{spanProcessorType="BatchSpanProcessor"} 0

# HELP default_queueSize The number of logs queued
# TYPE default_queueSize gauge
default_queueSize{logRecordProcessorType="BatchLogRecordProcessor"} 0

What version and what artifacts are you using?
oentelemetry-collector 0.68, 0.69.0.70
opentelemetry-java-agent 1.21.0

Collector version

0.68,0.69,0.70

Environment information

OS: Ubuntu 20.04

OpenTelemetry Collector configuration

receivers:
  otlp/2:
    protocols:
      grpc:

processors:
  batch:

exporters:
  otlphttp:
    endpoint: http://localhost:8080
  prometheus:
    endpoint: "0.0.0.0:9464"
    namespace: "default"

service:
  pipelines:
    traces:
      receivers: [ otlp/2 ]
      processors: [ batch ]
      exporters: [ otlphttp ]
    metrics:
      receivers: [ otlp/2 ]
      processors: [ batch ]
      exporters: [ prometheus ]
    logs:
      receivers: [ otlp/2 ]
      processors: [ batch ]
      exporters: [ otlphttp ]

Log output

2023-02-01T11:19:41.663Z	info	service/telemetry.go:90	Setting up own telemetry...
2023-02-01T11:19:41.663Z	info	service/telemetry.go:116	Serving Prometheus metrics	{"address": ":8888", "level": "Basic"}
2023-02-01T11:19:41.665Z	info	exporter/exporter.go:290	Development component. May change in the future.	{"kind": "exporter", "data_type": "metrics", "name": "logging"}
2023-02-01T11:19:41.665Z	warn	[email protected]/factory.go:109	'loglevel' option is deprecated in favor of 'verbosity'. Set 'verbosity' to equivalent value to preserve behavior.	{"kind": "exporter", "data_type": "metrics", "name": "logging", "loglevel": "debug", "equivalent verbosity level": "Detailed"}
2023-02-01T11:19:41.666Z	info	service/service.go:128	Starting otelcol...	{"Version": "0.70.0", "NumCPU": 8}
2023-02-01T11:19:41.666Z	info	extensions/extensions.go:41	Starting extensions...
2023-02-01T11:19:41.666Z	info	extensions/extensions.go:44	Extension is starting...	{"kind": "extension", "name": "pprof"}
2023-02-01T11:19:41.666Z	info	[email protected]/pprofextension.go:71	Starting net/http/pprof server	{"kind": "extension", "name": "pprof", "config": {"TCPAddr":{"Endpoint":"localhost:1777"},"BlockProfileFraction":0,"MutexProfileFraction":0,"SaveToFile":""}}
2023-02-01T11:19:41.666Z	info	extensions/extensions.go:48	Extension started.	{"kind": "extension", "name": "pprof"}
2023-02-01T11:19:41.666Z	info	extensions/extensions.go:44	Extension is starting...	{"kind": "extension", "name": "zpages"}
2023-02-01T11:19:41.666Z	info	[email protected]/zpagesextension.go:64	Registered zPages span processor on tracer provider	{"kind": "extension", "name": "zpages"}
2023-02-01T11:19:41.666Z	info	[email protected]/zpagesextension.go:74	Registered Host's zPages	{"kind": "extension", "name": "zpages"}
2023-02-01T11:19:41.666Z	info	[email protected]/zpagesextension.go:86	Starting zPages extension	{"kind": "extension", "name": "zpages", "config": {"TCPAddr":{"Endpoint":"localhost:55679"}}}
2023-02-01T11:19:41.666Z	info	extensions/extensions.go:48	Extension started.	{"kind": "extension", "name": "zpages"}
2023-02-01T11:19:41.666Z	info	extensions/extensions.go:44	Extension is starting...	{"kind": "extension", "name": "health_check"}
2023-02-01T11:19:41.666Z	info	[email protected]/healthcheckextension.go:45	Starting health_check extension	{"kind": "extension", "name": "health_check", "config": {"Endpoint":"0.0.0.0:13133","TLSSetting":null,"CORS":null,"Auth":null,"MaxRequestBodySize":0,"IncludeMetadata":false,"Path":"/","CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
2023-02-01T11:19:41.666Z	warn	internal/warning.go:51	Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks	{"kind": "extension", "name": "health_check", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2023-02-01T11:19:41.667Z	info	extensions/extensions.go:48	Extension started.	{"kind": "extension", "name": "health_check"}
2023-02-01T11:19:41.667Z	info	service/pipelines.go:86	Starting exporters...
2023-02-01T11:19:41.667Z	info	service/pipelines.go:90	Exporter is starting...	{"kind": "exporter", "data_type": "logs", "name": "otlphttp"}
2023-02-01T11:19:41.667Z	info	service/pipelines.go:94	Exporter started.	{"kind": "exporter", "data_type": "logs", "name": "otlphttp"}
2023-02-01T11:19:41.667Z	info	service/pipelines.go:90	Exporter is starting...	{"kind": "exporter", "data_type": "traces", "name": "otlphttp"}
2023-02-01T11:19:41.667Z	info	service/pipelines.go:94	Exporter started.	{"kind": "exporter", "data_type": "traces", "name": "otlphttp"}
2023-02-01T11:19:41.667Z	info	service/pipelines.go:90	Exporter is starting...	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-02-01T11:19:41.667Z	warn	internal/warning.go:51	Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks	{"kind": "exporter", "data_type": "metrics", "name": "prometheus", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2023-02-01T11:19:41.667Z	info	service/pipelines.go:94	Exporter started.	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-02-01T11:19:41.667Z	info	service/pipelines.go:90	Exporter is starting...	{"kind": "exporter", "data_type": "metrics", "name": "logging"}
2023-02-01T11:19:41.667Z	info	service/pipelines.go:94	Exporter started.	{"kind": "exporter", "data_type": "metrics", "name": "logging"}
2023-02-01T11:19:41.667Z	info	service/pipelines.go:98	Starting processors...
2023-02-01T11:19:41.667Z	info	service/pipelines.go:102	Processor is starting...	{"kind": "processor", "name": "batch", "pipeline": "logs"}
2023-02-01T11:19:41.667Z	info	service/pipelines.go:106	Processor started.	{"kind": "processor", "name": "batch", "pipeline": "logs"}
2023-02-01T11:19:41.667Z	info	service/pipelines.go:102	Processor is starting...	{"kind": "processor", "name": "batch", "pipeline": "traces"}
2023-02-01T11:19:41.667Z	info	service/pipelines.go:106	Processor started.	{"kind": "processor", "name": "batch", "pipeline": "traces"}
2023-02-01T11:19:41.667Z	info	service/pipelines.go:102	Processor is starting...	{"kind": "processor", "name": "batch", "pipeline": "metrics"}
2023-02-01T11:19:41.667Z	info	service/pipelines.go:106	Processor started.	{"kind": "processor", "name": "batch", "pipeline": "metrics"}
2023-02-01T11:19:41.667Z	info	service/pipelines.go:110	Starting receivers...
2023-02-01T11:19:41.667Z	info	service/pipelines.go:114	Receiver is starting...	{"kind": "receiver", "name": "otlp/2", "pipeline": "traces"}
2023-02-01T11:19:41.667Z	warn	internal/warning.go:51	Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks	{"kind": "receiver", "name": "otlp/2", "pipeline": "traces", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2023-02-01T11:19:41.667Z	info	[email protected]/otlp.go:94	Starting GRPC server	{"kind": "receiver", "name": "otlp/2", "pipeline": "traces", "endpoint": "0.0.0.0:4317"}
2023-02-01T11:19:41.667Z	info	service/pipelines.go:118	Receiver started.	{"kind": "receiver", "name": "otlp/2", "pipeline": "traces"}
2023-02-01T11:19:41.667Z	info	service/pipelines.go:114	Receiver is starting...	{"kind": "receiver", "name": "otlp/2", "pipeline": "metrics"}
2023-02-01T11:19:41.667Z	info	service/pipelines.go:118	Receiver started.	{"kind": "receiver", "name": "otlp/2", "pipeline": "metrics"}
2023-02-01T11:19:41.667Z	info	service/pipelines.go:114	Receiver is starting...	{"kind": "receiver", "name": "otlp/2", "pipeline": "logs"}
2023-02-01T11:19:41.667Z	info	service/pipelines.go:118	Receiver started.	{"kind": "receiver", "name": "otlp/2", "pipeline": "logs"}
2023-02-01T11:19:41.667Z	info	healthcheck/handler.go:129	Health Check state change	{"kind": "extension", "name": "health_check", "status": "ready"}
2023-02-01T11:19:41.667Z	info	service/service.go:145	Everything is ready. Begin running and processing data.
2023-02-01T11:19:54.693Z	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging", "#metrics": 2}
2023-02-01T11:19:54.693Z	info	ResourceMetrics #0
Resource SchemaURL: https://opentelemetry.io/schemas/1.16.0
Resource attributes:
     -> container.id: Str(2164232)
     -> deployment.environment: Str(pre)
     -> host.arch: Str(amd64)
     -> host.name: Str(cmunger-HP-Compaq-Elite-8300-CMT)
     -> os.description: Str(Linux 5.4.0-137-generic)
     -> os.type: Str(linux)
     -> process.command_line: Str(/usr/lib/jvm/zulu11-ca-amd64/bin/java -javaagent:./opentelemetry-javaagent.jar -Dotel.javaagent.configuration-file=./opentelemetry-one.properties -javaagent:/home/cmunger/dev/idea-IC-221.5080.210/lib/idea_rt.jar=39569:/home/cmunger/dev/idea-IC-221.5080.210/bin -Dfile.encoding=UTF-8)
     -> process.executable.path: Str(/usr/lib/jvm/zulu11-ca-amd64/bin/java)
     -> process.pid: Int(2186394)
     -> process.runtime.description: Str(Azul Systems, Inc. OpenJDK 64-Bit Server VM 11.0.17+8-LTS)
     -> process.runtime.name: Str(OpenJDK Runtime Environment)
     -> process.runtime.version: Str(11.0.17+8-LTS)
     -> service.name: Str(monitoring-poc-one)
     -> service.namespace: Str(poc)
     -> service.version: Str(1.1)
     -> telemetry.auto.version: Str(1.21.0)
     -> telemetry.sdk.language: Str(java)
     -> telemetry.sdk.name: Str(opentelemetry)
     -> telemetry.sdk.version: Str(1.21.0)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope io.opentelemetry.sdk.trace 
Metric #0
Descriptor:
     -> Name: queueSize
     -> Description: The number of spans queued
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> spanProcessorType: Str(BatchSpanProcessor)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:54.453442 +0000 UTC
Value: 0
ScopeMetrics #1
ScopeMetrics SchemaURL: 
InstrumentationScope io.opentelemetry.sdk.logs 
Metric #0
Descriptor:
     -> Name: queueSize
     -> Description: The number of logs queued
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> logRecordProcessorType: Str(BatchLogRecordProcessor)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:54.453442 +0000 UTC
Value: 0
	{"kind": "exporter", "data_type": "metrics", "name": "logging"}
2023-02-01T11:19:55.495Z	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging", "#metrics": 17}
2023-02-01T11:19:55.495Z	info	ResourceMetrics #0
Resource SchemaURL: https://opentelemetry.io/schemas/1.16.0
Resource attributes:
     -> container.id: Str(2164232)
     -> deployment.environment: Str(pre)
     -> host.arch: Str(amd64)
     -> host.name: Str(cmunger-HP-Compaq-Elite-8300-CMT)
     -> os.description: Str(Linux 5.4.0-137-generic)
     -> os.type: Str(linux)
     -> process.command_line: Str(/usr/lib/jvm/zulu11-ca-amd64/bin/java -javaagent:./opentelemetry-javaagent.jar -Dotel.javaagent.configuration-file=./opentelemetry-one.properties -javaagent:/home/cmunger/dev/idea-IC-221.5080.210/lib/idea_rt.jar=39569:/home/cmunger/dev/idea-IC-221.5080.210/bin -Dfile.encoding=UTF-8)
     -> process.executable.path: Str(/usr/lib/jvm/zulu11-ca-amd64/bin/java)
     -> process.pid: Int(2186394)
     -> process.runtime.description: Str(Azul Systems, Inc. OpenJDK 64-Bit Server VM 11.0.17+8-LTS)
     -> process.runtime.name: Str(OpenJDK Runtime Environment)
     -> process.runtime.version: Str(11.0.17+8-LTS)
     -> service.name: Str(monitoring-poc-one)
     -> service.namespace: Str(poc)
     -> service.version: Str(1.1)
     -> telemetry.auto.version: Str(1.21.0)
     -> telemetry.sdk.language: Str(java)
     -> telemetry.sdk.name: Str(opentelemetry)
     -> telemetry.sdk.version: Str(1.21.0)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope io.opentelemetry.sdk.trace 
Metric #0
Descriptor:
     -> Name: queueSize
     -> Description: The number of spans queued
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> spanProcessorType: Str(BatchSpanProcessor)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 0
ScopeMetrics #1
ScopeMetrics SchemaURL: 
InstrumentationScope io.opentelemetry.runtime-metrics 1.21.0-alpha
Metric #0
Descriptor:
     -> Name: process.runtime.jvm.buffer.count
     -> Description: The number of buffers in the pool
     -> Unit: {buffers}
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(mapped)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 0
NumberDataPoints #1
Data point attributes:
     -> pool: Str(direct)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 1
Metric #1
Descriptor:
     -> Name: process.runtime.jvm.classes.unloaded
     -> Description: Number of classes unloaded since JVM start
     -> Unit: 1
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 0
Metric #2
Descriptor:
     -> Name: process.runtime.jvm.classes.current_loaded
     -> Description: Number of classes currently loaded
     -> Unit: 1
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 5426
Metric #3
Descriptor:
     -> Name: process.runtime.jvm.buffer.usage
     -> Description: Memory that the Java virtual machine is using for this buffer pool
     -> Unit: By
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(mapped)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 0
NumberDataPoints #1
Data point attributes:
     -> pool: Str(direct)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 8192
Metric #4
Descriptor:
     -> Name: process.runtime.jvm.threads.count
     -> Description: Number of executing threads
     -> Unit: 1
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> daemon: Bool(false)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 1
NumberDataPoints #1
Data point attributes:
     -> daemon: Bool(true)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 16
Metric #5
Descriptor:
     -> Name: process.runtime.jvm.memory.usage_after_last_gc
     -> Description: Measure of memory used after the most recent garbage collection event on this pool
     -> Unit: By
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(G1 Old Gen)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 0
NumberDataPoints #1
Data point attributes:
     -> pool: Str(G1 Eden Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 0
NumberDataPoints #2
Data point attributes:
     -> pool: Str(G1 Survivor Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 10485760
Metric #6
Descriptor:
     -> Name: process.runtime.jvm.memory.limit
     -> Description: Measure of max obtainable memory
     -> Unit: By
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(Compressed Class Space)
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 1073741824
NumberDataPoints #1
Data point attributes:
     -> pool: Str(CodeHeap 'profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 122908672
NumberDataPoints #2
Data point attributes:
     -> pool: Str(G1 Old Gen)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 8405385216
NumberDataPoints #3
Data point attributes:
     -> pool: Str(CodeHeap 'non-profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 122912768
NumberDataPoints #4
Data point attributes:
     -> pool: Str(CodeHeap 'non-nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 5836800
Metric #7
Descriptor:
     -> Name: process.runtime.jvm.memory.usage
     -> Description: Measure of memory used
     -> Unit: By
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(Metaspace)
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 34584624
NumberDataPoints #1
Data point attributes:
     -> pool: Str(Compressed Class Space)
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 3931568
NumberDataPoints #2
Data point attributes:
     -> pool: Str(CodeHeap 'profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 8812928
NumberDataPoints #3
Data point attributes:
     -> pool: Str(G1 Old Gen)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 5678240
NumberDataPoints #4
Data point attributes:
     -> pool: Str(CodeHeap 'non-profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 2079616
NumberDataPoints #5
Data point attributes:
     -> pool: Str(G1 Eden Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 71303168
NumberDataPoints #6
Data point attributes:
     -> pool: Str(CodeHeap 'non-nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 1271808
NumberDataPoints #7
Data point attributes:
     -> pool: Str(G1 Survivor Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 10485760
Metric #8
Descriptor:
     -> Name: process.runtime.jvm.memory.init
     -> Description: Measure of initial memory requested
     -> Unit: By
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(Metaspace)
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 0
NumberDataPoints #1
Data point attributes:
     -> pool: Str(Compressed Class Space)
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 0
NumberDataPoints #2
Data point attributes:
     -> pool: Str(CodeHeap 'profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 2555904
NumberDataPoints #3
Data point attributes:
     -> pool: Str(G1 Old Gen)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 499122176
NumberDataPoints #4
Data point attributes:
     -> pool: Str(CodeHeap 'non-profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 2555904
NumberDataPoints #5
Data point attributes:
     -> pool: Str(G1 Eden Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 27262976
NumberDataPoints #6
Data point attributes:
     -> pool: Str(CodeHeap 'non-nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 2555904
NumberDataPoints #7
Data point attributes:
     -> pool: Str(G1 Survivor Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 0
Metric #9
Descriptor:
     -> Name: process.runtime.jvm.classes.loaded
     -> Description: Number of classes loaded since JVM start
     -> Unit: 1
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 5426
Metric #10
Descriptor:
     -> Name: process.runtime.jvm.memory.committed
     -> Description: Measure of memory committed
     -> Unit: By
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(Metaspace)
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 35700736
NumberDataPoints #1
Data point attributes:
     -> pool: Str(Compressed Class Space)
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 4325376
NumberDataPoints #2
Data point attributes:
     -> pool: Str(CodeHeap 'profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 8847360
NumberDataPoints #3
Data point attributes:
     -> pool: Str(G1 Old Gen)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 398458880
NumberDataPoints #4
Data point attributes:
     -> pool: Str(CodeHeap 'non-profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 2555904
NumberDataPoints #5
Data point attributes:
     -> pool: Str(G1 Eden Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 117440512
NumberDataPoints #6
Data point attributes:
     -> pool: Str(CodeHeap 'non-nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 2555904
NumberDataPoints #7
Data point attributes:
     -> pool: Str(G1 Survivor Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 10485760
Metric #11
Descriptor:
     -> Name: process.runtime.jvm.system.cpu.utilization
     -> Description: Recent cpu utilization for the whole system
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 0.000000
Metric #12
Descriptor:
     -> Name: process.runtime.jvm.system.cpu.load_1m
     -> Description: Average CPU load of the whole system for the last minute
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 0.890000
Metric #13
Descriptor:
     -> Name: process.runtime.jvm.cpu.utilization
     -> Description: Recent cpu utilization for the process
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 0.000000
Metric #14
Descriptor:
     -> Name: process.runtime.jvm.buffer.limit
     -> Description: Total capacity of the buffers in this pool
     -> Unit: By
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(mapped)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 0
NumberDataPoints #1
Data point attributes:
     -> pool: Str(direct)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 8192
ScopeMetrics #2
ScopeMetrics SchemaURL: 
InstrumentationScope io.opentelemetry.sdk.logs 
Metric #0
Descriptor:
     -> Name: queueSize
     -> Description: The number of logs queued
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> logRecordProcessorType: Str(BatchLogRecordProcessor)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:55.45147 +0000 UTC
Value: 0
	{"kind": "exporter", "data_type": "metrics", "name": "logging"}
2023-02-01T11:19:56.003Z	error	[email protected]/log.go:34	error gathering metrics: collected metric default_queueSize label:<name:"job" value:"poc/monitoring-poc-one" > label:<name:"logRecordProcessorType" value:"BatchLogRecordProcessor" > gauge:<value:0 >  has help "The number of logs queued" but should have "The number of spans queued"
	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*promLogger).Println
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/log.go:34
github.com/prometheus/client_golang/prometheus/promhttp.HandlerForTransactional.func1
	github.com/prometheus/[email protected]/prometheus/promhttp/http.go:139
net/http.HandlerFunc.ServeHTTP
	net/http/server.go:2109
net/http.(*ServeMux).ServeHTTP
	net/http/server.go:2487
go.opentelemetry.io/collector/config/confighttp.(*decompressor).wrap.func1
	go.opentelemetry.io/[email protected]/config/confighttp/compression.go:162
net/http.HandlerFunc.ServeHTTP
	net/http/server.go:2109
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*Handler).ServeHTTP
	go.opentelemetry.io/contrib/instrumentation/net/http/[email protected]/handler.go:210
go.opentelemetry.io/collector/config/confighttp.(*clientInfoHandler).ServeHTTP
	go.opentelemetry.io/[email protected]/config/confighttp/clientinfohandler.go:39
net/http.serverHandler.ServeHTTP
	net/http/server.go:2947
net/http.(*conn).serve
	net/http/server.go:1991
2023-02-01T11:19:56.297Z	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging", "#metrics": 18}
2023-02-01T11:19:56.298Z	info	ResourceMetrics #0
Resource SchemaURL: https://opentelemetry.io/schemas/1.16.0
Resource attributes:
     -> container.id: Str(2164232)
     -> deployment.environment: Str(pre)
     -> host.arch: Str(amd64)
     -> host.name: Str(cmunger-HP-Compaq-Elite-8300-CMT)
     -> os.description: Str(Linux 5.4.0-137-generic)
     -> os.type: Str(linux)
     -> process.command_line: Str(/usr/lib/jvm/zulu11-ca-amd64/bin/java -javaagent:./opentelemetry-javaagent.jar -Dotel.javaagent.configuration-file=./opentelemetry-one.properties -javaagent:/home/cmunger/dev/idea-IC-221.5080.210/lib/idea_rt.jar=39569:/home/cmunger/dev/idea-IC-221.5080.210/bin -Dfile.encoding=UTF-8)
     -> process.executable.path: Str(/usr/lib/jvm/zulu11-ca-amd64/bin/java)
     -> process.pid: Int(2186394)
     -> process.runtime.description: Str(Azul Systems, Inc. OpenJDK 64-Bit Server VM 11.0.17+8-LTS)
     -> process.runtime.name: Str(OpenJDK Runtime Environment)
     -> process.runtime.version: Str(11.0.17+8-LTS)
     -> service.name: Str(monitoring-poc-one)
     -> service.namespace: Str(poc)
     -> service.version: Str(1.1)
     -> telemetry.auto.version: Str(1.21.0)
     -> telemetry.sdk.language: Str(java)
     -> telemetry.sdk.name: Str(opentelemetry)
     -> telemetry.sdk.version: Str(1.21.0)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope io.opentelemetry.sdk.trace 
Metric #0
Descriptor:
     -> Name: queueSize
     -> Description: The number of spans queued
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> spanProcessorType: Str(BatchSpanProcessor)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0
ScopeMetrics #1
ScopeMetrics SchemaURL: 
InstrumentationScope io.opentelemetry.runtime-metrics 1.21.0-alpha
Metric #0
Descriptor:
     -> Name: process.runtime.jvm.buffer.count
     -> Description: The number of buffers in the pool
     -> Unit: {buffers}
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(mapped)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0
NumberDataPoints #1
Data point attributes:
     -> pool: Str(direct)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0
Metric #1
Descriptor:
     -> Name: process.runtime.jvm.classes.unloaded
     -> Description: Number of classes unloaded since JVM start
     -> Unit: 1
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0
Metric #2
Descriptor:
     -> Name: process.runtime.jvm.classes.current_loaded
     -> Description: Number of classes currently loaded
     -> Unit: 1
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 6654
Metric #3
Descriptor:
     -> Name: process.runtime.jvm.buffer.usage
     -> Description: Memory that the Java virtual machine is using for this buffer pool
     -> Unit: By
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(mapped)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0
NumberDataPoints #1
Data point attributes:
     -> pool: Str(direct)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0
Metric #4
Descriptor:
     -> Name: process.runtime.jvm.threads.count
     -> Description: Number of executing threads
     -> Unit: 1
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> daemon: Bool(false)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 2
NumberDataPoints #1
Data point attributes:
     -> daemon: Bool(true)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 15
Metric #5
Descriptor:
     -> Name: process.runtime.jvm.memory.usage_after_last_gc
     -> Description: Measure of memory used after the most recent garbage collection event on this pool
     -> Unit: By
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(G1 Old Gen)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0
NumberDataPoints #1
Data point attributes:
     -> pool: Str(G1 Eden Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0
NumberDataPoints #2
Data point attributes:
     -> pool: Str(G1 Survivor Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 10485760
Metric #6
Descriptor:
     -> Name: process.runtime.jvm.memory.limit
     -> Description: Measure of max obtainable memory
     -> Unit: By
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(Compressed Class Space)
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 1073741824
NumberDataPoints #1
Data point attributes:
     -> pool: Str(CodeHeap 'profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 122908672
NumberDataPoints #2
Data point attributes:
     -> pool: Str(G1 Old Gen)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 8405385216
NumberDataPoints #3
Data point attributes:
     -> pool: Str(CodeHeap 'non-profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 122912768
NumberDataPoints #4
Data point attributes:
     -> pool: Str(CodeHeap 'non-nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 5836800
Metric #7
Descriptor:
     -> Name: process.runtime.jvm.memory.usage
     -> Description: Measure of memory used
     -> Unit: By
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(Metaspace)
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 41074464
NumberDataPoints #1
Data point attributes:
     -> pool: Str(Compressed Class Space)
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 4784424
NumberDataPoints #2
Data point attributes:
     -> pool: Str(CodeHeap 'profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 10261248
NumberDataPoints #3
Data point attributes:
     -> pool: Str(G1 Old Gen)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 4569216
NumberDataPoints #4
Data point attributes:
     -> pool: Str(CodeHeap 'non-profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 2776448
NumberDataPoints #5
Data point attributes:
     -> pool: Str(G1 Eden Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 56623104
NumberDataPoints #6
Data point attributes:
     -> pool: Str(CodeHeap 'non-nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 1320704
NumberDataPoints #7
Data point attributes:
     -> pool: Str(G1 Survivor Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 10485760
Metric #8
Descriptor:
     -> Name: process.runtime.jvm.memory.init
     -> Description: Measure of initial memory requested
     -> Unit: By
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(Metaspace)
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0
NumberDataPoints #1
Data point attributes:
     -> pool: Str(Compressed Class Space)
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0
NumberDataPoints #2
Data point attributes:
     -> pool: Str(CodeHeap 'profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 2555904
NumberDataPoints #3
Data point attributes:
     -> pool: Str(G1 Old Gen)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 499122176
NumberDataPoints #4
Data point attributes:
     -> pool: Str(CodeHeap 'non-profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 2555904
NumberDataPoints #5
Data point attributes:
     -> pool: Str(G1 Eden Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 27262976
NumberDataPoints #6
Data point attributes:
     -> pool: Str(CodeHeap 'non-nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 2555904
NumberDataPoints #7
Data point attributes:
     -> pool: Str(G1 Survivor Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0
Metric #9
Descriptor:
     -> Name: process.runtime.jvm.classes.loaded
     -> Description: Number of classes loaded since JVM start
     -> Unit: 1
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 6654
Metric #10
Descriptor:
     -> Name: process.runtime.jvm.memory.committed
     -> Description: Measure of memory committed
     -> Unit: By
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(Metaspace)
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 42287104
NumberDataPoints #1
Data point attributes:
     -> pool: Str(Compressed Class Space)
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 5242880
NumberDataPoints #2
Data point attributes:
     -> pool: Str(CodeHeap 'profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 10289152
NumberDataPoints #3
Data point attributes:
     -> pool: Str(G1 Old Gen)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 232783872
NumberDataPoints #4
Data point attributes:
     -> pool: Str(CodeHeap 'non-profiled nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 2818048
NumberDataPoints #5
Data point attributes:
     -> pool: Str(G1 Eden Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 390070272
NumberDataPoints #6
Data point attributes:
     -> pool: Str(CodeHeap 'non-nmethods')
     -> type: Str(non_heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 2555904
NumberDataPoints #7
Data point attributes:
     -> pool: Str(G1 Survivor Space)
     -> type: Str(heap)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 10485760
Metric #11
Descriptor:
     -> Name: process.runtime.jvm.gc.duration
     -> Description: Duration of JVM garbage collection actions
     -> Unit: ms
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
Data point attributes:
     -> action: Str(end of minor GC)
     -> gc: Str(G1 Young Generation)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Count: 1
Sum: 15.000000
Min: 15.000000
Max: 15.000000
ExplicitBounds #0: 0.000000
ExplicitBounds #1: 5.000000
ExplicitBounds #2: 10.000000
ExplicitBounds #3: 25.000000
ExplicitBounds #4: 50.000000
ExplicitBounds #5: 75.000000
ExplicitBounds #6: 100.000000
ExplicitBounds #7: 250.000000
ExplicitBounds #8: 500.000000
ExplicitBounds #9: 750.000000
ExplicitBounds #10: 1000.000000
ExplicitBounds #11: 2500.000000
ExplicitBounds #12: 5000.000000
ExplicitBounds #13: 7500.000000
ExplicitBounds #14: 10000.000000
Buckets #0, Count: 0
Buckets #1, Count: 0
Buckets #2, Count: 0
Buckets #3, Count: 1
Buckets #4, Count: 0
Buckets #5, Count: 0
Buckets #6, Count: 0
Buckets #7, Count: 0
Buckets #8, Count: 0
Buckets #9, Count: 0
Buckets #10, Count: 0
Buckets #11, Count: 0
Buckets #12, Count: 0
Buckets #13, Count: 0
Buckets #14, Count: 0
Buckets #15, Count: 0
Metric #12
Descriptor:
     -> Name: process.runtime.jvm.system.cpu.utilization
     -> Description: Recent cpu utilization for the whole system
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0.613251
Metric #13
Descriptor:
     -> Name: process.runtime.jvm.system.cpu.load_1m
     -> Description: Average CPU load of the whole system for the last minute
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0.890000
Metric #14
Descriptor:
     -> Name: process.runtime.jvm.cpu.utilization
     -> Description: Recent cpu utilization for the process
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0.533951
Metric #15
Descriptor:
     -> Name: process.runtime.jvm.buffer.limit
     -> Description: Total capacity of the buffers in this pool
     -> Unit: By
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> pool: Str(mapped)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0
NumberDataPoints #1
Data point attributes:
     -> pool: Str(direct)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 0
ScopeMetrics #2
ScopeMetrics SchemaURL: 
InstrumentationScope io.opentelemetry.sdk.logs 
Metric #0
Descriptor:
     -> Name: queueSize
     -> Description: The number of logs queued
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> logRecordProcessorType: Str(BatchLogRecordProcessor)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:56.277929 +0000 UTC
Value: 1
	{"kind": "exporter", "data_type": "metrics", "name": "logging"}
2023-02-01T11:19:58.003Z	error	[email protected]/log.go:34	error gathering metrics: collected metric default_queueSize label:<name:"job" value:"poc/monitoring-poc-one" > label:<name:"spanProcessorType" value:"BatchSpanProcessor" > gauge:<value:0 >  has help "The number of spans queued" but should have "The number of logs queued"
	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*promLogger).Println
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/log.go:34
github.com/prometheus/client_golang/prometheus/promhttp.HandlerForTransactional.func1
	github.com/prometheus/[email protected]/prometheus/promhttp/http.go:139
net/http.HandlerFunc.ServeHTTP
	net/http/server.go:2109
net/http.(*ServeMux).ServeHTTP
	net/http/server.go:2487
go.opentelemetry.io/collector/config/confighttp.(*decompressor).wrap.func1
	go.opentelemetry.io/[email protected]/config/confighttp/compression.go:162
net/http.HandlerFunc.ServeHTTP
	net/http/server.go:2109
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*Handler).ServeHTTP
	go.opentelemetry.io/contrib/instrumentation/net/http/[email protected]/handler.go:210
go.opentelemetry.io/collector/config/confighttp.(*clientInfoHandler).ServeHTTP
	go.opentelemetry.io/[email protected]/config/confighttp/clientinfohandler.go:39
net/http.serverHandler.ServeHTTP
	net/http/server.go:2947
net/http.(*conn).serve
	net/http/server.go:1991
^C2023-02-01T11:20:08.494Z	info	otelcol/collector.go:234	Received signal from OS	{"signal": "interrupt"}
2023-02-01T11:20:08.494Z	info	service/service.go:154	Starting shutdown...
2023-02-01T11:20:08.494Z	info	healthcheck/handler.go:129	Health Check state change	{"kind": "extension", "name": "health_check", "status": "unavailable"}
2023-02-01T11:20:08.494Z	info	service/pipelines.go:130	Stopping receivers...
2023-02-01T11:20:08.494Z	info	service/pipelines.go:137	Stopping processors...
2023-02-01T11:20:08.494Z	info	service/pipelines.go:144	Stopping exporters...
2023-02-01T11:20:08.494Z	info	extensions/extensions.go:55	Stopping extensions...
2023-02-01T11:20:08.494Z	info	[email protected]/zpagesextension.go:109	Unregistered zPages span processor on tracer provider	{"kind": "extension", "name": "zpages"}
2023-02-01T11:20:08.494Z	info	service/service.go:168	Shutdown complete.

Additional context

This issue is similar to this one : open-telemetry/opentelemetry-java#4382
@cmunger cmunger added bug Something isn't working needs triage New item requiring triage labels Jan 31, 2023
@github-actions
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@Aneurysm9
Copy link
Member

Can you please add the logging exporter to your metrics pipeline at the detailed verbosity? Understanding the metrics being presented to the prometheus exporter here will be important.

@cmunger
Copy link
Author

cmunger commented Feb 1, 2023

updated with the full log output where you can see the issue with the 2 queueSize gauges, sorry the log is pretty big..

@Aneurysm9
Copy link
Member

2023-02-01T11:19:58.003Z error [email protected]/log.go:34 error gathering metrics: collected metric default_queueSize label:<name:"job" value:"poc/monitoring-poc-one" > label:<name:"spanProcessorType" value:"BatchSpanProcessor" > gauge:<value:0 > has help "The number of spans queued" but should have "The number of logs queued"

Looks like metrics with the same name from different instrumentation scopes with different descriptions not being handled well. The metrics given to the exporter:

ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope io.opentelemetry.sdk.trace 
Metric #0
Descriptor:
     -> Name: queueSize
     -> Description: The number of spans queued
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> spanProcessorType: Str(BatchSpanProcessor)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:54.453442 +0000 UTC
Value: 0
ScopeMetrics #1
ScopeMetrics SchemaURL: 
InstrumentationScope io.opentelemetry.sdk.logs 
Metric #0
Descriptor:
     -> Name: queueSize
     -> Description: The number of logs queued
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> logRecordProcessorType: Str(BatchLogRecordProcessor)
StartTimestamp: 2023-02-01 11:19:53.447367 +0000 UTC
Timestamp: 2023-02-01 11:19:54.453442 +0000 UTC
Value: 0

@cmunger
Copy link
Author

cmunger commented Feb 2, 2023 •
edited
Loading

Yep exactly the problem, this issue have been also encountered in the java instrumentation project.
This is kind of a show stopper as it will happen in any basic java project with logging instrumentation and tracing making the export of metrics using prometheus and the collector complety unusable.

@atoulme atoulme removed the needs triage New item requiring triage label Mar 8, 2023
@mviitane
Copy link
Member

I'm having the same problem, and as a workaround, decided to drop the queueSize metrics (for both spans and logs) using the filter processor. I'm sure there are more elegant alternatives but this gets rid of the problem.

  filter:
    metrics:
      exclude:
        match_type: strict
        metric_names:
          - queueSize

mviitane added a commit to mviitane/opentelemetry-demo that referenced this issue Mar 17, 2023
@mviitane
What’s included?
504a6f8 
- Enable few logs for Ad service and Recommendation service.
- Add OTLP exporters for logs
- Add the filter processor to prevent an error from the Prometheus exporter for duplicate queueSize metric, see
open-telemetry/opentelemetry-collector-contrib#18194. The filter processor can be removed when the fault gets fixed.

This PR doesn’t introduce any logs backend. Instead, logs are output only to Logging exporter and can be seen in the console (otelcol).
otel-col  | 2023-03-17T11:40:22.662Z	info	LogsExporter	{"kind": "exporter", "data_type": "logs", "name": "logging", "#logs": 2}

After this PR, different logging backends can be easily tested by configuring an additional exporter.
mviitane added a commit to mviitane/opentelemetry-demo that referenced this issue Mar 17, 2023
@mviitane
What’s included?
06dbf01 
- Enable logs for Ad service and Recommendation service.
- Add OTLP exporters for logs
- Add the filter processor to prevent an error from the Prometheus exporter for duplicate queueSize metric, see
open-telemetry/opentelemetry-collector-contrib#18194. The filter processor can be removed when the fault gets fixed.

This PR doesn’t introduce any logs backend. Instead, logs are output only to Logging exporter and can be seen in the console (otelcol).
otel-col  | 2023-03-17T11:40:22.662Z	info	LogsExporter	{"kind": "exporter", "data_type": "logs", "name": "logging", "#logs": 2}

After this PR, different logging backends can be easily tested by configuring an additional exporter.
@mviitane mviitane mentioned this issue Mar 17, 2023
Merged
1 task
@cmunger
Copy link
Author

cmunger commented Mar 19, 2023

thanks for the filter tip, working nicely ! hope a correct solution will be provided by the team.

austinlparker added a commit to open-telemetry/opentelemetry-demo that referenced this issue Mar 20, 2023
@mviitane @austinlparker @julianocosta89
Add logs for Ad service and Recommendation service (#796)
ed8ead1 
* What’s included?

- Enable logs for Ad service and Recommendation service.
- Add OTLP exporters for logs
- Add the filter processor to prevent an error from the Prometheus exporter for duplicate queueSize metric, see
open-telemetry/opentelemetry-collector-contrib#18194. The filter processor can be removed when the fault gets fixed.

This PR doesn’t introduce any logs backend. Instead, logs are output only to Logging exporter and can be seen in the console (otelcol).
otel-col  | 2023-03-17T11:40:22.662Z	info	LogsExporter	{"kind": "exporter", "data_type": "logs", "name": "logging", "#logs": 2}

After this PR, different logging backends can be easily tested by configuring an additional exporter.

* Add changelog and fix lint errors.

* Fix changelog and lint

* Fix lint

* Move protocol env variables to .env

* Update CHANGELOG.md

---------

Co-authored-by: Austin Parker <[email protected]>
Co-authored-by: Juliano Costa <[email protected]>
@puckpuck puckpuck mentioned this issue Apr 6, 2023
Closed
juliangiuca pushed a commit to juliangiuca/opentelemetry-demo that referenced this issue Apr 12, 2023
@mviitane @austinlparker
@julianocosta89 @juliangiuca
Add logs for Ad service and Recommendation service (open-telemetry#796)
05e5e29 
* What’s included?

- Enable logs for Ad service and Recommendation service.
- Add OTLP exporters for logs
- Add the filter processor to prevent an error from the Prometheus exporter for duplicate queueSize metric, see
open-telemetry/opentelemetry-collector-contrib#18194. The filter processor can be removed when the fault gets fixed.

This PR doesn’t introduce any logs backend. Instead, logs are output only to Logging exporter and can be seen in the console (otelcol).
otel-col  | 2023-03-17T11:40:22.662Z	info	LogsExporter	{"kind": "exporter", "data_type": "logs", "name": "logging", "#logs": 2}

After this PR, different logging backends can be easily tested by configuring an additional exporter.

* Add changelog and fix lint errors.

* Fix changelog and lint

* Fix lint

* Move protocol env variables to .env

* Update CHANGELOG.md

---------

Co-authored-by: Austin Parker <[email protected]>
Co-authored-by: Juliano Costa <[email protected]>
@github-actions
Copy link
Contributor

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot added the Stale label May 19, 2023
@github-actions
Copy link
Contributor

This issue has been closed as inactive because it has been stale for 120 days with no activity.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Jul 18, 2023
jmichalak9 pushed a commit to jmichalak9/opentelemetry-demo that referenced this issue Mar 22, 2024
@mviitane @austinlparker @julianocosta89
Add logs for Ad service and Recommendation service (open-telemetry#796)
a49b0d1 
* What’s included?

- Enable logs for Ad service and Recommendation service.
- Add OTLP exporters for logs
- Add the filter processor to prevent an error from the Prometheus exporter for duplicate queueSize metric, see
open-telemetry/opentelemetry-collector-contrib#18194. The filter processor can be removed when the fault gets fixed.

This PR doesn’t introduce any logs backend. Instead, logs are output only to Logging exporter and can be seen in the console (otelcol).
otel-col  | 2023-03-17T11:40:22.662Z	info	LogsExporter	{"kind": "exporter", "data_type": "logs", "name": "logging", "#logs": 2}

After this PR, different logging backends can be easily tested by configuring an additional exporter.

* Add changelog and fix lint errors.

* Fix changelog and lint

* Fix lint

* Move protocol env variables to .env

* Update CHANGELOG.md

---------

Co-authored-by: Austin Parker <[email protected]>
Co-authored-by: Juliano Costa <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working closed as inactive exporter/prometheus Stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@atoulme @Aneurysm9 @cmunger @mviitane