Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

exporter/prometheusremotewrite: failing Up metric export for Prometheus compliance tests #2961

Closed
5 of 15 tasks
odeke-em opened this issue Apr 19, 2021 · 1 comment · Fixed by #3423
Closed
5 of 15 tasks

exporter/prometheusremotewrite: failing Up metric export for Prometheus compliance tests #2961

odeke-em opened this issue Apr 19, 2021 · 1 comment · Fixed by #3423
Labels
area:exporter bug Something isn't working

Comments

@odeke-em
Copy link
Member

odeke-em commented Apr 19, 2021 •
edited
Loading

Describe the bug
Coming here from Prometheus compliance tests, TestRemoteWrite/otelcollector/Up fails

Steps to reproduce
Run the compliance tests at https://github.com/prometheus/compliance/tree/main/remote_write by

go test --tags=compliance -v ./

What did you expect to see?
PASS

What did you see instead?
10 out of 14 test failures

        --- PASS: TestRemoteWrite/otelcollector/Headers (48.87s)
        --- FAIL: TestRemoteWrite/otelcollector/Up (48.95s)
        --- FAIL: TestRemoteWrite/otelcollector/Invalid (48.95s)
        --- FAIL: TestRemoteWrite/otelcollector/Staleness (48.97s)
        --- FAIL: TestRemoteWrite/otelcollector/JobLabel (48.98s)
        --- FAIL: TestRemoteWrite/otelcollector/Counter (48.98s)
        --- PASS: TestRemoteWrite/otelcollector/Timestamp (48.99s)
        --- FAIL: TestRemoteWrite/otelcollector/Summary (48.99s)
        --- FAIL: TestRemoteWrite/otelcollector/RepeatedLabels (10.01s)
        --- FAIL: TestRemoteWrite/otelcollector/NameLabel (10.01s)
        --- PASS: TestRemoteWrite/otelcollector/EmptyLabels (10.01s)
        --- FAIL: TestRemoteWrite/otelcollector/Histogram (10.01s)
        --- PASS: TestRemoteWrite/otelcollector/Gauge (10.02s)
        --- PASS: TestRemoteWrite/otelcollector/SortedLabels (10.02s)
        --- FAIL: TestRemoteWrite/otelcollector/InstanceLabel (10.02s)

Here is the listing of failures

  • TestRemoteWrite/otelcollector/Up (48.95s)
  • TestRemoteWrite/otelcollector/Headers (48.87s)
  • TestRemoteWrite/otelcollector/Invalid (48.95s)
  • TestRemoteWrite/otelcollector/Staleness (48.97s)
  • TestRemoteWrite/otelcollector/JobLabel (48.98s)
  • TestRemoteWrite/otelcollector/Counter (48.98s)
  • TestRemoteWrite/otelcollector/Timestamp (48.99s)
  • TestRemoteWrite/otelcollector/Summary (48.99s)
  • TestRemoteWrite/otelcollector/RepeatedLabels (10.01s)
  • TestRemoteWrite/otelcollector/NameLabel (10.01s)
  • TestRemoteWrite/otelcollector/EmptyLabels (10.01s)
  • TestRemoteWrite/otelcollector/Histogram (10.01s)
  • TestRemoteWrite/otelcollector/Gauge (10.02s)
  • TestRemoteWrite/otelcollector/SortedLabels (10.02s)
  • TestRemoteWrite/otelcollector/InstanceLabel (10.02s)

and the full failure print out is at

=== CONT  TestRemoteWrite/otelcollector/Up
    up.go:23: 
        	Error Trace:	up.go:23
        	            				main_test.go:101
        	            				main_test.go:65
        	Error:      	Should be true
        	Test:       	TestRemoteWrite/otelcollector/Up
        	Messages:   	found zero samples for up{job="test"}
=== CONT  TestRemoteWrite/otelcollector/NameLabel
=== CONT  TestRemoteWrite/otelcollector/Invalid
    up.go:39: 
        	Error Trace:	up.go:39
        	            				main_test.go:101
        	            				main_test.go:65
        	Error:      	Should be true
        	Test:       	TestRemoteWrite/otelcollector/Invalid
        	Messages:   	found zero samples for up{job="test"}
=== CONT  TestRemoteWrite/otelcollector/EmptyLabels
=== CONT  TestRemoteWrite/otelcollector/Staleness
    staleness.go:43: 
        	Error Trace:	staleness.go:43
        	            				main_test.go:101
        	            				main_test.go:65
        	Error:      	Should be true
        	Test:       	TestRemoteWrite/otelcollector/Staleness
        	Messages:   	found no staleness markers for stale{job="test"}
=== CONT  TestRemoteWrite/otelcollector/Histogram
=== CONT  TestRemoteWrite/otelcollector/JobLabel
    job_label.go:23: 
        	Error Trace:	job_label.go:23
        	            				main_test.go:101
        	            				main_test.go:65
        	Error:      	Should be true
        	Test:       	TestRemoteWrite/otelcollector/JobLabel
        	Messages:   	found zero samples for gauge{job="test"}
=== CONT  TestRemoteWrite/otelcollector/Counter
    counter.go:35: 
        	Error Trace:	counter.go:35
        	            				main_test.go:101
        	            				main_test.go:65
        	Error:      	Should be true
        	Test:       	TestRemoteWrite/otelcollector/Counter
        	Messages:   	found zero samples for {__name__="counter"}
=== CONT  TestRemoteWrite/otelcollector/SortedLabels
=== CONT  TestRemoteWrite/otelcollector/Gauge
=== CONT  TestRemoteWrite/otelcollector/InstanceLabel
=== CONT  TestRemoteWrite/otelcollector/Summary
    helpers.go:30: 
        	Error Trace:	helpers.go:30
        	            				helpers.go:41
        	            				helpers.go:52
        	            				helpers.go:40
        	            				helpers.go:29
        	            				summary.go:30
        	            				main_test.go:101
        	            				main_test.go:65
        	Error:      	Not equal: 
        	            	expected: 6
        	            	actual  : 0
        	Test:       	TestRemoteWrite/otelcollector/Summary
=== CONT  TestRemoteWrite/otelcollector/RepeatedLabels
    labels.go:58: 
        	Error Trace:	labels.go:58
        	            				main_test.go:101
        	            				main_test.go:65
        	Error:      	Should be true
        	Test:       	TestRemoteWrite/otelcollector/RepeatedLabels
        	Messages:   	found zero samples for up{job="test"} = 0
=== CONT  TestRemoteWrite/otelcollector/NameLabel
    labels.go:108: 
        	Error Trace:	labels.go:108
        	            				main_test.go:101
        	            				main_test.go:65
        	Error:      	Should be true
        	Test:       	TestRemoteWrite/otelcollector/NameLabel
        	Messages:   	found zero samples for up{job="test"} = 0
=== CONT  TestRemoteWrite/otelcollector/Histogram
    helpers.go:30: 
        	Error Trace:	helpers.go:30
        	            				helpers.go:41
        	            				helpers.go:52
        	            				helpers.go:40
        	            				helpers.go:29
        	            				histogram.go:25
        	            				main_test.go:101
        	            				main_test.go:65
        	Error:      	Not equal: 
        	            	expected: 1
        	            	actual  : 0
        	Test:       	TestRemoteWrite/otelcollector/Histogram
=== CONT  TestRemoteWrite/otelcollector/InstanceLabel
    helpers.go:21: 
        	Error Trace:	helpers.go:21
        	            				helpers.go:52
        	            				helpers.go:13
        	            				instance_label.go:26
        	            				main_test.go:101
        	            				main_test.go:65
        	Error:      	Should be true
        	Test:       	TestRemoteWrite/otelcollector/InstanceLabel
        	Messages:   	label 'instance' not found

What version did you use?
The latest release v0.24.0

What config did you use?
N/A

Environment
N/A

Additional context
Add any other context about the problem here.
@odeke-em odeke-em added the bug Something isn't working label Apr 19, 2021
@rakyll rakyll mentioned this issue Apr 20, 2021
Closed
@odeke-em
Copy link
Member Author

The actual issue is with the Prometheus receiver not emitting staleness markers.

@odeke-em odeke-em mentioned this issue Jun 10, 2021
Closed
odeke-em added a commit to orijtech/opentelemetry-collector that referenced this issue Jun 10, 2021
@odeke-em
receiver/prometheus: add store to track stale metrics
72934eb 
Implements a simple store that will be used to record stale metrics;
those that are NOT present in the current scrape, but were in the prior
scrape. This logic will then be attached to our receiver logic
to then complete the mechanism and then issue stale markers.

Updates open-telemetry#3413
Updates open-telemetry#2961
@odeke-em odeke-em mentioned this issue Jun 10, 2021
Merged
odeke-em added a commit to orijtech/opentelemetry-collector that referenced this issue Jun 10, 2021
@odeke-em
receiver/prometheus: add store to track stale metrics
04ac4d6 
Implements a simple store that will be used to record stale metrics;
those that are NOT present in the current scrape, but were in the prior
scrape. This logic will then be attached to our receiver logic
to then complete the mechanism and then issue stale markers.

Updates open-telemetry#3413
Updates open-telemetry#2961
odeke-em added a commit to orijtech/opentelemetry-collector that referenced this issue Jun 10, 2021
@odeke-em
receiver/prometheus: add store to track stale metrics
ef1d34d 
Implements a simple store that will be used to record stale metrics;
those that are NOT present in the current scrape, but were in the prior
scrape. This logic will then be attached to our receiver logic
to then complete the mechanism and then issue stale markers.

Updates open-telemetry#3413
Updates open-telemetry#2961
@odeke-em odeke-em mentioned this issue Jun 11, 2021
Merged
odeke-em added a commit to orijtech/opentelemetry-collector that referenced this issue Jun 11, 2021
@odeke-em
receiver/prometheus: add store to track stale metrics
7c6b172 
Implements a simple store that will be used to record stale metrics;
those that are NOT present in the current scrape, but were in the prior
scrape. This logic will then be attached to our receiver logic
to then complete the mechanism and then issue stale markers.

Updates open-telemetry#3413
Updates open-telemetry#2961
odeke-em added a commit to orijtech/opentelemetry-collector that referenced this issue Jun 14, 2021
@odeke-em
receiver/prometheus: add store to track stale metrics
658d42d 
Implements a simple store that will be used to record stale metrics;
those that are NOT present in the current scrape, but were in the prior
scrape. This logic will then be attached to our receiver logic
to then complete the mechanism and then issue stale markers.

Updates open-telemetry#3413
Updates open-telemetry#2961
odeke-em added a commit to orijtech/opentelemetry-collector that referenced this issue Jun 15, 2021
@odeke-em
receiver/prometheus: add store to track stale metrics
eb85405 
Implements a simple store that will be used to record stale metrics;
those that are NOT present in the current scrape, but were in the prior
scrape. This logic will then be attached to our receiver logic
to then complete the mechanism and then issue stale markers.

Updates open-telemetry#3413
Updates open-telemetry#2961
tigrannajaryan pushed a commit that referenced this issue Jun 16, 2021
@odeke-em
receiver/prometheus: add store to track stale metrics (#3414)
cdc1634 
Implements a simple store that will be used to record stale metrics;
those that are NOT present in the current scrape, but were in the prior
scrape. This logic will then be attached to our receiver logic
to then complete the mechanism and then issue stale markers.

Updates #3413
Updates #2961
odeke-em added a commit to orijtech/opentelemetry-collector that referenced this issue Jun 16, 2021
@odeke-em
receiver/prometheus: glue and complete staleness marking for disappea…
1a1339f 
…red metrics

Adds the staleness store to the metricsBuilder and transaction,
to track when metrics disappear between scrapes.

Fixes open-telemetry#2961
Fixes open-telemetry#3413
odeke-em added a commit to orijtech/opentelemetry-collector that referenced this issue Jun 17, 2021
@odeke-em
receiver/prometheus: glue and complete staleness marking for disappea…
6ab3223 
…red metrics

Adds the staleness store to the metricsBuilder and transaction,
to track when metrics disappear between scrapes.

Fixes open-telemetry#2961
Fixes open-telemetry#3413
odeke-em added a commit to orijtech/opentelemetry-collector that referenced this issue Jun 17, 2021
@odeke-em
receiver/prometheus: glue and complete staleness marking for disappea…
4e062a8 
…red metrics

Adds the staleness store to the metricsBuilder and transaction,
to track when metrics disappear between scrapes.

Fixes open-telemetry#2961
Fixes open-telemetry#3413
odeke-em added a commit to orijtech/opentelemetry-collector that referenced this issue Jun 22, 2021
@odeke-em
receiver/prometheus: glue and complete staleness marking for disappea…
cc40e65 
…red metrics

Adds the staleness store to the metricsBuilder and transaction,
to track when metrics disappear between scrapes.

Fixes open-telemetry#2961
Fixes open-telemetry#3413
odeke-em added a commit to orijtech/opentelemetry-collector that referenced this issue Jul 6, 2021
@odeke-em
receiver/prometheus: glue and complete staleness marking for disappea…
8fa960f 
…red metrics

Adds the staleness store to the metricsBuilder and transaction,
to track when metrics disappear between scrapes.

Fixes open-telemetry#2961
Fixes open-telemetry#3413
odeke-em added a commit to orijtech/opentelemetry-collector that referenced this issue Jul 8, 2021
@odeke-em
receiver/prometheus: glue and complete staleness marking for disappea…
5a16fd7 
…red metrics

Adds the staleness store to the metricsBuilder and transaction,
to track when metrics disappear between scrapes.

Fixes open-telemetry#2961
Fixes open-telemetry#3413
odeke-em added a commit to orijtech/opentelemetry-collector that referenced this issue Jul 8, 2021
@odeke-em
receiver/prometheus: glue and complete staleness marking for disappea…
572bb06 
…red metrics

Adds the staleness store to the metricsBuilder and transaction,
to track when metrics disappear between scrapes.

Fixes open-telemetry#2961
Fixes open-telemetry#3413
tigrannajaryan pushed a commit that referenced this issue Jul 8, 2021
@odeke-em
receiver/prometheus: glue and complete staleness marking for disappea…
8b79380 
…ring metrics (#3423)

Adds the staleness store to the metricsBuilder and transaction to track when metrics
disappear between scrapes.

Depends on #3414
Fixes #3413
Fixes #2961
@modulitos modulitos mentioned this issue Sep 14, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area:exporter bug Something isn't working
Projects
None yet
Milestone
No milestone
2 participants
@alolita @odeke-em