feat: added translation rw2 translation for gauges #35703

jmichalek132 · 2024-10-08T19:29:30Z

Description

Draft starting the work on adding support for remote write 2.0 in the translation package.
Adding support for translating gauges.

This is first iteration and to keep the PR small

we don't handle duplicate metrics
only support gauges
don't handle other labels than metric name
don't handle exemplars
don't handle metadata

Link to tracking issue #33661

Fixes

Testing

Documentation

Signed-off-by: Juraj Michalek <[email protected]>

pkg/translator/prometheusremotewrite/metrics_to_prw_v2.go

pkg/translator/prometheusremotewrite/metrics_to_prw_v2_test.go

pkg/translator/prometheusremotewrite/number_data_points_v2_test.go

jmichalek132 · 2024-10-08T20:20:11Z

Do we add changelog entry for this PR too?
Do we want this partial PRs to show up in changelog?

dashpole · 2024-10-08T20:22:52Z

Yes, please add a changelog for this PR, and future PRs if there are any notable changes.

jmichalek132 · 2024-10-09T08:06:15Z

Not sure why the unit tests (link) keep failing when the expected and the output matches https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/11243820134/job/31260663519?pr=35703#step:8:321.

dashpole · 2024-10-09T14:18:53Z

Yeah strange. Almost seems like it is something to do with the NaNs.

jmichalek132 · 2024-10-09T16:27:45Z

@dashpole could you please add @ArthurSens and @aknuds1 as reviewers too?

dashpole · 2024-10-09T17:00:09Z

I can't add @aknuds1, but i've added arthur

aknuds1 · 2024-10-09T17:07:11Z

I'm having a look.

dashpole · 2024-10-09T17:48:44Z

prometheusremotewrite/metrics_to_prw_v2_test.go:11: File is not `gci`-ed with --skip-generated -s standard -s default -s prefix(github.com/open-telemetry/opentelemetry-collector-contrib) (gci)
	writev2 "github.com/prometheus/prometheus/prompb/io/prometheus/write/v2"
prometheusremotewrite/metrics_to_prw_v2_test.go:15: File is not `gci`-ed with --skip-generated -s standard -s default -s prefix(github.com/open-telemetry/opentelemetry-collector-contrib) (gci)

	"github.com/stretchr/testify/require"
prometheusremotewrite/number_data_points_v2_test.go:10: File is not `gci`-ed with --skip-generated -s standard -s default -s prefix(github.com/open-telemetry/opentelemetry-collector-contrib) (gci)

looks like imports are not grouped corrrectly

jmichalek132 · 2024-10-09T17:50:36Z

prometheusremotewrite/metrics_to_prw_v2_test.go:11: File is not `gci`-ed with --skip-generated -s standard -s default -s prefix(github.com/open-telemetry/opentelemetry-collector-contrib) (gci)
	writev2 "github.com/prometheus/prometheus/prompb/io/prometheus/write/v2"
prometheusremotewrite/metrics_to_prw_v2_test.go:15: File is not `gci`-ed with --skip-generated -s standard -s default -s prefix(github.com/open-telemetry/opentelemetry-collector-contrib) (gci)

	"github.com/stretchr/testify/require"
prometheusremotewrite/number_data_points_v2_test.go:10: File is not `gci`-ed with --skip-generated -s standard -s default -s prefix(github.com/open-telemetry/opentelemetry-collector-contrib) (gci)

looks like imports are not grouped corrrectly

should be fixed.

ArthurSens · 2024-10-09T18:01:16Z

pkg/translator/prometheusremotewrite/metrics_to_prw_v2.go

+// FromMetricsV2 converts pmetric.Metrics to Prometheus remote write format 2.0.
+func FromMetricsV2(md pmetric.Metrics, settings Settings) (map[string]*writev2.TimeSeries, error) {
+	c := newPrometheusConverterV2()
+	errs := c.fromMetrics(md, settings)
+	tss := c.timeSeries()
+	out := make(map[string]*writev2.TimeSeries, len(tss))
+	for i := range tss {
+		out[strconv.Itoa(i)] = &tss[i]
+	}
+
+	return out, errs
+}
+
+// prometheusConverterV2 converts from OTLP to Prometheus write 2.0 format.
+type prometheusConverterV2 struct {
+	// TODO handle conflicts
+	unique      map[uint64]*writev2.TimeSeries
+	symbolTable writev2.SymbolsTable
+}
+
+func newPrometheusConverterV2() *prometheusConverterV2 {
+	return &prometheusConverterV2{
+		unique:      map[uint64]*writev2.TimeSeries{},
+		symbolTable: writev2.NewSymbolTable(),
+	}
+}


Since prometheusconverter is a private object, users of this package need a way of retrieving the symbols table somehow.

Maybe FromMetricsV2 should also return that? (It can return nil for now)

Is the symbol table meant to be per-request, or long-lived? But regardless it probably makes sense to return both, since both are needed in the request.

It's per request. PRW is meant to be stateless

then returning it makes sense to me.

Will update the PR, should we also make sure it's empty before each request?

FromMetricsV2() starts with newPrometheusConverterV2(), so I'm assuming it's impossible to not be empty, right?

oh yeah your are right

Addressed in new PR #35734

Fixes open-telemetry#35626

Signed-off-by: Juraj Michalek <[email protected]>

Co-authored-by: David Ashpole <[email protected]>

Signed-off-by: Juraj Michalek <[email protected]>

jmichalek132 · 2024-10-10T19:25:46Z

sorry my bad closing this PR and opening a new one

bertysentry · 2024-10-10T21:57:17Z

[like] Bertrand Martin reacted to your message:

________________________________ From: Juraj Michálek ***@***.***> Sent: Thursday, October 10, 2024 7:34:04 PM To: open-telemetry/opentelemetry-collector-contrib ***@***.***> Cc: Bertrand Martin ***@***.***>; Review requested ***@***.***> Subject: Re: [open-telemetry/opentelemetry-collector-contrib] feat: added translation rw2 translation for gauges (PR #35703) @jmichalek132 commented on this pull request.

________________________________ In pkg/translator/prometheusremotewrite/metrics_to_prw_v2.go<#35703 (comment)>:

+// FromMetricsV2 converts pmetric.Metrics to Prometheus remote write format 2.0.

+func FromMetricsV2(md pmetric.Metrics, settings Settings) (map[string]*writev2.TimeSeries, error) { + c := newPrometheusConverterV2() + errs := c.fromMetrics(md, settings) + tss := c.timeSeries() + out := make(map[string]*writev2.TimeSeries, len(tss)) + for i := range tss { + out[strconv.Itoa(i)] = &tss[i] + } + + return out, errs +} + +// prometheusConverterV2 converts from OTLP to Prometheus write 2.0 format. +type prometheusConverterV2 struct { + // TODO handle conflicts + unique map[uint64]*writev2.TimeSeries + symbolTable writev2.SymbolsTable +} + +func newPrometheusConverterV2() *prometheusConverterV2 { + return &prometheusConverterV2{ + unique: map[uint64]*writev2.TimeSeries{}, + symbolTable: writev2.NewSymbolTable(), + } +} Addressed in new PR #35734<#35734> — Reply to this email directly, view it on GitHub<#35703 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHYD3YWKY4OLYHUJQXVXAKDZ23JCZAVCNFSM6AAAAABPS64XRKVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDGNRRGI2DCNZYG4>. You are receiving this because your review was requested.Message ID: ***@***.***>

#### Description Follow up from #35703. Draft starting the work on adding support for remote write 2.0 in the translation package. Adding support for translating gauges. This is first iteration and to keep the PR small * we don't handle duplicate metrics * only support gauges * don't handle other labels than metric name * don't handle exemplars * don't handle metadata  #### Link to tracking issue #33661 Fixes  #### Testing  #### Documentation  --------- Signed-off-by: Juraj Michalek <[email protected]> Co-authored-by: Arthur Silva Sens <[email protected]> Co-authored-by: David Ashpole <[email protected]>

feat: added translation rw2 translation for gauges

c36ef2b

Signed-off-by: Juraj Michalek <[email protected]>

github-actions bot added the pkg/translator/prometheus label Oct 8, 2024

github-actions bot requested review from bertysentry and dashpole October 8, 2024 19:29

jmichalek132 mentioned this pull request Oct 8, 2024

feat: prometheus translation add support for rw2 #35583

Closed

dashpole reviewed Oct 8, 2024

View reviewed changes

pkg/translator/prometheusremotewrite/metrics_to_prw_v2.go Outdated Show resolved Hide resolved

pkg/translator/prometheusremotewrite/metrics_to_prw_v2_test.go Outdated Show resolved Hide resolved

pkg/translator/prometheusremotewrite/number_data_points_v2_test.go Outdated Show resolved Hide resolved

jmichalek132 requested a review from dashpole October 9, 2024 16:27

dashpole requested a review from ArthurSens October 9, 2024 16:59

jmichalek132 marked this pull request as ready for review October 9, 2024 17:33

jmichalek132 requested a review from a team as a code owner October 9, 2024 17:33

github-actions bot assigned djaglowski Oct 9, 2024

dashpole approved these changes Oct 9, 2024

View reviewed changes

ArthurSens reviewed Oct 9, 2024

View reviewed changes

atoulme and others added 8 commits October 10, 2024 21:24

[chore] update sapm-proto to 0.16.0 (open-telemetry#35630)

2a16ec6

Fixes open-telemetry#35626

chore: fix issues from linter

adf4c59

Signed-off-by: Juraj Michalek <[email protected]>

chore: add test case showing we don't handle duplicates

19843ff

Signed-off-by: Juraj Michalek <[email protected]>

chore: ran make fmt

ffd70d3

Signed-off-by: Juraj Michalek <[email protected]>

Update pkg/translator/prometheusremotewrite/metrics_to_prw_v2.go

41d1dc8

Co-authored-by: David Ashpole <[email protected]>

chore: added test case for nan value

3157b47

Signed-off-by: Juraj Michalek <[email protected]>

chore: ran make fmt

0d79cee

Signed-off-by: Juraj Michalek <[email protected]>

jmichalek132 closed this Oct 10, 2024

jmichalek132 mentioned this pull request Oct 10, 2024

feat: added prometheus rw2 translation for gauges #35734

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: added translation rw2 translation for gauges #35703

feat: added translation rw2 translation for gauges #35703

jmichalek132 commented Oct 8, 2024 •

edited

Loading

jmichalek132 commented Oct 8, 2024

dashpole commented Oct 8, 2024

jmichalek132 commented Oct 9, 2024

dashpole commented Oct 9, 2024

jmichalek132 commented Oct 9, 2024

dashpole commented Oct 9, 2024

aknuds1 commented Oct 9, 2024

dashpole commented Oct 9, 2024 •

edited

Loading

jmichalek132 commented Oct 9, 2024

ArthurSens Oct 9, 2024 •

edited

Loading

dashpole Oct 9, 2024

ArthurSens Oct 9, 2024

dashpole Oct 9, 2024

jmichalek132 Oct 10, 2024

ArthurSens Oct 10, 2024 •

edited

Loading

jmichalek132 Oct 10, 2024

jmichalek132 Oct 10, 2024

jmichalek132 commented Oct 10, 2024

bertysentry commented Oct 10, 2024 via email

feat: added translation rw2 translation for gauges #35703

feat: added translation rw2 translation for gauges #35703

Conversation

jmichalek132 commented Oct 8, 2024 • edited Loading

Description

Link to tracking issue #33661

Testing

Documentation

jmichalek132 commented Oct 8, 2024

dashpole commented Oct 8, 2024

jmichalek132 commented Oct 9, 2024

dashpole commented Oct 9, 2024

jmichalek132 commented Oct 9, 2024

dashpole commented Oct 9, 2024

aknuds1 commented Oct 9, 2024

dashpole commented Oct 9, 2024 • edited Loading

jmichalek132 commented Oct 9, 2024

ArthurSens Oct 9, 2024 • edited Loading

Choose a reason for hiding this comment

dashpole Oct 9, 2024

Choose a reason for hiding this comment

ArthurSens Oct 9, 2024

Choose a reason for hiding this comment

dashpole Oct 9, 2024

Choose a reason for hiding this comment

jmichalek132 Oct 10, 2024

Choose a reason for hiding this comment

ArthurSens Oct 10, 2024 • edited Loading

Choose a reason for hiding this comment

jmichalek132 Oct 10, 2024

Choose a reason for hiding this comment

jmichalek132 Oct 10, 2024

Choose a reason for hiding this comment

jmichalek132 commented Oct 10, 2024

bertysentry commented Oct 10, 2024 via email

jmichalek132 commented Oct 8, 2024 •

edited

Loading

dashpole commented Oct 9, 2024 •

edited

Loading

ArthurSens Oct 9, 2024 •

edited

Loading

ArthurSens Oct 10, 2024 •

edited

Loading