Add Datadog Exporter #25035

dineshg13 · 2024-04-23T20:58:53Z

What does this PR do?

The PR adds Datadog Exporter . It is thin wrapper around other exporter we have.
Datadog Exporter in Agent and Datadog Exporter in collector will share the code for exporting each data type .

We can't convert DD exporter as module at the moment as it has other agent dependencies. Will do so in the future PRs.

Motivation

Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

dineshg13 · 2024-04-23T21:01:54Z

comp/otelcol/otlp/components/exporter/datadogexporter/config.go

+// APIConfig defines the API configuration options
+type APIConfig struct {
+	// Key is the Datadog API key to associate your Agent's data with your organization.


This config is same as Datadog exporter config. In future we should ensure both exporters use same config.

pr-commenter · 2024-04-23T22:20:03Z

Test changes on VM

Use this command from test-infra-definitions to manually test this PR changes on a VM:

inv create-vm --pipeline-id=33669179 --os-family=ubuntu

pr-commenter · 2024-04-23T22:52:40Z

Regression Detector

Regression Detector Results

Run ID: 7e1677ae-4db0-4185-a527-f277d9511792
Baseline: 57020d1
Comparison: 4a3bb49

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

No significant changes in experiment optimization goals

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf	experiment	goal	Δ mean %	Δ mean % CI
➖	file_to_blackhole	% cpu utilization	+3.21	[-2.67, +9.09]

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI
➖	file_to_blackhole	% cpu utilization	+3.21	[-2.67, +9.09]
➖	file_tree	memory utilization	+1.64	[+1.52, +1.76]
➖	process_agent_real_time_mode	memory utilization	+0.36	[+0.31, +0.41]
➖	otel_to_otel_logs	ingress throughput	+0.09	[-0.28, +0.46]
➖	process_agent_standard_check	memory utilization	+0.07	[+0.01, +0.13]
➖	uds_dogstatsd_to_api	ingress throughput	+0.04	[-0.16, +0.24]
➖	tcp_dd_logs_filter_exclude	ingress throughput	+0.03	[-0.01, +0.07]
➖	trace_agent_json	ingress throughput	-0.00	[-0.01, +0.01]
➖	trace_agent_msgpack	ingress throughput	-0.01	[-0.02, +0.01]
➖	idle	memory utilization	-0.16	[-0.21, -0.11]
➖	process_agent_standard_check_with_stats	memory utilization	-0.17	[-0.22, -0.12]
➖	uds_dogstatsd_to_api_cpu	% cpu utilization	-0.33	[-3.17, +2.51]
➖	basic_py_check	% cpu utilization	-0.46	[-3.04, +2.13]
➖	tcp_syslog_to_blackhole	ingress throughput	-1.25	[-22.57, +20.08]
➖	pycheck_1000_100byte_tags	% cpu utilization	-1.73	[-6.20, +2.75]

Explanation

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

songy23

One meta comment: a bunch of the configs do not apply to otel agent, e.g. logs.dump_payloads host_metadata etc. If those are set should they result in config warnings?

comp/otelcol/otlp/components/exporter/datadogexporter/config.go

comp/otelcol/otlp/components/exporter/datadogexporter/metadata.go

dineshg13 · 2024-04-24T18:05:37Z

@songy23 We should aim to have same config for both DD exporter in Agent vs DD exporter in collector. Hence I tried to replicate all the config. Some of the config would be noop for OTel agent

songy23 · 2024-04-24T18:17:19Z

Yes - my point is we should warn the users in config_warnings for configs that are no-op in OTel agent. Otherwise they will wonder why the same config has effect in collector contrib but not in otel agent.

songy23

Discussed offline, will make the change above in a follow-up PR.

…IS-13

comp/otelcol/otlp/components/exporter/datadogexporter/factory.go

…IS-13

songy23 · 2024-04-29T18:12:29Z

comp/otelcol/otlp/components/exporter/datadogexporter/factory.go

+	lf := logsagentexporter.NewFactory(logch)
+	lc := &logsagentexporter.Config{
+		OtelSource:    "otel_agent",
+		LogSourceName: "otelcol",


Suggested change

LogSourceName: "otelcol",

LogSourceName: "otlp_log_ingestion",

as we discussed

liustanley · 2024-04-29T18:12:33Z

comp/otelcol/otlp/components/exporter/datadogexporter/factory.go

+	lf := logsagentexporter.NewFactory(logch)
+	lc := &logsagentexporter.Config{
+		OtelSource:    "otel_agent",
+		LogSourceName: "otelcol",


This should be set to logsagentexporter.logSourceName ("OTLP log ingestion") to be consistent

dineshg13 · 2024-05-06T14:12:34Z

/merge

CelianR

LGTM for ci-experience files

dd-devflow · 2024-05-06T14:17:26Z

🚂 MergeQueue

Pull request added to the queue.

There are 6 builds ahead! (estimated merge in less than 3h)

Use /merge -c to cancel this operation!

dd-devflow · 2024-05-06T14:39:56Z

❌ MergeQueue

Build pipeline has failing jobs for c3765c9

Since those jobs are not marked as being allowed to fail, the pipeline will most likely fail.
Therefore, and to allow other builds to be processed, this merge request has been rejected before the end of the pipeline.

You should have a look at the pipeline, wait for the build to finish and investigate the failures.
When you are ready, re-add your pull request to the queue!

⚠️

Do NOT retry failed jobs directly.

They were executed on a temporary branch created by the merge queue. If you retry them, they are going to fail because the branch no longer exists.

Details

List of failed jobs:

go_mod_tidy_check

Go to failed Gitlab pipeline.

If you need support, contact us on Slack #devflow with those details!

dineshg13 · 2024-05-06T17:27:32Z

/merge

dd-devflow · 2024-05-06T17:27:38Z

🚂 MergeQueue

Pull request added to the queue.

There are 2 builds ahead! (estimated merge in less than 1h)

Use /merge -c to cancel this operation!

Add Datadog Exporter

3db9ad8

github-actions bot added the team/opentelemetry OpenTelemetry team label Apr 23, 2024

dineshg13 added the changelog/no-changelog label Apr 23, 2024

dineshg13 commented Apr 23, 2024

View reviewed changes

dineshg13 marked this pull request as ready for review April 23, 2024 21:02

dineshg13 requested a review from a team as a code owner April 23, 2024 21:02

dineshg13 requested review from dinooliva and songy23 and removed request for dinooliva April 23, 2024 21:02

fix lint issues

9f0c993

songy23 reviewed Apr 24, 2024

View reviewed changes

comp/otelcol/otlp/components/exporter/datadogexporter/config.go Show resolved Hide resolved

comp/otelcol/otlp/components/exporter/datadogexporter/metadata.go Show resolved Hide resolved

Merge branch 'main' into dinesh.gurumurthy/OASIS-13

848737c

songy23 approved these changes Apr 29, 2024

View reviewed changes

dineshg13 added 3 commits April 29, 2024 11:06

use logspipeline

93ef5b7

Merge remote-tracking branch 'origin/main' into dinesh.gurumurthy/OAS…

08a7133

…IS-13

create module of datadogexporter

4642a62

dineshg13 requested review from a team as code owners April 29, 2024 15:24

fix lint issues

9fc4986

songy23 reviewed Apr 29, 2024

View reviewed changes

comp/otelcol/otlp/components/exporter/datadogexporter/factory.go Outdated Show resolved Hide resolved

dineshg13 added 3 commits April 29, 2024 13:00

update the license

5a6519a

Merge remote-tracking branch 'origin/main' into dinesh.gurumurthy/OAS…

35fb1ee

…IS-13

update the license

c829755

songy23 reviewed Apr 29, 2024

View reviewed changes

liustanley reviewed Apr 29, 2024

View reviewed changes

add replace

3311ecf

dineshg13 added 4 commits April 29, 2024 14:30

update the log source name

8aa4ae9

use 0.98

bdd1b33

merge main

a50517c

update replace statements

a345fa3

CelianR approved these changes May 6, 2024

View reviewed changes

FlorentClarret approved these changes May 6, 2024

View reviewed changes

dineshg13 added 2 commits May 6, 2024 10:40

Merge branch 'main' into dinesh.gurumurthy/OASIS-13

46ba499

run go mod

4a3bb49

dd-mergequeue bot merged commit 4e02f47 into main May 6, 2024
195 of 196 checks passed

dd-mergequeue bot deleted the dinesh.gurumurthy/OASIS-13 branch May 6, 2024 18:07

github-actions bot added this to the 7.55.0 milestone May 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Datadog Exporter #25035

Add Datadog Exporter #25035

dineshg13 commented Apr 23, 2024 •

edited

Loading

dineshg13 Apr 23, 2024

pr-commenter bot commented Apr 23, 2024 •

edited

Loading

pr-commenter bot commented Apr 23, 2024 •

edited

Loading

Experiments ignored for regressions

Fine details of change detection per experiment

Explanation

songy23 left a comment

dineshg13 commented Apr 24, 2024

songy23 commented Apr 24, 2024

songy23 left a comment

songy23 Apr 29, 2024

liustanley Apr 29, 2024

dineshg13 commented May 6, 2024

CelianR left a comment

dd-devflow bot commented May 6, 2024

dd-devflow bot commented May 6, 2024

dineshg13 commented May 6, 2024

dd-devflow bot commented May 6, 2024

	LogSourceName: "otelcol",
	LogSourceName: "otlp_log_ingestion",

Add Datadog Exporter #25035

Add Datadog Exporter #25035

Conversation

dineshg13 commented Apr 23, 2024 • edited Loading

What does this PR do?

Motivation

Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

dineshg13 Apr 23, 2024

Choose a reason for hiding this comment

pr-commenter bot commented Apr 23, 2024 • edited Loading

Test changes on VM

pr-commenter bot commented Apr 23, 2024 • edited Loading

Regression Detector

Regression Detector Results

No significant changes in experiment optimization goals

Experiments ignored for regressions

Fine details of change detection per experiment

Explanation

songy23 left a comment

Choose a reason for hiding this comment

dineshg13 commented Apr 24, 2024

songy23 commented Apr 24, 2024

songy23 left a comment

Choose a reason for hiding this comment

songy23 Apr 29, 2024

Choose a reason for hiding this comment

liustanley Apr 29, 2024

Choose a reason for hiding this comment

dineshg13 commented May 6, 2024

CelianR left a comment

Choose a reason for hiding this comment

dd-devflow bot commented May 6, 2024

dd-devflow bot commented May 6, 2024

dineshg13 commented May 6, 2024

dd-devflow bot commented May 6, 2024

dineshg13 commented Apr 23, 2024 •

edited

Loading

pr-commenter bot commented Apr 23, 2024 •

edited

Loading

pr-commenter bot commented Apr 23, 2024 •

edited

Loading