Skip to content

Commit

Permalink
[exporter/datadog] Refactor Datadog Exporter to use logs agent export…
Browse files Browse the repository at this point in the history 
…er (#32327)

**Description:** <Describe what has changed.>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
This PR adds support for the logs agent exporter in order to export logs
to Datadog. The following changes were made in order to support this:
- Update datadog-agent modules to the following commit hash version:
DataDog/datadog-agent@a1ef7ed
- Remove `peer_service_aggregation` config option which was deprecated
and removed since last datadog-agent version:
DataDog/datadog-agent#23284
- Add logs agent exporter setup in datadog exporter factory. This is
disabled by default and can be enabled via feature gate.

**Link to tracking Issue:** 

Fixes
#30099

**Testing:** <Describe what testing was performed and which tests were
added.>

**Documentation:** <Describe the documentation added.>

---------

Co-authored-by: Pablo Baeyens <[email protected]>
  • Loading branch information
@liustanley @mx-psi
liustanley and mx-psi authored May 6, 2024
1 parent 34e64e0 commit b270022
Show file tree
Hide file tree
Showing 28 changed files with 2,762 additions and 191 deletions.
13 changes: 13 additions & 0 deletions .chloggen/datadogexporter_logsagentexporter.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Use this changelog template to create an entry for release notes.

# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
change_type: enhancement

# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
component: exporter/datadog

# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
note: Introduces the Datadog Agent logs pipeline for exporting logs to Datadog under the "exporter.datadogexporter.UseLogsAgentExporter" feature gate.

# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
issues: [32327]
70 changes: 62 additions & 8 deletions cmd/configschema/go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -218,23 +218,67 @@ require (
github.com/ClickHouse/clickhouse-go/v2 v2.23.0 // indirect
github.com/Code-Hex/go-generics-cache v1.3.1 // indirect
github.com/DataDog/agent-payload/v5 v5.0.114 // indirect
github.com/DataDog/datadog-agent/pkg/obfuscate v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/proto v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/remoteconfig/state v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/trace v0.54.0-rc.1.0.20240502131837-32187be66dbd // indirect
github.com/DataDog/datadog-agent/pkg/util/cgroups v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/util/log v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/util/pointer v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/pkg/util/scrubber v0.54.0-rc.1 // indirect
github.com/DataDog/datadog-agent/comp/core/config v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/comp/core/hostname/hostnameinterface v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/comp/core/log v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/comp/core/secrets v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/comp/core/telemetry v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/comp/def v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/comp/logs/agent/config v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/comp/otelcol/logsagentpipeline v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/comp/otelcol/logsagentpipeline/logsagentpipelineimpl v0.54.0-rc.2.0.20240502211937-f7706eb2beae // indirect
github.com/DataDog/datadog-agent/comp/otelcol/otlp/components/exporter/logsagentexporter v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/collector/check/defaults v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/config/env v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/config/model v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/config/setup v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/config/utils v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/logs/auditor v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/logs/client v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/logs/diagnostic v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/logs/message v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/logs/metrics v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/logs/pipeline v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/logs/processor v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/logs/sds v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/logs/sender v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/logs/sources v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/logs/status/statusinterface v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/logs/status/utils v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/obfuscate v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/proto v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/remoteconfig/state v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/status/health v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/telemetry v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/trace v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/util/backoff v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/util/cgroups v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/util/executable v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/util/filesystem v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/util/fxutil v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/util/hostname/validate v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/util/http v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/util/log v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/util/optional v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/util/pointer v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/util/scrubber v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/util/startstop v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/util/statstracker v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/util/system v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/util/system/socket v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/util/winutil v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-agent/pkg/version v0.54.0-rc.2 // indirect
github.com/DataDog/datadog-api-client-go/v2 v2.25.0 // indirect
github.com/DataDog/datadog-go/v5 v5.5.0 // indirect
github.com/DataDog/dd-sensitive-data-scanner/sds-go/go v0.0.0-20240419161837-f1b2f553edfe // indirect
github.com/DataDog/go-sqllexer v0.0.9 // indirect
github.com/DataDog/go-tuf v1.1.0-0.5.2 // indirect
github.com/DataDog/gohai v0.0.0-20230524154621-4316413895ee // indirect
github.com/DataDog/opentelemetry-mapping-go/pkg/inframetadata v0.15.0 // indirect
github.com/DataDog/opentelemetry-mapping-go/pkg/otlp/logs v0.15.0 // indirect
github.com/DataDog/opentelemetry-mapping-go/pkg/quantile v0.15.0 // indirect
github.com/DataDog/sketches-go v1.4.4 // indirect
github.com/DataDog/viper v1.13.2 // indirect
github.com/DataDog/zstd v1.5.2 // indirect
github.com/GehirnInc/crypt v0.0.0-20200316065508-bb7000b8a962 // indirect
github.com/GoogleCloudPlatform/grpc-gcp-go/grpcgcp v1.5.0 // indirect
Expand Down Expand Up @@ -283,9 +327,11 @@ require (
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.23.4 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.28.6 // indirect
github.com/aws/smithy-go v1.20.2 // indirect
github.com/benbjohnson/clock v1.3.5 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/blang/semver/v4 v4.0.0 // indirect
github.com/bmatcuk/doublestar/v4 v4.6.1 // indirect
github.com/briandowns/spinner v1.23.0 // indirect
github.com/buger/jsonparser v1.1.1 // indirect
github.com/cenkalti/backoff v2.2.1+incompatible // indirect
github.com/cenkalti/backoff/v4 v4.3.0 // indirect
Expand Down Expand Up @@ -398,6 +444,7 @@ require (
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/hashicorp/nomad/api v0.0.0-20240306004928-3e7191ccb702 // indirect
github.com/hashicorp/serf v0.10.1 // indirect
github.com/hectane/go-acl v0.0.0-20190604041725-da78bae5fc95 // indirect
github.com/hetznercloud/hcloud-go/v2 v2.6.0 // indirect
github.com/iancoleman/strcase v0.3.0 // indirect
github.com/imdario/mergo v0.3.16 // indirect
Expand All @@ -424,6 +471,7 @@ require (
github.com/jpillora/backoff v1.0.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/julienschmidt/httprouter v1.3.0 // indirect
github.com/kardianos/osext v0.0.0-20190222173326-2bc1f35cddc0 // indirect
github.com/karrick/godirwalk v1.17.0 // indirect
github.com/klauspost/compress v1.17.8 // indirect
github.com/klauspost/cpuid/v2 v2.2.5 // indirect
Expand Down Expand Up @@ -462,6 +510,7 @@ require (
github.com/moby/sys/mountinfo v0.6.2 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826 // indirect
github.com/mongodb-forks/digest v1.1.0 // indirect
github.com/montanaflynn/stats v0.7.0 // indirect
github.com/mostynb/go-grpc-compression v1.2.2 // indirect
Expand Down Expand Up @@ -548,6 +597,7 @@ require (
github.com/ovh/go-ovh v1.4.3 // indirect
github.com/patrickmn/go-cache v2.1.0+incompatible // indirect
github.com/paulmach/orb v0.11.1 // indirect
github.com/pelletier/go-toml v1.9.3 // indirect
github.com/pelletier/go-toml/v2 v2.1.0 // indirect
github.com/philhofer/fwd v1.1.2 // indirect
github.com/pierrec/lz4 v2.6.1+incompatible // indirect
Expand Down Expand Up @@ -588,8 +638,10 @@ require (
github.com/spf13/afero v1.11.0 // indirect
github.com/spf13/cast v1.6.0 // indirect
github.com/spf13/cobra v1.8.0 // indirect
github.com/spf13/jwalterweatherman v1.1.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/spf13/viper v1.18.2 // indirect
github.com/stormcat24/protodep v0.1.8 // indirect
github.com/stretchr/objx v0.5.2 // indirect
github.com/subosito/gotenv v1.6.0 // indirect
github.com/syndtr/gocapability v0.0.0-20200815063812-42c35b437635 // indirect
Expand Down Expand Up @@ -679,6 +731,8 @@ require (
go.opentelemetry.io/otel/trace v1.26.0 // indirect
go.opentelemetry.io/proto/otlp v1.2.0 // indirect
go.uber.org/atomic v1.11.0 // indirect
go.uber.org/dig v1.17.0 // indirect
go.uber.org/fx v1.18.2 // indirect
go.uber.org/multierr v1.11.0 // indirect
go.uber.org/zap v1.27.0 // indirect
golang.org/x/crypto v0.22.0 // indirect
Expand Down
Loading

0 comments on commit b270022

Please sign in to comment.