-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add datadog trace translation helpers and tests #1208
Conversation
Codecov Report
@@ Coverage Diff @@
## master #1208 +/- ##
==========================================
+ Coverage 88.98% 89.65% +0.66%
==========================================
Files 284 287 +3
Lines 13804 14097 +293
==========================================
+ Hits 12283 12638 +355
+ Misses 1162 1076 -86
- Partials 359 383 +24
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
@rmfitzpatrick lmk if any questions about ddog format, happy to clarify anything, any feedback is much appreciated as i'm quite new to the collector |
keySpanName string = "span.name" | ||
httpStatusCode string = "http.status_code" | ||
httpMethod string = "http.method" | ||
httpRoute string = "http.route" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These are available as http attribute names: https://github.com/open-telemetry/opentelemetry-collector/blob/c8622d2353485e8aa9ebde351feee431fc46f2f9/translator/conventions/opentelemetry.go#L183
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ah, nice, updated these to rely on convention constants when available.
|
||
attrs := resource.Attributes() | ||
if attrs.Len() == 0 { | ||
return tracetranslator.ResourceNoAttrs, datadogTags |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Curious why this and ResourceNotSet would be preferable to ResourceNoServiceName
as a Datadog service name.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's no real strong beliefs here, I think I was mostly mimicking the behavior of zipkin's translation logic, here: https://github.com/open-telemetry/opentelemetry-collector/blob/035aa5cf6c92678c3f8362d6916e18444a8250fb/translator/trace/zipkin/zipkinv2_to_traces.go#L353-L360
That being said it's less to worry about to just have a canonical "There's no resource service.name
tag", so I've followed your advice here and just simplified to return ResourceNoServiceName
in either case.
} | ||
|
||
// converts Traces into an array of datadog trace payloads grouped by env | ||
func ConvertToDatadogTd(td pdata.Traces, cfg *Config, globalTags []string) ([]*pb.TracePayload, error) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Where do these :-separated global tags come from?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These are passed in as a config option in setup
Tags []string `mapstructure:"tags"` |
"feature_flag:xyz, cloud_provider:alibaba"
or anything along those lines.
@ericmustin this looks good to me, though I do have a few nits and questions. |
@rmfitzpatrick ty for taking a look, feedback is super reasonable and makes sense, i should be able to address/incorporate each point mentioned relatively quickly |
@rmfitzpatrick tried to address all the feedback/questions here, lmk if anything else pops up, appreciate the review. |
It looks like master is failing after this PR was merged: https://app.circleci.com/pipelines/github/open-telemetry/opentelemetry-collector-contrib?branch=master How did this PR pass the build? |
@ericmustin I am going to revert this commit to fix the master build while you figure out what is the right solution. Please note that we don't allow CGO in the Collector. |
@tigrannajaryan I believe the The followup PR to this ( #1266) also fixes master as I've marked the files causing issues bc off gcc/cgo issues to not build on windows. If you'd like I can separate that out so that we can roll forward here? |
…d tests (open-telemetry#1208)" This reverts commit 8d6f04c. The "master" build was failing: https://app.circleci.com/pipelines/github/open-telemetry/opentelemetry-collector-contrib/4960/workflows/85f1e1cf-60fd-44d1-88f5-2c86884038c0/jobs/36766 It appears it was because the new commit needs CGO. We don't allow CGO in the Collector. I am reverting this temporarily until a proper solution is found.
This PR adds flushing+export of traces and trace-related statistics to the `datadogexporter`, as well as some very minor changes to the translation of internal traces into Datadog format. It represents the second of two PRs for the work contained in #1203. It builds on top of current master branch, and follows up to the work [done here](#1208). The final PR explicitly enabling The Datadog exporter will follow, and will allow users to export traces to Datadog's API Intake. This PR Split was requested by @tigrannajaryan and hopefully should make code review a bit less cumbersome. However if there are any questions or changes to the PR format needed, please let me know. **Testing:** There are unit tests for the different methods and helper methods within the export code. **Documentation:** Appropriate usage, including best practices for which processors to also enable, has been documented in the README, `testdata/config.yaml` and `example/config.yaml` samples. **Notes**: This PR includes a trace exporter for non-windows environments only (metrics are fine in windows, just traces that are the issue), due to reasons explained in this pr #1274 . tl;dr is our trace export code for windows env would rely on CGO for now, which is not permitted in the collector
* Add API key validation (#1216) Adds API key validation to the Datadog metrics exporter. When created, the Datadog metrics exporter now sends a requests to the `/api/v1/validate` endpoint of the Datadog backend to check that the configured API key is valid. If it's not, a warning log is emitted. Tests were amended to take into account that validation call. Test utils were added to mock an HTTP server that performs validation. * sapmexporter: make span source attribute and destination dimension names configurable (#1286) If dimension names are being translated in the signalfxexporter then the map values should be set to the signalfx names. Ideally we can sync to OT dimension names with translation being done on the backend (the default). * Update README (#1294) * Release v0.13.0 (#1295) * Remove duplicate definition of cloud providers with core conventions (#1288) * Remove duplicate definition of cloud providers Signed-off-by: Bogdan Drutu <[email protected]> * Fix more duplicate usage of the cloud providers semconv Signed-off-by: Bogdan Drutu <[email protected]> * Splunkhec receiver metrics (#1276) Adds the ability for Splunk HEC to ingest metrics. This is a follow up to #1268 which adds the ability to ingest logs. * Add jpkrohling as an approver (#1296) * Remove pjanotti from maintainers (#1300) * Auto assign approver and maintainers to PRs (#1301) Signed-off-by: Bogdan Drutu <[email protected]> * Add codeowners to ensure components are assigned to the appropriate reviewers (#1304) This is the initial list extracted from README. * Moved the groupbytrace processor to contrib (#1179) Signed-off-by: Juraci Paixão Kröhling <[email protected]> Co-authored-by: Bogdan Drutu <[email protected]> * Add codeowners for interanl components (#1307) Signed-off-by: Bogdan Drutu <[email protected]> * Small fixes to CODEOWNERS (#1312) Signed-off-by: Juraci Paixão Kröhling <[email protected]> **Description:** This PR changes the CODEOWNERS in a couple of aspects: 1. Fixed the order of the directories, so that 'internal' comes after 'extension' 1. Fixed the name of a few components 1. Added missing components and directories Verified with: ``` for component in exporter extension processor receiver; do ls ${component}/ -1 > /tmp/${component}.txt grep ${component} .github/CODEOWNERS | awk -F\/ '{print $2}' > /tmp/${component}-codeowners.txt diff /tmp/${component}.txt /tmp/${component}-codeowners.txt done ``` Result of the script before this PR: ```diff 11d10 < loadbalancingexporter 2c2 < jmxmetricsextension --- > jmxmetrics 1d0 < groupbytraceprocessor 5c4 < routingprocessor --- > routing ``` * Update collector version in groupbytraceprocessor (#1309) Signed-off-by: Bogdan Drutu <[email protected]> * Update dependabot to ensure all projects are added (#1303) * Update dependabot to ensure all projects are added Signed-off-by: Bogdan Drutu <[email protected]> * Update dependabot.yml * Do not run tests/lint/etc for all component tags (e.g. tag testbed/v0.13.0) (#1298) Signed-off-by: Bogdan Drutu <[email protected]> * tests: increase TestTrace10kSPS memory limits (#1314) * Bump k8s.io/client-go in /receiver/k8sclusterreceiver (#1323) Bumps [k8s.io/client-go](https://github.com/kubernetes/client-go) from 0.19.2 to 0.19.3. - [Release notes](https://github.com/kubernetes/client-go/releases) - [Changelog](https://github.com/kubernetes/client-go/blob/master/CHANGELOG.md) - [Commits](kubernetes/client-go@v0.19.2...v0.19.3) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump github.com/aliyun/aliyun-log-go-sdk (#1321) Bumps [github.com/aliyun/aliyun-log-go-sdk](https://github.com/aliyun/aliyun-log-go-sdk) from 0.1.13 to 0.1.14. - [Release notes](https://github.com/aliyun/aliyun-log-go-sdk/releases) - [Commits](aliyun/aliyun-log-go-sdk@v0.1.13...v0.1.14) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump go.opencensus.io in /processor/groupbytraceprocessor (#1320) Bumps [go.opencensus.io](https://github.com/census-instrumentation/opencensus-go) from 0.22.4 to 0.22.5. - [Release notes](https://github.com/census-instrumentation/opencensus-go/releases) - [Commits](census-instrumentation/opencensus-go@v0.22.4...v0.22.5) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump k8s.io/client-go from 0.19.2 to 0.19.3 in /internal/k8sconfig (#1318) Bumps [k8s.io/client-go](https://github.com/kubernetes/client-go) from 0.19.2 to 0.19.3. - [Release notes](https://github.com/kubernetes/client-go/releases) - [Changelog](https://github.com/kubernetes/client-go/blob/master/CHANGELOG.md) - [Commits](kubernetes/client-go@v0.19.2...v0.19.3) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add more @elastic folks to codeowners (#1313) In answer to open-telemetry#1304 (comment) add two more codeowners for the Elastic exporter. * Add contrib approvers as owners to all the components. (#1325) Without this change if there is a listed owner with write permission in the component owners list, the contrib approvers will lose their power see #1316. Signed-off-by: Bogdan Drutu <[email protected]> * Bump github.com/aws/aws-sdk-go in /internal/awsxray (#1316) Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.35.9 to 1.35.10. - [Release notes](https://github.com/aws/aws-sdk-go/releases) - [Changelog](https://github.com/aws/aws-sdk-go/blob/master/CHANGELOG.md) - [Commits](aws/aws-sdk-go@v1.35.9...v1.35.10) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump github.com/aws/aws-sdk-go in /internal/awsxray/testdata/sampleapp (#1317) Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.35.9 to 1.35.10. - [Release notes](https://github.com/aws/aws-sdk-go/releases) - [Changelog](https://github.com/aws/aws-sdk-go/blob/master/CHANGELOG.md) - [Commits](aws/aws-sdk-go@v1.35.9...v1.35.10) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Clarify PR reviewing and facilitating (#1315) We recently introduced the automatic assignments of PRs to reviewers and to facilitators. This change explains the process. * Handle nil references from the kubelet API (#1326) * Update to latest collector, update deprecated calls (#1308) Signed-off-by: Bogdan Drutu <[email protected]> * signalfx Receiver: Better Pipeline Error Handling (#1329) If logs aren't configured and events are sent, return a clear error response instead of panicing. Vice-versa for metrics. * [datadogexporter] Improve hostname resolution (#1285) Improve system hostname detection for the Datadog exporter. This PR: - Moves config and host code to their own packages to avoid dependency cycles - Adds hostname validation - Adds fully qualified domain name hostname resolution on some platforms - Adds support for caching hostname Added unit tests, tested on an end to end test environment with the component activated. Documentation was added to all public functions. * Temporarily remove dmitryax from PR facilitators (#1330) dmitryax will be unavailable for a while, removing him from the list of PR facilitators. * Update otel collector, fix breaking change for renaming TracesConsumer (#1328) * Update otel collector, fix breaking change for renaming TracesConsumer Signed-off-by: Bogdan Drutu <[email protected]> * More fixes of usages Signed-off-by: Bogdan Drutu <[email protected]> * Add batchpertrace library (#1257) Signed-off-by: Juraci Paixão Kröhling <[email protected]> Adds a library that will split the incoming batch into several batches, one per trace. **Link to tracking Issue:** Closes #1235. * Fix the link to the release notes (#1327) * Datadog trace flushing/export (#1266) This PR adds flushing+export of traces and trace-related statistics to the `datadogexporter`, as well as some very minor changes to the translation of internal traces into Datadog format. It represents the second of two PRs for the work contained in open-telemetry#1203. It builds on top of current master branch, and follows up to the work [done here](open-telemetry#1208). The final PR explicitly enabling The Datadog exporter will follow, and will allow users to export traces to Datadog's API Intake. This PR Split was requested by @tigrannajaryan and hopefully should make code review a bit less cumbersome. However if there are any questions or changes to the PR format needed, please let me know. **Testing:** There are unit tests for the different methods and helper methods within the export code. **Documentation:** Appropriate usage, including best practices for which processors to also enable, has been documented in the README, `testdata/config.yaml` and `example/config.yaml` samples. **Notes**: This PR includes a trace exporter for non-windows environments only (metrics are fine in windows, just traces that are the issue), due to reasons explained in this pr open-telemetry#1274 . tl;dr is our trace export code for windows env would rely on CGO for now, which is not permitted in the collector * Logzio exporter impl (#1161) Added a logz.io traces exporter **Link to tracking Issue**: #686 **Testing**: Added test for each of the components in the new exporter **Documentation**: Added a readme specifying how to use the exporter and its parameters with an example. * Add the notion of unstable components and unstable executable (#1299) The list of experimental components is defined in unstable_components_enabled.go. These components are only enabled when enable_unstable build tag is defined. We define this tag and produce an executable named otelcontribcol_unstable_$(GOOS)_$(GOARCH)$(EXTENSION) when `make otelcontribcol-unstable` is invoked. For now the new executable is not used anywhere. Next I will look into modifying the testbed to call the new unstable executable for certain tests. To verify that the unstable build functionality is enabled I added stanzareceiver to the list of unstable components and manually verified that it is indeed enabled in the unstable executable but is not available in the regular otelcontribcol executable. Contributes to open-telemetry#873 * JMX Metric Extension: Initial implementation (#1182) * Add JMX Metric Extension implementation * rename package to jmxmetricextension * jmxmetricextension s/metrics/metric * jmx metrics: fix prometheus typo * jmx metrics: capitalize acronyms * jmx metrics: clarify interval * Enable stale PR action (#1341) To help reviewers and authors remember to make progress on PR this action will mark PRs as stale after inactivity of 7 days and will close the PR after 7 more days of inactivity. * [datadogexporter] Enable traces on Windows (#1340) * Re-enable traces code on Windows Use a custom-made version of the Datadog Agent repository that greatly reduces the number of dependencies needed and removes the osext one that depends on CGo * Address linter issue * Empty commit to retrigger CI * Build traces flush/export code on Windows * Add kind type to root span to fix the empty parentID problem (#1338) * Add kind type to root span to fix the empty parentID problem * Set kind type for root span in Xray receiver * Update receiver/awsxrayreceiver/internal/translator/translator.go Co-authored-by: Anuraag Agrawal <[email protected]> Co-authored-by: Bogdan Drutu <[email protected]> Co-authored-by: Anuraag Agrawal <[email protected]> * [awsecscontainermetrics] receiver- Update README (#1358) * [awsecscontainermetrics] receiver- Update README Signed-off-by: Rayhan Hossain <[email protected]> * Use full form of metric units Signed-off-by: Rayhan Hossain <[email protected]> * Add timer support for statsD receiver (#1335) * [datadogexporter] Add Datadog exporter to the otelcontribcol binary (#1352) * Add datadogexporter to the binary * Disable environment variables They don't work; we will revisit it in the future * [datadogexporter] Update go-datadog-api.v2 dependency to v2.30.0 (#1365) * [signalfx_correlation] Add signalfx_correlation exporter skeleton (#1332) * [signalfx_correlation] Add signalfx_correlation exporter skeleton This is for moving the correlation out of sapmexporter into a dedicated exporter so that the correlation can be used even when sapm isn't (for example, on an agent that is exporting in otlp to a gateway instead of sapm.) * fix readme * [awsemfexporter] Restructure Metric Translator Logic (#1353) * Restructure buildCWMetric logic (#1) * Restructure code to remove duplicated logic * Update format * Improve function and variable names * Extract logic for dimension creation and add test * Implement minor fixes * Remove changes to go.sum * Implement tests for getCWMetrics * Implement tests for buildCWMetric * Format metric_translator_test.go * Run with gofmt -s * Disregard ordering of dimensions in test case * Perform dimension equality checking as a helper function * Setting the tlsconfig InsecureSkipVerify using NoVerifySSL (#1350) Co-authored-by: Kylian Serrania <[email protected]> Co-authored-by: Jay Camp <[email protected]> Co-authored-by: Steve Flanders <[email protected]> Co-authored-by: Jeff Cheng <[email protected]> Co-authored-by: Bogdan Drutu <[email protected]> Co-authored-by: Antoine Toulme <[email protected]> Co-authored-by: Tigran Najaryan <[email protected]> Co-authored-by: Paulo Janotti <[email protected]> Co-authored-by: Juraci Paixão Kröhling <[email protected]> Co-authored-by: Eric Mustin <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Andrew Wilkins <[email protected]> Co-authored-by: Pablo Collins <[email protected]> Co-authored-by: Ben Keith <[email protected]> Co-authored-by: Pablo Baeyens <[email protected]> Co-authored-by: Yogev Mets <[email protected]> Co-authored-by: Ryan Fitzpatrick <[email protected]> Co-authored-by: John <[email protected]> Co-authored-by: Bogdan Drutu <[email protected]> Co-authored-by: Anuraag Agrawal <[email protected]> Co-authored-by: Rayhan Hossain (Mukla.C) <[email protected]> Co-authored-by: Gavin Zhang (Kunyuan Zhang) <[email protected]> Co-authored-by: shreyas Darwhatkar <[email protected]>
This PR adds flushing+export of traces and trace-related statistics to the `datadogexporter`, as well as some very minor changes to the translation of internal traces into Datadog format. It represents the second of two PRs for the work contained in open-telemetry#1203. It builds on top of current master branch, and follows up to the work [done here](open-telemetry#1208). The final PR explicitly enabling The Datadog exporter will follow, and will allow users to export traces to Datadog's API Intake. This PR Split was requested by @tigrannajaryan and hopefully should make code review a bit less cumbersome. However if there are any questions or changes to the PR format needed, please let me know. **Testing:** There are unit tests for the different methods and helper methods within the export code. **Documentation:** Appropriate usage, including best practices for which processors to also enable, has been documented in the README, `testdata/config.yaml` and `example/config.yaml` samples. **Notes**: This PR includes a trace exporter for non-windows environments only (metrics are fine in windows, just traces that are the issue), due to reasons explained in this pr open-telemetry#1274 . tl;dr is our trace export code for windows env would rely on CGO for now, which is not permitted in the collector
…ent protocols (#1208) Signed-off-by: Bogdan Drutu <[email protected]>
- Test by span attributes added by processors in the order they were registered. Signed-off-by: Hui Kang <[email protected]> Co-authored-by: Tyler Yahn <[email protected]>
Description: This PR adds utilities to the datadog exporter to translate internal trace representation to Datadog format. It represents the first of two PRs for the work contained in #1203. It builds on top of current master branch.
The 2nd PR for this work will follow, and will add functionality to export traces to Datadog's API Intake.
This PR Split was requested by @tigrannajaryan and hopefully should make code review a bit less cumbersome. However if there are any questions or changes to the PR format needed, please let me know.
For the trace translation we are not relying on OpenCensus helpers as other exporters do based on this Gitter thread.
Testing: There are unit tests for the different methods and helper methods within the translation code.
Fwiw everything passes locally, on the linked PR there appeared to be some compilation issues for windows due to a dependancy. I can investigate this more if it persist.transient issue, disregardDocumentation: Currently the trace exporter code isn't contained in this PR, but when it is added (in a subsequent PR), it's usage will be documented in the
testdata/config.yaml
andexample/config.yaml
samples.example trace