-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add logs agent pipeline performance telemetry #30744
Merged
dd-mergequeue
merged 23 commits into
main
from
brian/logs-pipeline-telemetry-AMLII-2134
Nov 15, 2024
Merged
Changes from 22 commits
Commits
Show all changes
23 commits
Select commit
Hold shift + click to select a range
0cea77d
Make channel sizes configurable
gh123man 185ea96
Introduce pipeline monitor, utilization monitor, capacity monitor, an…
gh123man a19dbb0
Refactor logs agent to use pipeline telemetry
gh123man a0c07a0
cleanup go mods
gh123man 22d27a6
Increase sample window to 15 seconds
gh123man 7201bae
Stop processor utilization in early returns.
gh123man fa2cba8
Use ewma instead of time buckets
gh123man 5fc6dbf
Merge branch 'main' into brian/logs-pipeline-telemetry-AMLII-2134
gh123man f9454d7
update experiments to use logrotate_FS
blt b7e1c64
Fix deps list
gh123man 899c878
Merge branch 'main' into brian/logs-pipeline-telemetry-AMLII-2134
gh123man d9929e8
PR feedback
gh123man 5033c78
PR feedback
gh123man b894b8f
PR feedback
gh123man f015f35
Defer processor egress
gh123man b1dd831
Remove dependency on 3rd party ewma library
gh123man 6e27f61
Refactor utilization tracker
gh123man 5c2e611
Refactor utilization monitor to use utilization_tracker
gh123man 2b953cc
Remaining PR feedback
gh123man 66dc166
Allow utilization monitor to call start/stop multiple times
gh123man beaf565
Merge branch 'main' into brian/logs-pipeline-telemetry-AMLII-2134
gh123man 815ad2a
Merge branch 'main' into brian/logs-pipeline-telemetry-AMLII-2134
gh123man 7f57f8c
Fix format
gh123man File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| // Unless explicitly stated otherwise all files in this repository are licensed | ||
| // under the Apache License Version 2.0. | ||
| // This product includes software developed at Datadog (https://www.datadoghq.com/). | ||
| // Copyright 2016-present Datadog, Inc. | ||
|
|
||
| //nolint:revive // TODO(AML) Fix revive linter | ||
| package client | ||
|
|
||
| import ( | ||
| "fmt" | ||
| ) | ||
|
|
||
| // DestinationMetadata contains metadata about a destination | ||
| type DestinationMetadata struct { | ||
| componentName string | ||
| instanceID string | ||
| kind string | ||
| endpointId string | ||
| ReportingEnabled bool | ||
| } | ||
|
|
||
| // NewDestinationMetadata returns a new DestinationMetadata | ||
| func NewDestinationMetadata(componentName, instanceID, kind, endpointId string) *DestinationMetadata { | ||
| return &DestinationMetadata{ | ||
| componentName: componentName, | ||
| instanceID: instanceID, | ||
| kind: kind, | ||
| endpointId: endpointId, | ||
| ReportingEnabled: true, | ||
| } | ||
| } | ||
|
|
||
| // NewNoopDestinationMetadata returns a new DestinationMetadata with reporting disabled | ||
| func NewNoopDestinationMetadata() *DestinationMetadata { | ||
| return &DestinationMetadata{ | ||
| ReportingEnabled: false, | ||
| } | ||
| } | ||
|
|
||
| // TelemetryName returns the telemetry name for the destination | ||
| func (d *DestinationMetadata) TelemetryName() string { | ||
| if !d.ReportingEnabled { | ||
| return "" | ||
| } | ||
| return fmt.Sprintf("%s_%s_%s_%s", d.componentName, d.instanceID, d.kind, d.endpointId) | ||
| } | ||
|
|
||
| // MonitorTag returns the monitor tag for the destination | ||
| func (d *DestinationMetadata) MonitorTag() string { | ||
| if !d.ReportingEnabled { | ||
| return "" | ||
| } | ||
| return fmt.Sprintf("destination_%s_%s", d.kind, d.endpointId) | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ideally we should provide the config component as part of
Newfunction, so we do not use thepkgconfigsetup.Datadog()global 😄Maybe something we could do on a separate PR