Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[8.x](backport #40775) chore: add debug histograms for s3 object size and events per processed s3 object #40819

Merged
merged 1 commit into from
Sep 13, 2024

Conversation

mergify[bot]
Copy link
Contributor

@mergify mergify bot commented Sep 13, 2024

Proposed commit message

Fixes #40306 by adding s3_events_per_object & s3_object_size_in_bytes histograms. These histograms provide valuable insights on the size of S3 objects filebeat processed and the number of events produced by each S3 object.

Consider below screenshot,

image

  • Uploads two documents with 15 total logs (10 logs in one and another with 5 logs)
  • See how s3_events_per_object tracks events (logs) per object with max, min and count
  • See how s3_object_size_in_bytes tracks object size with max and min and count

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

  • Run filebeat with AWS S3 connectivity
  • Access debug metrics through http://<http.host>:<http.port>/inputs/?type=aws-s3
  • Observe s3_object_size_in_bytes & s3_events_per_object histograms

Related issues

…ed s3 object (#40775)

* add debug histograms for s3 object size and events per object

Signed-off-by: Kavindu Dodanduwa <[email protected]>

* changelog entry

Signed-off-by: Kavindu Dodanduwa <[email protected]>

* move changelog to developer next

Signed-off-by: Kavindu Dodanduwa <[email protected]>

---------

Signed-off-by: Kavindu Dodanduwa <[email protected]>
(cherry picked from commit 8f2c3f9)
@mergify mergify bot requested a review from a team as a code owner September 13, 2024 16:17
@mergify mergify bot added the backport label Sep 13, 2024
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 13, 2024
@botelastic
Copy link

botelastic bot commented Sep 13, 2024

This pull request doesn't have a Team:<team> label.

@Kavindu-Dodan Kavindu-Dodan requested review from Kavindu-Dodan and a team and removed request for a team September 13, 2024 16:18
Copy link
Contributor

@Kavindu-Dodan Kavindu-Dodan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM : This is a developer focused feature backport.

@Kavindu-Dodan Kavindu-Dodan merged commit 16a5280 into 8.x Sep 13, 2024
19 of 20 checks passed
@Kavindu-Dodan Kavindu-Dodan deleted the mergify/bp/8.x/pr-40775 branch September 13, 2024 18:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport needs_team Indicates that the issue/PR needs a Team:* label
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant