Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[8.x](backport #41206) [Filebeat] [AWS] add support to source logs from AWS linked source accounts when using log_group_name_prefix #41247

Merged
merged 1 commit into from
Oct 15, 2024

Conversation

mergify[bot]
Copy link
Contributor

@mergify mergify bot commented Oct 15, 2024

Proposed commit message

This is a follow-up to #41188 where I am adding support to source linked accounts when using log_group_name_prefix to derive log groups.

PR introduce include_linked_accounts_for_prefix_mode boolean property, which is disabled by default. If enabled (include_linked_accounts_for_prefix_mode : true), then we set includeLinkedAccounts property of the DescribeLogGroups API [1] to obtain log groups matching prefix and included in linked accounts of the monitoring account.

ex:-

- type: aws-cloudwatch
  ...
  log_group_name_prefix : /development/AppA/
  include_linked_accounts_for_prefix_mode: true
  ... 

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

This require a linked cloudwatch account. If already has one, then,

  • Push logs to a newly created log group OR use an already existing log group in a source account
    • Note - you may use data-gen Go program to generate and push logs to your log group (using output CLOUDWATCH_LOG) [2]
  • Configure filebeat cloudwatch input with log_group_name_prefix with desired prefix & set include_linked_accounts_for_prefix_mode to value true (enabled)
  • Run filebeat and observe logs in Kibana discover which include logs from log groups (that match provided prefix)

Related issues

[1] - https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_DescribeLogGroups.html
[2] - https://github.com/Kavindu-Dodan/data-gen


This is an automatic backport of pull request #41206 done by Mergify.

…counts when using log_group_name_prefix (#41206)

* configuration parsing to support arn & linked accounts

Signed-off-by: Kavindu Dodanduwa <[email protected]>

# Conflicts:
#	x-pack/filebeat/input/awscloudwatch/input.go

* code review change - fix typo

Signed-off-by: Kavindu Dodanduwa <[email protected]>

* add support to linked accounts when using prefix mode

Signed-off-by: Kavindu Dodanduwa <[email protected]>

* add changelog entry

Signed-off-by: Kavindu Dodanduwa <[email protected]>

* review suggestion

Signed-off-by: Kavindu Dodanduwa <[email protected]>

* use non-pointer struct property

Signed-off-by: Kavindu Dodanduwa <[email protected]>

---------

Signed-off-by: Kavindu Dodanduwa <[email protected]>
(cherry picked from commit 7e1b528)
@mergify mergify bot requested review from a team as code owners October 15, 2024 20:11
@mergify mergify bot added the backport label Oct 15, 2024
@mergify mergify bot requested review from mauri870 and faec and removed request for a team October 15, 2024 20:11
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 15, 2024
@Kavindu-Dodan Kavindu-Dodan added the Team:obs-ds-hosted-services Label for the Observability Hosted Services team label Oct 15, 2024
@elasticmachine
Copy link
Collaborator

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 15, 2024
@pierrehilbert pierrehilbert enabled auto-merge (squash) October 15, 2024 21:00
@pierrehilbert pierrehilbert merged commit ac439c8 into 8.x Oct 15, 2024
22 checks passed
@pierrehilbert pierrehilbert deleted the mergify/bp/8.x/pr-41206 branch October 15, 2024 21:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport Team:obs-ds-hosted-services Label for the Observability Hosted Services team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants