Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[8.x](backport #40699) Add asynchronous ACK handling to S3 and SQS inputs #41249

Merged
merged 2 commits into from
Oct 16, 2024

Conversation

mergify[bot]
Copy link
Contributor

@mergify mergify bot commented Oct 16, 2024

Modify SQS ingestion to listen for ACKs asynchronously so that input workers can keep reading new objects after a previous one has been published, instead of blocking on full upstream ingestion. This addresses the bottleneck where ingesting many small objects is slow as each one waits for a full ingestion round trip.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

This can best be tested by ingesting data from a live S3 or SQS queue. The scenario that most highlights the changed performance is ingesting many small individual objects.

Related issues

Modify SQS ingestion to listen for ACKs asynchronously so that input workers can keep reading new objects after a previous one has been published, instead of blocking on full upstream ingestion. This addresses the bottleneck where ingesting many small objects is slow as each one waits for a full ingestion round trip. With a default configuration, SQS queues with many small objects are now ingested up to 60x faster.

(cherry picked from commit d2867fd)

# Conflicts:
#	x-pack/filebeat/input/awss3/input_benchmark_test.go
#	x-pack/filebeat/input/awss3/sqs_s3_event_test.go
@mergify mergify bot requested review from a team as code owners October 16, 2024 02:53
@mergify mergify bot added backport conflicts There is a conflict in the backported pull request labels Oct 16, 2024
@mergify mergify bot requested a review from a team as a code owner October 16, 2024 02:53
@mergify mergify bot requested review from AndersonQ and leehinman and removed request for a team October 16, 2024 02:53
@mergify mergify bot assigned faec Oct 16, 2024
Copy link
Contributor Author

mergify bot commented Oct 16, 2024

Cherry-pick of d2867fd has failed:

On branch mergify/bp/8.x/pr-40699
Your branch is up to date with 'origin/8.x'.

You are currently cherry-picking commit d2867fdd9f.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   CHANGELOG.next.asciidoc
	modified:   NOTICE.txt
	modified:   go.mod
	modified:   go.sum
	modified:   x-pack/filebeat/_meta/config/filebeat.inputs.reference.xpack.yml.tmpl
	modified:   x-pack/filebeat/docs/inputs/input-aws-s3.asciidoc
	modified:   x-pack/filebeat/filebeat.reference.yml
	new file:   x-pack/filebeat/input/awss3/acks.go
	modified:   x-pack/filebeat/input/awss3/config.go
	modified:   x-pack/filebeat/input/awss3/config_test.go
	modified:   x-pack/filebeat/input/awss3/input_integration_test.go
	modified:   x-pack/filebeat/input/awss3/interfaces.go
	modified:   x-pack/filebeat/input/awss3/mock_interfaces_test.go
	modified:   x-pack/filebeat/input/awss3/s3.go
	modified:   x-pack/filebeat/input/awss3/s3_input.go
	modified:   x-pack/filebeat/input/awss3/s3_objects.go
	modified:   x-pack/filebeat/input/awss3/s3_objects_test.go
	modified:   x-pack/filebeat/input/awss3/s3_test.go
	modified:   x-pack/filebeat/input/awss3/sqs_input.go
	modified:   x-pack/filebeat/input/awss3/sqs_s3_event.go
	modified:   x-pack/filebeat/input/awss3/sqs_test.go
	modified:   x-pack/filebeat/module/aws/_meta/config.yml
	modified:   x-pack/filebeat/module/aws/cloudtrail/config/aws-s3.yml
	modified:   x-pack/filebeat/module/aws/cloudtrail/manifest.yml
	modified:   x-pack/filebeat/module/aws/s3access/config/aws-s3.yml
	modified:   x-pack/filebeat/module/aws/s3access/manifest.yml
	modified:   x-pack/filebeat/module/aws/vpcflow/config/input.yml
	modified:   x-pack/filebeat/module/aws/vpcflow/manifest.yml
	modified:   x-pack/filebeat/modules.d/aws.yml.disabled

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   x-pack/filebeat/input/awss3/input_benchmark_test.go
	both modified:   x-pack/filebeat/input/awss3/sqs_s3_event_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 16, 2024
@botelastic
Copy link

botelastic bot commented Oct 16, 2024

This pull request doesn't have a Team:<team> label.

@faec faec enabled auto-merge (squash) October 16, 2024 04:37
@faec faec merged commit c5a983f into 8.x Oct 16, 2024
140 of 142 checks passed
@faec faec deleted the mergify/bp/8.x/pr-40699 branch October 16, 2024 05:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport conflicts There is a conflict in the backported pull request needs_team Indicates that the issue/PR needs a Team:* label
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant