-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add asynchronous ACK handling to S3 and SQS inputs #40699
Merged
Merged
Changes from all commits
Commits
Show all changes
41 commits
Select commit
Hold shift + click to select a range
e9ccebe
working on S3 ack tracking
faec b17f7b5
handle S3 acks asynchronously
faec d023027
fix loop variable scope
faec aedb5a2
Merge branch 'main' of github.com:elastic/beats into awss3-ack-handling
faec ed4953a
remove leftover print
faec 12a39b4
Merge branch 'main' of github.com:elastic/beats into awss3-ack-handling
faec 7e70ea6
cleanup
faec ff4f57c
Merge branch 'main' of github.com:elastic/beats into awss3-ack-handling
faec 2d18b28
restore struct field
faec 6bc5e42
Merge branch 'main' of github.com:elastic/beats into awss3-ack-handling
faec b4cc30e
updates in progress
faec d4ecc1f
leftover file
faec 64a96d4
use a common ack helper in s3 and sqs
faec 7a20250
in progress
faec 977e4c5
Merge branch 'main' of github.com:elastic/beats into awss3-ack-handling
faec 5282671
test updates
faec 1548ac2
fix/update error SQS error handling logic
faec 2358241
close ack handler in s3 worker
faec 8c447bc
Pass through all processing parameters
faec 2deefa7
Merge branch 'main' of github.com:elastic/beats into awss3-ack-handling
faec 8cde326
regenerate mock interfaces
faec c2131e1
update mock calls
faec 1a07561
clean up dead code
faec c99b94d
Merge branch 'main' of github.com:elastic/beats into awss3-ack-handling
faec da1c5ed
Merge branch 'main' into awss3-ack-handling
pierrehilbert 70821fa
Merge branch 'main' into awss3-ack-handling
pierrehilbert 5977072
finish config/doc updates, address review comments
faec 6fe1183
remove helper that was replaced with shared lib
faec ba6ecfe
make check
faec f1732f8
re-correct doc yml
faec 83ba548
make check again
faec 577759c
update changelog
faec 7ba1374
Merge branch 'awss3-ack-handling' of github.com:faec/beats into awss3…
faec cf90022
Merge branch 'main' of github.com:elastic/beats into awss3-ack-handling
faec a743c08
make update
faec 48025b5
linter / CI fixes
faec 10ecfd3
Fix benchmark test ack counting
faec 7a38f55
edit changelog
faec 84cb422
more mock test fixes
faec b088ca7
fix more mock details
faec bbbd1f9
fix race condition in mock sequencing
faec File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -23112,6 +23112,38 @@ Contents of probable licence file $GOMODCACHE/github.com/xdg-go/[email protected]/LIC | |
| of your accepting any such warranty or additional liability. | ||
|
|
||
|
|
||
| -------------------------------------------------------------------------------- | ||
| Dependency : github.com/zyedidia/generic | ||
| Version: v1.2.1 | ||
| Licence type (autodetected): MIT | ||
| -------------------------------------------------------------------------------- | ||
|
|
||
| Contents of probable licence file $GOMODCACHE/github.com/zyedidia/[email protected]/LICENSE: | ||
|
|
||
| MIT License | ||
|
|
||
| Copyright (c) 2021: Zachary Yedidia. | ||
|
|
||
| Permission is hereby granted, free of charge, to any person obtaining | ||
| a copy of this software and associated documentation files (the | ||
| "Software"), to deal in the Software without restriction, including | ||
| without limitation the rights to use, copy, modify, merge, publish, | ||
| distribute, sublicense, and/or sell copies of the Software, and to | ||
| permit persons to whom the Software is furnished to do so, subject to | ||
| the following conditions: | ||
|
|
||
| The above copyright notice and this permission notice shall be | ||
| included in all copies or substantial portions of the Software. | ||
|
|
||
| THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, | ||
| EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF | ||
| MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. | ||
| IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY | ||
| CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, | ||
| TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE | ||
| SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. | ||
|
|
||
|
|
||
| -------------------------------------------------------------------------------- | ||
| Dependency : go.elastic.co/apm/module/apmelasticsearch/v2 | ||
| Version: v2.6.0 | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,106 @@ | ||
| // Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| // or more contributor license agreements. Licensed under the Elastic License; | ||
| // you may not use this file except in compliance with the Elastic License. | ||
|
|
||
| package awss3 | ||
|
|
||
| import ( | ||
| "github.com/zyedidia/generic/queue" | ||
|
|
||
| "github.com/elastic/beats/v7/libbeat/beat" | ||
| "github.com/elastic/beats/v7/libbeat/common/acker" | ||
| ) | ||
|
|
||
| type awsACKHandler struct { | ||
| pending *queue.Queue[pendingACK] | ||
| ackedCount int | ||
|
|
||
| pendingChan chan pendingACK | ||
| ackChan chan int | ||
| } | ||
|
|
||
| type pendingACK struct { | ||
| eventCount int | ||
| ackCallback func() | ||
| } | ||
|
|
||
| func newAWSACKHandler() *awsACKHandler { | ||
| handler := &awsACKHandler{ | ||
| pending: queue.New[pendingACK](), | ||
|
|
||
| // Channel buffer sizes are somewhat arbitrary: synchronous channels | ||
| // would be safe, but buffers slightly reduce scheduler overhead since | ||
| // the ack loop goroutine doesn't need to wake up as often. | ||
| // | ||
| // pendingChan receives one message each time an S3/SQS worker goroutine | ||
| // finishes processing an object. If it is full, workers will not be able | ||
| // to advance to the next object until the ack loop wakes up. | ||
| // | ||
| // ackChan receives approximately one message every time an acknowledged | ||
| // batch of events contains at least one event from this input. (Sometimes | ||
| // fewer if messages can be coalesced.) If it is full, acknowledgement | ||
| // notifications for inputs/queue will stall until the ack loop wakes up. | ||
| // (This is a much worse consequence than pendingChan, but ackChan also | ||
| // receives fewer messages than pendingChan by a factor of ~thousands, | ||
| // so in practice it's still low-impact.) | ||
| pendingChan: make(chan pendingACK, 10), | ||
| ackChan: make(chan int, 10), | ||
| } | ||
| go handler.run() | ||
| return handler | ||
| } | ||
|
|
||
| func (ah *awsACKHandler) Add(eventCount int, ackCallback func()) { | ||
| ah.pendingChan <- pendingACK{ | ||
| eventCount: eventCount, | ||
| ackCallback: ackCallback, | ||
| } | ||
| } | ||
|
|
||
| // Called when a worker is closing, to indicate to the ack handler that it | ||
| // should shut down as soon as the current pending list is acknowledged. | ||
| func (ah *awsACKHandler) Close() { | ||
| close(ah.pendingChan) | ||
| } | ||
|
|
||
| func (ah *awsACKHandler) pipelineEventListener() beat.EventListener { | ||
| return acker.TrackingCounter(func(_ int, total int) { | ||
| // Notify the ack handler goroutine | ||
| ah.ackChan <- total | ||
| }) | ||
| } | ||
|
|
||
| // Listener that handles both incoming metadata and ACK | ||
| // confirmations. | ||
| func (ah *awsACKHandler) run() { | ||
| for { | ||
| select { | ||
| case result, ok := <-ah.pendingChan: | ||
| if ok { | ||
| ah.pending.Enqueue(result) | ||
| } else { | ||
| // Channel is closed, reset so we don't receive any more values | ||
| ah.pendingChan = nil | ||
| } | ||
| case count := <-ah.ackChan: | ||
| ah.ackedCount += count | ||
| } | ||
|
|
||
| // Finalize any objects that are now completed | ||
| for !ah.pending.Empty() && ah.ackedCount >= ah.pending.Peek().eventCount { | ||
| result := ah.pending.Dequeue() | ||
| ah.ackedCount -= result.eventCount | ||
| // Run finalization asynchronously so we don't block the SQS worker | ||
| // or the queue by ignoring the ack handler's input channels. Ordering | ||
| // is no longer important at this point. | ||
| if result.ackCallback != nil { | ||
| go result.ackCallback() | ||
| } | ||
| } | ||
|
|
||
| // If the input is closed and all acks are completed, we're done | ||
| if ah.pending.Empty() && ah.pendingChan == nil { | ||
| return | ||
| } | ||
| } | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
At a minimum I think we need a comment on why
10. But I've got a couple of questions. Do these always need to be in sync? Would we ever want to change these?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, documenting why we picked up 10, and if they need to stay in sync would be useful. Do we ever need to make this configurable?