Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(kafka source): fix acknowledgement handling during shutdown and rebalance events #17497

Merged
merged 40 commits into from
Oct 12, 2023
Merged

fix(kafka source): fix acknowledgement handling during shutdown and rebalance events #17497

Changes from 35 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
488aa63
test(kafka source): integration tests for acknowledgement handling du…
jches Apr 27, 2023
2b81df3
fix(kafka source): drain pending acknowledgements on shutdown and reb…
jches May 16, 2023
03aedbd
fix(kafka source): performance improvements for acknowledgement handl…
jches May 25, 2023
0191497
clippy fixes, and remove unnecessary last_offset tracking
jches May 25, 2023
bf4a5c3
cargo fmt again
jches May 26, 2023
9d32a82
fmt
jches May 26, 2023
3582940
clean up handle_messages loop and add a tracing span for metrics coll…
jches Jun 29, 2023
1707266
Merge branch 'master' into kafka-shutdown-rebalance-pending-acks
jches Jun 29, 2023
c21e174
fixup changes lost after merging master
jches Jun 29, 2023
28b2b44
clippy warning
jches Jul 3, 2023
ab243cb
enhancement(kafka source): kafka source uses a dedicated task per par…
jches Aug 28, 2023
fe4ae3e
Merge branch 'master' into kafka-shutdown-rebalance-pending-acks
jches Aug 28, 2023
485524f
make the spelling checker happy, maybe?
jches Aug 28, 2023
dc723a7
emit a debug log instead of panicking if a shutdown happens during a …
jches Aug 29, 2023
1f49643
improved partition eof handling
jches Aug 29, 2023
a69b2c4
add OptionFuture to drain deadline and EOF handling, and use is_subse…
jches Sep 6, 2023
0b6d17e
replace OnceCell with OnceLock
jches Sep 7, 2023
34f2f85
Merge branch 'master' into kafka-shutdown-rebalance-pending-acks
jches Sep 7, 2023
6865470
cargo fmt
jches Sep 7, 2023
84dfe92
create clear distinction between consuming and draining states
jches Sep 7, 2023
d823820
add "complete" as a terminal state, and "keep_consuming", "keep_drain…
jches Sep 8, 2023
c6082f3
use state transition methods consistently for all state transitions
jches Sep 8, 2023
444b3e1
slightly clearer assertion messages about what is expected
jches Sep 8, 2023
88cd2e5
update obsolete comment, make coordinator loop condition explicit
jches Sep 8, 2023
db999e2
use keep_consuming from the drain_timeout while consuming handler
jches Sep 8, 2023
1edb270
rely solely on adding/removing entries in expect_drain to detect when…
jches Sep 11, 2023
d2cdeb1
fix comment :P
jches Sep 12, 2023
e4e57b9
clippy/fmt fixes
jches Sep 12, 2023
7999fa7
minor cleanup: during shutdown, use is_drain_complete to detect the a…
jches Sep 13, 2023
78c7a0e
integration test uses `FuturesUnordered` for better performance
jches Sep 20, 2023
c88e688
use FuturesUnordered
jches Sep 20, 2023
b3cef9b
use 6 partitions for integration test
jches Sep 20, 2023
61784a3
integration test using 125k messages
jches Sep 21, 2023
89dbaa9
add drain_timeout_ms option for kafka source
jches Sep 22, 2023
bb4b544
enforce drain_timeout_ms < session_timeout_ms when building kafka source
jches Sep 22, 2023
afb7fa1
generate component docs
jches Sep 22, 2023
e455ce0
use Option::{unzip, map_or} methods
jches Oct 3, 2023
4ad434b
remove OnceLock on callback channel sender, and other review cleanups
jches Oct 4, 2023
6353bbc
members of Keys struct are cloned once per consumed partition, instea…
jches Oct 4, 2023
ff086d7
cargo fmt and fix clippy warnings
jches Oct 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading