Add the ability to use Kafka transactional producers #223
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original is not maintained anymore
…ransactional id on producer config
… topic to publish the message from the message itself and validate topic names
mateusjunges
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #217
Transactional producers
Transactions provide the guarantee of atomicity, consistency, isolation, and durability (ACID) for a group of messages that are published to Kafka. This means that either all messages within the transaction will be successfully written to Kafka or none of the messages will be written. This is a crucial feature when it comes to ensuring data consistency and avoiding any data loss.
When using Kafka with transactional producers, each producer MUST configure its own static and unique
transactional.id. This configuration option is used to uniquely identify the same logical producer across process restarts.To guarantee message ordering, a given producer can have at most one ongoing transaction, and they are executed serially.
Consuming messages from transactional producers
When consuming messages from transactional producers it is important to set the
isolation.levelproperty of your consumer toread_committed, which will prevent consumers from advancing to offsets which includes open transactions. Messages from aborted transactions are filtered out within the consumer, and are never observed by the client application.