Add optional manual commit to seldon kafka server #4117
Conversation
Signed-off-by: Emirhan Karagül <[email protected]>
|
Hi @YmirKhang. Thanks for your PR. I'm waiting for a SeldonIO or todo member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository. |
| @@ -101,6 +104,11 @@ func NewKafkaServer(fullGraph bool, workers int, deploymentName, namespace, prot | |||
|
|
|||
| } | |||
| } | |||
|
|
|||
| if !autoCommit && workers > 1 { | |||
| log.Info("Disabling auto commit for kafka can have undesired side effects with multiple workers") | |||
Signed-off-by: Emirhan Karagül <[email protected]>
|
/assign @SachinVarghese |
|
/assign @axsaucedo |
|
/test notebooks |
|
Nice one @YmirKhang thank you for the contribution /approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: axsaucedo, cliveseldon The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
You're welcome! |
Signed-off-by: Emirhan Karagül [email protected]
What this PR does / why we need it: This PR provides an optional flag to control auto commit behavior of kafka consumer. When no environment variable is set in the
svcOrchSpec, the executor works as usual. If auto commit is disabled, the messages will be committed only after they have been processed and response has been sent to the output queue.Which issue(s) this PR fixes:
Fixes #4097
Special notes for your reviewer: As mentioned before, disabled auto commit might cause data duplication when
kafka_workers > 1.