Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pull all available messages in the backend topic (state topic) before creating the topology plan #553

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

yoniv87
Copy link

@yoniv87 yoniv87 commented Nov 10, 2022

… continuing to build the topology plan

  • Please check if the PR fulfills these requirements
  • The commit messages are descriptive
  • Tests for the changes have been added (for bug fixes/features)
  • Docs have been added/updated (for bug fixes/features)
  • An issue has been created for the pull requests. Some issues might require the previous discussion.
  • What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

Bug fix #552

  • What is the current behavior? (You can also link to an open issue here)

When using Kafka Backend, after the message puller can retrieve some messages ( records.count() > 0 ), it goes to callback.initialLoadFinish(), even in case there are more available messages in the topic.

  • What is the new behavior (if this is a feature change)?

In case there are messages (records.count() > 0), the application will reset the counter (times = 0) and will try to run the iteration again to fetch more messages.
Only in the case that no messages are available to fetch ( records.count() == 0 ), the application will do several retries (times >= config.getKafkaBackendConsumerRetries()) and then continue to callback.initialLoadFinish();

  • Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)
    No breaking changes. However, it will take a little more time (depending on KafkaBackendConsumerRetries config) to ensure that the application retrieves all available messages from the topic in case of using Kafka Backend.

  • Other information:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

When using KafkaBackend, the application ignores messages with an offset higher than 75/100
1 participant