-
Notifications
You must be signed in to change notification settings - Fork 138
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Commit timeout #1106
Comments
@ouertani What are your consumer settings? |
|
@erikvanoosten this behavior is reproducible with versions |
Sorry, I was referring to the zio-kafka ConsumerSettings. In particular, what is the pollInterval? Ah, in 2.3 also. Let's see what could have caused that... |
@ouertani I have no idea. Can you create a minimal reproducer please? |
Here is more details:
|
@erikvanoosten The consumer is on eventhub what is important that messages have a big sizes with 10+ parallel consumers/pods. I get this from RN:
|
How large are the messages roughly? |
message size less than 1Mb roughly around 600kb. I will try to update timeout while this is wired as the same behavior with 2.3.0 |
@erikvanoosten indeed another java.lang.OutOfMemoryError: Java heap space is raised instead |
With such large messages it doesn't make sense to prefetch. You'll go OOM quickly. Alternatively, you can write your own FetchStrategy. |
What was wired for me that it was working for month withs 2.2 version without issues. |
I am happy that disabling prefetch works! After 2.2 we made performance optimization by pre-fetching quicker and more. Unfortunately this doesn't work well for some less common use cases such as large messages. |
Thank you @erikvanoosten |
Start getting
zio.kafka.consumer.Consumer$CommitTimeout$: Commit timeout
after upgrading zio kafka from2.2
to2.6.0
The text was updated successfully, but these errors were encountered: