Avoid extra latency potentially happening between Poll calls
#668
Conversation
guizmaii
force-pushed
the
reduce_latency
branch
from
a00694e
to
ca45be5
Compare
Yes, the same thought had crossed my mind this weekend. |
| @@ -25,4 +26,6 @@ object DiagnosticEvent { | |||
| final case class Assigned(partitions: Set[TopicPartition]) extends Rebalance | |||
| final case class Lost(partitions: Set[TopicPartition]) extends Rebalance | |||
| } | |||
|
|
|||
| final case class RunloopEvent(command: Command) extends DiagnosticEvent | |||
There was a problem hiding this comment.
Not a huge fan of this, it means exposing one of the Runloop's internals. Shall we at least document that it's interface is unstable, so we can change the Commands a bit more freely if we need?
There was a problem hiding this comment.
Agreed. TBH, I'm not clear what this Diagnostics thing is for
I assumed it already worked this way, but now I see it doesn't :) |
|
Not yet, I'm experimenting a bit with some changed polling behavior though. |
Related to work done here:
I didn't manage to find a way to test that this "extra latency" is not stacked anymore 🤔
@svroonland I wonder if there's not a risk of generating/stacking a lot of
Pollin the Stream 🤔I wonder if using a
Queue.dropping[Command.Poll](1)wouldn't be a good idea.At any time, I think that we only need to have 1 or 0 Poll command: if we have one, at the next iteration of the loop, it'll consume it from the queue and will execute the corresponding action. If 0, it'll do nothing.
If the queue was containing 1 Poll command and we're emitting a new one, we're replacing the old one with the new one, so we don't waste memory and we have no risks of stacking too many of them (which could potentially lead to an OOM)
WDYT? 🤔