Emit larger chunks in HTTP and gRPC consumers #330
Conversation
This should improve performance as it avoids calling looping through `takeNextElements` as long as elements are still available.
Avoids wrapping potential `null` into an `Option`. Also reduces number of `delay` blocks as there were no external side effects.
This applies to both the gRPC and HTTP consumers. It could lead to performance improvements in applications which are sensitive to the chunksize. There is a subtle user-facing change as a result: if any message in a chunk fails to decode the `errorHandler` is invoked _before_ any successfully decoded messages in the chunk are emitted downstream; even if the message which failed to decode comes after the successful ones. This change is probably not a problem though...
|
@istreeter, @CremboC has got the PubSub emulator working with gRPC so we may be able to commit benchmarks into the repo... |
Codecov Report
@@ Coverage Diff @@
## master #330 +/- ##
===========================================
+ Coverage 35.00% 47.49% +12.49%
===========================================
Files 29 29
Lines 300 299 -1
Branches 8 6 -2
===========================================
+ Hits 105 142 +37
+ Misses 195 157 -38
Continue to review full report at Codecov.
|
|
Hi @bastewart - I was just wondering when this change might get merged and released? No pressure, and not urgent - it's just so I know when to expect it. Many thanks! |
|
Hey @istreeter , I'm trying to get the GRPC tests to run. Hopefully we can get them to work this week and then merge and release. |
|
@istreeter do you need a CE2 version of this PR as well? |
|
Hey @CremboC a CE2 version would be extremely helpful. I hope I'm not asking for too much 😇. If it is beyond scope for you to release a CE2 version then I will release a version from my forked repo. |
CE3 port/tweak of #329.
There are two subtle user-affecting changes as a result of this:
decode the
errorHandleris invoked before any successfullydecoded messages in the chunk are emitted downstream; even
if the message which failed to decode comes after the successful
ones.
failure occurs in the Java library. This should not matter at all.