Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The previous implementation of the PubSub Source was a wrapper around
Subscriber
provided by the 3rd-party pubsub sdk. ThatSubscriber
is a wrapper around a lower-level GRPC stub. It used the "Streaming Pull" GRPC method to fetch messages from PubSub.This commit abandons using the
Subscriber
and instead wraps the GRPC stub directly. It uses the "Unary Pull" GRPC method to fetch messages from PubSub.We found that "Unary Pull" alleviates a problem in which PubSub occasionally re-delivers the same messages, causing downstream duplicates. The problem happened especially in apps like Lake Loader, which builds up a very large number of un-acked messages and then acks them all in one go at the end of a timed window.
Compared with the previous Source implementation it has these differences in behaviour:
Subscriber
managed ack extensions (a.k.a. modifying ack deadlines). Ack extension periods were adjusted dynamically according to runtime heuristics of message processing times. Whereas in this new Source, the ack extension period is a fixed configurable period.Subscriber
periodically mod-acked all unacked messages currently held in memory. Whereas the new Source only mod-acks messages when they are approaching their ack deadline. This is an improvement for apps like Lake Loader which might have a very large number of outstanding unacked messages.