rewrite gcp_pubsub output #1836
Conversation
| "github.com/benthosdev/benthos/v4/internal/message" | ||
| "github.com/benthosdev/benthos/v4/internal/metadata" | ||
| "github.com/benthosdev/benthos/v4/public/service" | ||
| "github.com/sourcegraph/conc/pool" |
There was a problem hiding this comment.
This library has a useful result pool, kind of like of Promise.allSettled. If it's a no go then I'd be happy to put something together with errgroup or similar.
https://pkg.go.dev/github.com/sourcegraph/conc/pool#ResultPool
| service.NewMetadataFilterField("metadata"). | ||
| Optional(). | ||
| Description("Specify criteria for which metadata values are sent as attributes."), |
There was a problem hiding this comment.
This is a breaking change because the old api added metadata.exclude_prefixes whereas the new api added metadata.include_prefixes and metadata.include_patterns. Should I manually implement the old config somehow?
There was a problem hiding this comment.
I might expand the public field definition so that you can toggle exclusive versus inclusive defaults. Ideally all components will eventually converge but we need to do it in stages (with backwards compatibility).
disintegrator
force-pushed
the
gcp-pubsub-next
branch
4 times, most recently
from
84d60aa to
854e5f6
Compare
disintegrator
force-pushed
the
gcp-pubsub-next
branch
9 times, most recently
from
34604a4 to
9e35295
Compare
disintegrator
force-pushed
the
gcp-pubsub-next
branch
from
9e35295 to
81c4914
Compare
This change rewrites the gcp_pubsub output to use the new config API and to lean on the pubsub client's internal batching mechanism. This is referred to as the "internal buffer" in docs to distinguish it from Benthos' batching mechanism. The rewrite also benefits from added tests and a pubsub mock. Signed-off-by: Georges Haidar <[email protected]>
disintegrator
force-pushed
the
gcp-pubsub-next
branch
from
81c4914 to
da84c26
Compare
There was a problem hiding this comment.
LGTM, thanks @disintegrator, going to switch to the new NewMetadataExcludeFilter to fill the gap in backwards compat.
This change rewrites the gcp_pubsub output to use the new config API and to lean on the pubsub client's internal batching mechanism. This is referred to as the "internal buffer" in docs to distinguish it from Benthos' batching mechanism. The rewrite also benefits from added tests and a pubsub mock.
Stress test results on an M1 Pro (8-core, 16GB mem):
(
gc_gcp_pubsubis the working name of the output in this PR that doesn't clash with the existig output)