Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enrich-pubsub: add reference.conf and provide minimal config example #505

Closed
Tracked by #481
benjben opened this issue Sep 24, 2021 · 0 comments
Closed
Tracked by #481
Milestone

Comments

@benjben
Copy link
Contributor

benjben commented Sep 24, 2021

Take the opportunity to make these configurable :

/**
   * Set the delay threshold to use for batching. After this amount of time has elapsed (counting
   * from the first element added), the elements will be wrapped up in a batch and sent.
   *
   * If the source MaxQueueSize is small, then the queue could block the app by waiting between
   * batches if DelayThreshold is too large.  If the source MaxQueueSize is sufficiently large,
   * then the DelayThreshold should not ever cause blocking.
   */
  val DefaultDelayThreshold: FiniteDuration = 200.milliseconds

  /**
   * A batch of messages will be emitted to PubSub when the batch reaches 1000 messages.
   * We use 1000 because it is the maximum batch size allowed by PubSub.
   * This overrides the permutive library default of `5`
   */
  val DefaultPubsubMaxBatchSize = 1000L

  /**
   * A batch of messages will be emitted to PubSub when the batch reaches 10 MB.
   * We use 10MB because it is the maximum batch size allowed by PubSub.
   */
  val DefaultPubsubMaxBatchBytes = 10000000L

And to make more parameters of enrich-kinesis configurable (e.g. maxConcurrency and bufferSize)

@benjben benjben added this to the 3.0.0 milestone Sep 24, 2021
@benjben benjben mentioned this issue Oct 12, 2021
15 tasks
@benjben benjben changed the title PubSub: add reference.conf and provide minimal config example enrich-pubsub: add reference.conf and provide minimal config example Jan 25, 2022
@benjben benjben closed this as completed in 6707103 Mar 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant