Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Kinesis sink #84

Merged
merged 33 commits into from
Jul 6, 2024
Merged

feat: Kinesis sink #84

merged 33 commits into from
Jul 6, 2024

Conversation

turtleDev
Copy link
Contributor

@turtleDev turtleDev commented Jun 26, 2024

This PR adds support for AWS Kinesis sink(#78).

Configuration

This change introduces the following new configurations

  • a new PUBLISHER_TYPE value called kinesis
  • PUBLISHER_KINESIS_AWS_REGION region with the target kinesis stream resides (fallbacks back to AWS_REGION)
  • PUBLISHER_KINESIS_CREDENTIALS path to AWS Credentials file (defaults to `$HOME/.aws/credentials)
  • PUBLISHER_KINESIS_STREAM_PROBE_INTERVAL_MS time interval for stream readiness check.
  • PUBLISHER_KINESIS_STREAM_AUTOCREATE whether Raccoon should create streams that don't exist.
  • PUBLISHER_KINESIS_STREAM_MODE when creating streams, what mode to set. (ON_DEMAND or PROVISIONED)
  • PUBLISHER_KINESIS_STREAM_SHARDS when creating streams, how many shards to configure.
  • PUBLISHER_KINESIS_PUBLISH_TIMEOUT_MS how long to wait for before aborting a publish operation (per batch)

Metrics

This change also introduces the following new metrics

  • kinesis_messages_delivered_total total number of events processed (includes false negatives)
  • kinesis_messages_undelivered_total total number of events that failed
  • kinesis_unknown_stream_failure_total events that failed to publish to an unknown stream
  • kinesis_producebulk_tt_ms batch production time.

turtleDev and others added 30 commits June 17, 2024 17:32
pubsub publisher was reporting `pubsub_unknown_topic_failure_total`
for errors that were not strictly non-existent topic errors.
@ravisuhag ravisuhag changed the title Feat: Kinesis sink feat: Kinesis sink Jul 6, 2024
@ravisuhag ravisuhag merged commit cfa245c into main Jul 6, 2024
10 checks passed
@ravisuhag ravisuhag deleted the kinesis-sink branch July 6, 2024 14:27
@ravisuhag ravisuhag linked an issue Jul 6, 2024 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add AWS Kinesis sink
2 participants