Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Pubsub sink #79

Merged
merged 52 commits into from
Jun 15, 2024
Merged

feat: Pubsub sink #79

merged 52 commits into from
Jun 15, 2024

Commits on Jun 8, 2024

  1. publisher: refactor to a generic sink interface

    @turtleDev
    turtleDev committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    8a3827f View commit details
    Browse the repository at this point in the history

  2. fix: worker tests deadlock

    @turtleDev
    turtleDev committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    6232247 View commit details
    Browse the repository at this point in the history

  3. publisher: kafka: move metric reporter to Constructor

    @turtleDev
    turtleDev committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    418f6e4 View commit details
    Browse the repository at this point in the history

Commits on Jun 9, 2024

  1. publisher: add stub for pubsub

    @turtleDev
    turtleDev committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    8ecba01 View commit details
    Browse the repository at this point in the history

  2. publisher: pubsub: add event publishing

    @turtleDev
    turtleDev committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    7f32b3d View commit details
    Browse the repository at this point in the history

  3. bump go version to 1.21

    @turtleDev
    turtleDev committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    1c3d457 View commit details
    Browse the repository at this point in the history

  4. ci: bump go version to 1.21

    @turtleDev
    turtleDev committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    8fb73d8 View commit details
    Browse the repository at this point in the history

Commits on Jun 10, 2024

  1. worker: handle non BulkError gracefully

    @turtleDev
    turtleDev committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    51cf8fa View commit details
    Browse the repository at this point in the history

  2. publisher: pubsub: add topic caching

    @turtleDev
    turtleDev committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    06d1705 View commit details
    Browse the repository at this point in the history

  3. bump go version in docker build to 1.21

    @turtleDev
    turtleDev committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    77e9ae6 View commit details
    Browse the repository at this point in the history

  4. init: integrate pubsub publisher 

    This change introduces a new config flag PUBLISHER_TYPE

This is used to initialise & configure amongst available publishers.
(currently kafka & pubsub). Defaults to "kafka"
    @turtleDev
    turtleDev committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    01d40a5 View commit details
    Browse the repository at this point in the history

Commits on Jun 11, 2024

  1. publisher: pubsub: add metrics for events

    @turtleDev
    turtleDev committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    b1f454b View commit details
    Browse the repository at this point in the history

  2. publisher: pubsub: use metrics comparable to kafka publisher 

    This commit changes the metrics exported by pubsub publisher
to be semantically compatiable with metrics exported by kafka
publisher.
    @turtleDev
    turtleDev committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    11d34aa View commit details
    Browse the repository at this point in the history

  3. publisher: pubsub: make topic auto creation configurable

    @turtleDev
    turtleDev committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    cc1ce2c View commit details
    Browse the repository at this point in the history

  4. publisher: pubsub: add support for configuration topic retention

    @turtleDev
    turtleDev committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    8fff421 View commit details
    Browse the repository at this point in the history

  5. config: fix duration parsing for pubsub retention period

    @turtleDev
    turtleDev committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    a082056 View commit details
    Browse the repository at this point in the history

  6. app: report configured publisher name in shutdown messages

    @turtleDev
    turtleDev committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    f40c910 View commit details
    Browse the repository at this point in the history

  7. publisher: pubsub: add configuration for publish delay threshold

    @turtleDev
    turtleDev committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    5e78892 View commit details
    Browse the repository at this point in the history

  8. publisher: pubsub: add config for publish count threshold

    @turtleDev
    turtleDev committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    27f415b View commit details
    Browse the repository at this point in the history

  9. publisher: pubsub: add config for publish byte threshold

    @turtleDev
    turtleDev committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    2862f58 View commit details
    Browse the repository at this point in the history

  10. publisher: pubsub: add configuration for publish timeout

    @turtleDev
    turtleDev committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    afbb1b0 View commit details
    Browse the repository at this point in the history

  11. publisher: pubsub: wire in publish settings

    @turtleDev
    turtleDev committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    58a39ab View commit details
    Browse the repository at this point in the history

  12. server: fix erronous log statement

    @turtleDev
    turtleDev committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    d0327d7 View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2024

  1. misc: kafka.Close() only returns error if unflushed events > 0

    @turtleDev
    turtleDev committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    2bda8f2 View commit details
    Browse the repository at this point in the history

  2. metrics: whitelist pubsub metrics

    @turtleDev
    turtleDev committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    50bea23 View commit details
    Browse the repository at this point in the history

  3. docs: ref: add pubsub metrics

    @turtleDev
    turtleDev committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    6758e7e View commit details
    Browse the repository at this point in the history

  4. docs: add pubsub configs

    @turtleDev
    turtleDev committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    3da9c13 View commit details
    Browse the repository at this point in the history

  5. metrics: update tests

    @turtleDev
    turtleDev committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    3e5aad1 View commit details
    Browse the repository at this point in the history

  6. publisher: pubsub: add configuration for service account creds

    @turtleDev
    turtleDev committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    26e1b6b View commit details
    Browse the repository at this point in the history

  7. ci: refactor integration test setup to a common workflow

    @turtleDev
    turtleDev committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    7e7cc08 View commit details
    Browse the repository at this point in the history

  8. ci: fix integration environment setup

    @turtleDev
    turtleDev committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    604830f View commit details
    Browse the repository at this point in the history

  9. ci: fix missing run-on field

    @turtleDev
    turtleDev committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    4b4cf73 View commit details
    Browse the repository at this point in the history

  10. ci: debug action run

    @turtleDev
    turtleDev committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    61bb23a View commit details
    Browse the repository at this point in the history

  11. ci: revert integration test workflow

    @turtleDev
    turtleDev committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    97d63cb View commit details
    Browse the repository at this point in the history

  12. ci: convert integration setup to action

    @turtleDev
    turtleDev committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    6d3f6d4 View commit details
    Browse the repository at this point in the history

Commits on Jun 13, 2024

  1. ci: remove checkout from integration test setup action

    @turtleDev
    turtleDev committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    5235845 View commit details
    Browse the repository at this point in the history

  2. publisher: pubsub: add initial tests

    @turtleDev
    turtleDev committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    8bb1a75 View commit details
    Browse the repository at this point in the history

  3. publisher: pubsub: add tests for topics

    @turtleDev
    turtleDev committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    26d1b22 View commit details
    Browse the repository at this point in the history

  4. publisher: kafka: bugfix static topics 

    There was a bug in kafka publisher, where when a static topic
was specified as the topic format, it would produce the result
 "static-format%!(EXTRA string=event-type)"

This is caused by fmt.Sprintf's behaviour of adding any unspecified
operands to the output.
    @turtleDev
    turtleDev committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    35dcaea View commit details
    Browse the repository at this point in the history

  5. chore: remove erronous use of project id in pubsub

    @turtleDev
    turtleDev committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    6ad3f50 View commit details
    Browse the repository at this point in the history

  6. misc: shorten pubsub publisher opts

    @turtleDev
    turtleDev committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    8f998b9 View commit details
    Browse the repository at this point in the history

  7. ci: add integration test for pubsub publisher

    @turtleDev
    turtleDev committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    9e43d29 View commit details
    Browse the repository at this point in the history

  8. metrics: add histogram for pubsub_producebulk_tt_ms

    @turtleDev
    turtleDev committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    9d96630 View commit details
    Browse the repository at this point in the history

  9. metrics: fix histogram inventory test

    @turtleDev
    turtleDev committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    4f927c3 View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2024

  1. chore: cleanup variable names

    @turtleDev
    turtleDev committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    9a9b79c View commit details
    Browse the repository at this point in the history

  2. refactor: move pubsub publisher to it's own dedicated package

    @turtleDev
    turtleDev committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    b1ac705 View commit details
    Browse the repository at this point in the history

  3. build: use go version 1.22.4

    @turtleDev
    turtleDev committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    de7d91a View commit details
    Browse the repository at this point in the history

  4. publisher: move kafka to dedicated package

    @turtleDev
    turtleDev committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    1993004 View commit details
    Browse the repository at this point in the history

  5. chore: fix erronous use of BulkError

    @turtleDev
    turtleDev committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    67270f9 View commit details
    Browse the repository at this point in the history

  6. app: update reference to kafka publisher

    @turtleDev
    turtleDev committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    d23c073 View commit details
    Browse the repository at this point in the history

  7. ci: build: use go version 1.22.4 (latest stable)

    @turtleDev
    turtleDev committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    a1d2cc0 View commit details
    Browse the repository at this point in the history

  8. misc: use new for range <N> loops

    @turtleDev
    turtleDev committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    041866d View commit details
    Browse the repository at this point in the history