changefeedccl: add pubsub attributes #115426
Assignees
Labels
A-cdc
Change Data Capture
C-enhancement
Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
T-cdc
Comments
jayshrivastava
added
C-enhancement
![blathers[crl]](https://avatars.githubusercontent.com/in/59700?s=40&u=373d9101a12edc3847d2fb6ae78e3adcfb4ca41e&v=4){kind=avatar}
|
cc @cockroachdb/cdc |
jayshrivastava
added a commit
to jayshrivastava/cockroach
that referenced
this issue
Dec 11, 2023
This change adds support for including the table name along with each row/batch sent by the v2 pubsub sink (enabled by default). The table name is passed inside pubsub attributes. Attributes are stored in a `map[string]string` and passed emitted alongside each with each message/batch. To enable this feature, the environment variable `COCKROACH_ENABLE_TABLE_NAME_PUSBUB_ATTRIBUTE` needs to be set to true. The key for the table name in the attribute map will be `TABLE_NAME`. Because this change needs to be backported, it is as small as possible to minimize risk. This feature can be expanded upon later to be more generic (ex. use changefeed options instead of env var, use cdc queries to specify custom attributes, use a generic metadata struct instead of tablename string, pass metadata to different sinks and not just pubsub etc). Release note: None Closes: cockroachdb#115426
jayshrivastava
added a commit
to jayshrivastava/cockroach
that referenced
this issue
Dec 19, 2023
This change adds support for including the table name along with each row/batch sent by the v2 pubsub sink (enabled by default). The table name is passed inside pubsub attributes. Attributes are stored in a `map[string]string` and passed emitted alongside each with each message/batch. To enable this feature, the environment variable `COCKROACH_ENABLE_TABLE_NAME_PUSBUB_ATTRIBUTE` needs to be set to true. The key for the table name in the attribute map will be `TABLE_NAME`. Because this change needs to be backported, it is as small as possible to minimize risk. This feature can be expanded upon later to be more generic (ex. use changefeed options instead of env var, use cdc queries to specify custom attributes, use a generic metadata struct instead of tablename string, pass metadata to different sinks and not just pubsub etc). Release note: None Closes: cockroachdb#115426
jayshrivastava
added a commit
to jayshrivastava/cockroach
that referenced
this issue
Dec 19, 2023
This change adds support for including the table name along with each row/batch sent by the v2 pubsub sink (enabled by default). The table name is passed inside pubsub attributes. Attributes are stored in a `map[string]string` and passed emitted alongside each with each message/batch. To enable this feature, the environment variable `COCKROACH_ENABLE_TABLE_NAME_PUSBUB_ATTRIBUTE` needs to be set to true. The key for the table name in the attribute map will be `TABLE_NAME`. Because this change needs to be backported, it is as small as possible to minimize risk. This feature can be expanded upon later to be more generic (ex. use changefeed options instead of env var, use cdc queries to specify custom attributes, use a generic metadata struct instead of tablename string, pass metadata to different sinks and not just pubsub etc). Release note: None Closes: cockroachdb#115426
jayshrivastava
added a commit
to jayshrivastava/cockroach
that referenced
this issue
Dec 19, 2023
This change adds support for including the table name along with each row/batch sent by the v2 pubsub sink (enabled by default). The table name is passed inside pubsub attributes. Attributes are stored in a `map[string]string` and passed emitted alongside each with each message/batch. To enable this feature, the environment variable `COCKROACH_ENABLE_TABLE_NAME_PUSBUB_ATTRIBUTE` needs to be set to true. The key for the table name in the attribute map will be `TABLE_NAME`. Because this change needs to be backported, it is as small as possible to minimize risk. This feature can be expanded upon later to be more generic (ex. use changefeed options instead of env var, use cdc queries to specify custom attributes, use a generic metadata struct instead of tablename string, pass metadata to different sinks and not just pubsub etc). Release note: None Closes: cockroachdb#115426
jayshrivastava
added a commit
to jayshrivastava/cockroach
that referenced
this issue
Dec 19, 2023
This change adds support for including the table name along with each row/batch sent by the v2 pubsub sink (enabled by default). The table name is passed inside pubsub attributes. Attributes are stored in a `map[string]string` and passed emitted alongside each with each message/batch. To enable this feature, the environment variable `COCKROACH_ENABLE_TABLE_NAME_PUSBUB_ATTRIBUTE` needs to be set to true. The key for the table name in the attribute map will be `TABLE_NAME`. Because this change needs to be backported, it is as small as possible to minimize risk. This feature can be expanded upon later to be more generic (ex. use changefeed options instead of env var, use cdc queries to specify custom attributes, use a generic metadata struct instead of tablename string, pass metadata to different sinks and not just pubsub etc). Release note: None Closes: cockroachdb#115426
jayshrivastava
added a commit
to jayshrivastava/cockroach
that referenced
this issue
Dec 19, 2023
This change adds support for including the table name along with each row/batch sent by the v2 pubsub sink (enabled by default). The table name is passed inside pubsub attributes. Attributes are stored in a `map[string]string` and passed emitted alongside each with each message/batch. To enable this feature, the environment variable `COCKROACH_ENABLE_TABLE_NAME_PUSBUB_ATTRIBUTE` needs to be set to true. The key for the table name in the attribute map will be `TABLE_NAME`. Because this change needs to be backported, it is as small as possible to minimize risk. This feature can be expanded upon later to be more generic (ex. use changefeed options instead of env var, use cdc queries to specify custom attributes, use a generic metadata struct instead of tablename string, pass metadata to different sinks and not just pubsub etc). Release note: None Closes: cockroachdb#115426
jayshrivastava
added a commit
to jayshrivastava/cockroach
that referenced
this issue
Dec 20, 2023
This change adds support for including the table name along with each row/batch sent by the v2 pubsub sink (enabled by default). The table name is passed inside pubsub attributes. Attributes are stored in a `map[string]string` and passed emitted alongside each with each message/batch. To enable this feature, the environment variable `COCKROACH_ENABLE_TABLE_NAME_PUSBUB_ATTRIBUTE` needs to be set to true. The key for the table name in the attribute map will be `TABLE_NAME`. Because this change needs to be backported, it is as small as possible to minimize risk. This feature can be expanded upon later to be more generic (ex. use changefeed options instead of env var, use cdc queries to specify custom attributes, use a generic metadata struct instead of tablename string, pass metadata to different sinks and not just pubsub etc). Release note: None Closes: cockroachdb#115426
jayshrivastava
added a commit
to jayshrivastava/cockroach
that referenced
this issue
Dec 21, 2023
This change adds support for including the table name along with each row/batch sent by the v2 pubsub sink (enabled by default). The table name is passed inside pubsub attributes. Attributes are stored in a `map[string]string` and passed emitted alongside each with each message/batch. To enable this feature, the uri parameter `with_table_name_attribute=true` must be added to the sink uri. The key for the table name in the attribute map will be `TABLE_NAME`. Because this change needs to be backported, it is as small as possible to minimize risk. This feature can be expanded upon later to be more generic (ex. use changefeed options instead of env var, use cdc queries to specify custom attributes, use a generic metadata struct instead of tablename string, pass metadata to different sinks and not just pubsub etc). Because this feature will be expanded in the future, the release note is intentionally left blank. Release note: None Closes: cockroachdb#115426
craig bot
pushed a commit
that referenced
this issue
Dec 21, 2023
116089: changefeedccl: support attributes in pubsub sink r=jayshrivastava a=jayshrivastava This change adds support for including the table name along with each row/batch sent by the v2 pubsub sink (enabled by default). The table name is passed inside pubsub attributes. Attributes are stored in a `map[string]string` and passed emitted alongside each with each message/batch. To enable this feature, the uri parameter `with_table_name_attribute=true` must be added to the sink uri. The key for the table name in the attribute map will be `TABLE_NAME`. Because this change needs to be backported, it is as small as possible to minimize risk. This feature can be expanded upon later to be more generic (ex. use changefeed options instead of env var, use cdc queries to specify custom attributes, use a generic metadata struct instead of tablename string, pass metadata to different sinks and not just pubsub etc). Because this feature will be expanded in the future, the release note is intentionally left blank. Release note: None Closes: #115426 116892: storage: add setting to seed up consistency checks in tests r=RaduBerinde a=itsbilal Previously, we'd fall back to the 3 second consistency checker EventuallyFileOnlySnapshot (EFOS) wait in roachtests, which was slowing all of them down when we ran every replica through the consistency checker in post-test assertions. This change speeds up that consistency check in roachtest post-test assertions by flipping a new cluster setting to speed up EFOS waits for consistency checks after a roachtest finishes. Epic: none Unblocks #116330. Release note: None Co-authored-by: Jayant Shrivastava <[email protected]> Co-authored-by: Bilal Akhtar <[email protected]>
jayshrivastava
added a commit
to jayshrivastava/cockroach
that referenced
this issue
Jan 1, 2024
This change adds support for including the table name along with each row/batch sent by the v2 pubsub sink (enabled by default). The table name is passed inside pubsub attributes. Attributes are stored in a `map[string]string` and passed emitted alongside each with each message/batch. To enable this feature, the uri parameter `with_table_name_attribute=true` must be added to the sink uri. The key for the table name in the attribute map will be `TABLE_NAME`. Because this change needs to be backported, it is as small as possible to minimize risk. This feature can be expanded upon later to be more generic (ex. use changefeed options instead of env var, use cdc queries to specify custom attributes, use a generic metadata struct instead of tablename string, pass metadata to different sinks and not just pubsub etc). Because this feature will be expanded in the future, the release note is intentionally left blank. Release note: None Closes: cockroachdb#115426
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Support attributes in pubsub messages.
For 23.2.1, we need to backport this for a customer. Can backport a minimal codechange to support attributes to keep the footprint and risk now.
For 24.1, we can ship a more robust version. See work in progress #111958
Jira issue: CRDB-34028
The text was updated successfully, but these errors were encountered: