Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

changefeedccl: filter schema changes that don't affect any watched families #80982

Closed
HonoreDB opened this issue May 4, 2022 · 1 comment · Fixed by #84674
Closed

changefeedccl: filter schema changes that don't affect any watched families #80982

HonoreDB opened this issue May 4, 2022 · 1 comment · Fixed by #84674
Assignees
@jayshrivastava
Labels
A-cdc Change Data Capture C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) E-quick-win Likely to be a quick win for someone experienced. E-starter Might be suitable for a starter project for new employees or team members. T-cdc

Comments

@HonoreDB
Copy link
Contributor

HonoreDB commented May 4, 2022 •
edited by cockroach-jira-scripts
Loading

Currently, if you do

CREATE TABLE hasfams (id int primary key, a string, b string, FAMILY id_a (id, a), FAMILY onlyb (b));
CREATE CHANGEFEED FOR TABLE hasfams FAMILY onlyb WITH schema_change_policy='stop';
ALTER TABLE hasfams DROP COLUMN a;

The changefeed will stop. Without the schema_change_policy='stop' it will instead re-emit all records. Both of these behaviors are suboptimal, as the schema change hasn't affected the primary key or the watched column family. Instead, the schema change should be ignored using logic in schemafeed/table_event_filter.go.

Jira issue: CRDB-15409
@HonoreDB HonoreDB added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) E-starter Might be suitable for a starter project for new employees or team members. A-cdc Change Data Capture T-cdc E-quick-win Likely to be a quick win for someone experienced. labels May 4, 2022
@blathers-crl
Copy link

blathers-crl bot commented May 4, 2022

cc @cockroachdb/cdc

@jlinder jlinder added sync-me and removed sync-me labels May 20, 2022
@jayshrivastava jayshrivastava self-assigned this Jul 14, 2022
jayshrivastava added a commit to jayshrivastava/cockroach that referenced this issue Jul 20, 2022
@jayshrivastava
changefeedccl/kvfeed: ignore unwatched column drops
7766f7b 
See release note.

Fixes cockroachdb#80982

Release note (enterprise change): Previously, if you dropped
a column with the schema_change_policy 'stop', the changefeed
would stop. Dropping a column with a different policy would
result in previous rows being retransmitted with the
dropped column omitted.

In some cases, a changefeed may target specific columns
(a column family) of a table. In these cases, if a non-target
column is dropped, it does not make sense to stop the changefeed
or retransmit values because the column was not visible to
a consumer sink to begin with.

With this change, dropping an non-target column from a
table will not stop the changefeed when the
schema_change_policy is 'stop'. With any other policy,
dropping a non-target column will not trigger a backfill.
jayshrivastava added a commit to jayshrivastava/cockroach that referenced this issue Jul 21, 2022
@jayshrivastava
changefeedccl/schemafeed: ignore unwatched column drops
31266fe 
See release note.

Fixes cockroachdb#80982

Release note (enterprise change): Previously, if you dropped
a column with the schema_change_policy 'stop', the changefeed
would stop. Dropping a column with a different policy would
result in previous rows being retransmitted with the
dropped column omitted.

In some cases, a changefeed may target specific columns
(a column family) of a table. In these cases, if a non-target
column is dropped, it does not make sense to stop the changefeed
or retransmit values because the column was not visible to
a consumer sink to begin with.

With this change, dropping an non-target column from a
table will not stop the changefeed when the
schema_change_policy is 'stop'. With any other policy,
dropping a non-target column will not trigger a backfill.
@jayshrivastava jayshrivastava mentioned this issue Jul 21, 2022
Merged
jayshrivastava added a commit to jayshrivastava/cockroach that referenced this issue Jul 22, 2022
@jayshrivastava
changefeedccl/schemafeed: ignore unwatched column drops
c6f667c 
See release note.

Fixes cockroachdb#80982

Release note (enterprise change): Previously, if you dropped
a column with the schema_change_policy 'stop', the changefeed
would stop. Dropping a column with a different policy would
result in previous rows being retransmitted with the
dropped column omitted.

In some cases, a changefeed may target specific columns
(a column family) of a table. In these cases, if a non-target
column is dropped, it does not make sense to stop the changefeed
or retransmit values because the column was not visible to
a consumer sink to begin with.

With this change, dropping an non-target column from a
table will not stop the changefeed when the
schema_change_policy is 'stop'. With any other policy,
dropping a non-target column will not trigger a backfill.
@jayshrivastava jayshrivastava mentioned this issue Jul 25, 2022
Closed
craig bot pushed a commit that referenced this issue Jul 26, 2022
@maryliag @jayshrivastava @yuzefovich
Merge #84618 #84674 #85034 #85042
d7b901d 
84618: sql: add new index_recommendation column r=maryliag a=maryliag

This commit adds a new column index_recommendations
STRING[] to:
crdb_internal.node_statement_statistics
crdb_internal.cluster_statement_statistics
system.statement_statistics
crdb_internal.statement_statistics

Part of #83782

Release note (sql change): Adding new column index_recommendations
to crdb_internal.node_statement_statistics,
crdb_internal.cluster_statement_statistics, system.statement_statistics
and crdb_internal.statement_statistics

84674: changefeedccl/schemafeed: ignore unwatched column drops  r=jayshrivastava a=jayshrivastava

Closes #80982

Release note (enterprise change): Previously, if you dropped
a column with the schema_change_policy 'stop', the changefeed
would stop. Dropping a column with a different policy would
result in previous rows being retransmitted with the
dropped column omitted.

In some cases, a changefeed may target specific columns
(a column family) of a table. In these cases, if a non-target
column is dropped, it does not make sense to stop the changefeed
or retransmit values because the column was not visible to
a consumer sink to begin with.

With this change, dropping an non-target column from a
table will not stop the changefeed when the
schema_change_policy is 'stop'. With any other policy,
dropping a non-target column will not trigger a backfill.

85034: sql: better tracing of apply joins r=yuzefovich a=yuzefovich

**sql: add tracing spans for each iteration of apply join**

This commit creates a separate tracing span for each iteration of apply
join and recursive CTE execution to make the traces easier to digest.

Release note: None

**sql: log DistSQL diagrams when tracing is enabled**

This commit makes it so that we always include all DistSQL diagrams into
the trace. This could be especially helpful when executing apply join
iterations to understand the plan that each iteration gets. This commit
also removes an environment variable that used to control this logging
previously since I don't think anyone has used it in years now that we
have better tools for debugging (like a stmt bundle).

Informs: #https://github.com/cockroachlabs/support/issues/1681.

Release note: None

85042: logictest: remove spec-planning configs r=yuzefovich a=yuzefovich

This commit removes `local-spec-planning`, `fakedist-spec-planning`, and
`5node-spec-planning` logic test configurations since they seem to be
not very useful at the moment. They were introduced to support the new
DistSQL spec factory, but that work has been postponed with no active
development at the moment, so it seems silly to run most of the logic
tests through the configs that are largely duplicate of the other
default ones (because most of the `Construct*` methods are not
implemented in the new factory). Once the active development on the new
factory resumes, it'll be pretty easy to bring them back to life, but at
the moment let's reduce the amount of tests we run without really losing
any test coverage.

Release note: None

Co-authored-by: Marylia Gutierrez <[email protected]>
Co-authored-by: Jayant Shrivastava <[email protected]>
Co-authored-by: Yahor Yuzefovich <[email protected]>
@craig craig bot closed this as completed in 74bccc7 Jul 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jayshrivastava jayshrivastava

Labels
A-cdc Change Data Capture C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) E-quick-win Likely to be a quick win for someone experienced. E-starter Might be suitable for a starter project for new employees or team members. T-cdc
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

changefeedccl/schemafeed: ignore unwatched column drops jayshrivastava/cockroach
3 participants
@jlinder @HonoreDB @jayshrivastava