-
Notifications
You must be signed in to change notification settings - Fork 289
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ddl events should not be filtered in puller #10524
Labels
affects-6.1
This bug affects the 6.1.x(LTS) versions.
affects-6.5
This bug affects the 6.5.x(LTS) versions.
affects-7.1
This bug affects the 7.1.x(LTS) versions.
affects-7.5
This bug affects the 7.5.x(LTS) versions.
area/ticdc
Issues or PRs related to TiCDC.
report/customer
Customers have encountered this bug.
severity/moderate
type/bug
The issue is confirmed as a bug.
Comments
CharlesCheung96
added
type/bug
The issue is confirmed as a bug.
severity/moderate
affects-6.5
This bug affects the 6.5.x(LTS) versions.
affects-7.1
This bug affects the 7.1.x(LTS) versions.
affects-7.5
This bug affects the 7.5.x(LTS) versions.
area/ticdc
Issues or PRs related to TiCDC.
labels
Jan 22, 2024
This was referenced Jan 24, 2024
CharlesCheung96
added a commit
to ti-chi-bot/tiflow
that referenced
this issue
Jan 24, 2024
CharlesCheung96
added a commit
to ti-chi-bot/tiflow
that referenced
this issue
Jan 24, 2024
CharlesCheung96
added a commit
to ti-chi-bot/tiflow
that referenced
this issue
Jan 24, 2024
CharlesCheung96
added a commit
to ti-chi-bot/tiflow
that referenced
this issue
Jan 24, 2024
/found customer |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
affects-6.1
This bug affects the 6.1.x(LTS) versions.
affects-6.5
This bug affects the 6.5.x(LTS) versions.
affects-7.1
This bug affects the 7.1.x(LTS) versions.
affects-7.5
This bug affects the 7.5.x(LTS) versions.
area/ticdc
Issues or PRs related to TiCDC.
report/customer
Customers have encountered this bug.
severity/moderate
type/bug
The issue is confirmed as a bug.
The dataflow of ddl in TiCDC is:
There are two methods to filter ddl event:
In the current implementation, TiCDC discard ddl events, which should be filtered, in ddlPuller and the subsequent modules of puller do not handle such ddl events. This causes cdc to behave unreasonably when using
event-filter rules
.For example, if we create a changefeed-test with the following config:
Then,
create table test.t0
, cdc will capture table t0.create table test.t1
, cdc will discard the ddl since it match event-filter rules. Therefore, cdc will not capture table t1.In conclusion, the behavior of TiCDC in ignoring create/delete/truncate/alter DDL in puller module is as follows:
Case 1: After a table or partition is created, TiCDC does not synchronize the new physical table. However, cdc will synchronize the new physical table after the changefeed restarts.
Case 2: After a table or partition is dropped, TiCDC still synchronizes the deleted physical table. However, cdc will not synchronize the deleted physical table after the changefeed restarts.
Case 3: After a table or partition is truncated, TiCDC does not synchronize the new physical table, but still synchronizes the deleted physical table.
Case 4: After the Alter table or partition ddl, CDC still uses the schma of the old table to parse data, wich leads to unpredictable data inconsistencies.
The text was updated successfully, but these errors were encountered: