TiCDC: The sync-point table got replicated (esp. in bdr-mode)

[kennytm]

What did you do?

1. Create two clusters

   tiup playground v7.6.0 --host 10.0.0.2 --db 1 --kv 1 --pd 1 --ticdc 1 --tiflash 0 --without-monitor
   rm /tmp/tidb-4000.sock
   tiup playground v7.6.0 --host 10.0.0.48 --db 1 --kv 1 --pd 1 --ticdc 1 --tiflash 0 --without-monitor

2. Setup two changefeeds between them

   # 2to48.toml
   bdr-mode = true
   enable-sync-point = true
   sync-point-interval = "31s"

   # 48to2.toml
   bdr-mode = true
   enable-sync-point = true
   sync-point-interval = "37s"

   tiup cdc:v7.6.0 cli changefeed create --pd http://10.0.0.2:2379 --sink-uri mysql://[email protected] -c 2to48 --config 2to48.toml
   tiup cdc:v7.6.0 cli changefeed create --pd http://10.0.0.48:2379 --sink-uri mysql://[email protected] -c 48to2 --config 48to2.toml

3. Wait for several minutes (because minSyncPointInterval is 30s 🤷)

4. Check the syncpoint tables

   mysql --host 10.0.0.2 --port 4000 -u root -e 'select * from tidb_cdc.syncpoint_v1;'
   mysql --host 10.0.0.2 --port 4000 -u root -e 'select count(*) from tidb_cdc.syncpoint_v1;'
   
   mysql --host 10.0.0.48 --port 4000 -u root -e 'select * from tidb_cdc.syncpoint_v1;'
   mysql --host 10.0.0.48 --port 4000 -u root -e 'select count(*) from tidb_cdc.syncpoint_v1;'

What did you expect to see?

The syncpoint table in 10.0.0.2 only contained entries of the 48to2 changefeed,
and similarly 10.0.0.48 only contained entries of the 2to48 changefeed.

What did you see instead?

Both clusters contained sync-point entries from both changefeeds.

Versions of the cluster

v7.6.0

[kennytm]

What happened here is that tidb_cdc.* is not treated as a system schema (unlike DM_HEARTBEAT.*), so the syncpoint_v1 table will be captured by CDC and replicated.

I think two changes should be made:

1. Make isSysSchema() also treat tidb_cdc.* as a system schema. This also helps the case of non-BDR "A→B→C" scenario, where the syncpoint table of A→B polluted B→C.
2. Set @@TIDB_CDC_WRITE_SOURCE for all operations in https://github.com/pingcap/tiflow/blob/v7.6.0/cdc/syncpointstore/mysql_syncpoint_store.go, so the A→B change won't even get pulled from B's TiKV in B→A.

[fubinzh]

/severity major

[asddongmen]

What happened here is that tidb_cdc.* is not treated as a system schema (unlike DM_HEARTBEAT.*), so the syncpoint_v1 table will be captured by CDC and replicated.

I think two changes should be made:

1. Make isSysSchema() also treat tidb_cdc.* as a system schema. This also helps the case of non-BDR "A→B→C" scenario, where the syncpoint table of A→B polluted B→C.
2. Set @@TIDB_CDC_WRITE_SOURCE for all operations in v7.6.0/cdc/syncpointstore/mysql_syncpoint_store.go, so the A→B change won't even get pulled from B's TiKV in B→A.

Thank you for the advice. In my opinion, only step 1 is necessary. When the schema tidb_cdc is considered as a system schema, it will not be listened to by CDC at all.

[seiya-annie]

/found customer