ddl_puller(ticdc): discard DDLs that irrelevant with changefeed

[asddongmen]

What problem does this PR solve?

Issue Number: ref #6447 #3607

What is changed and how it works?

After this PR, TiCDC filters out DDLs that are not related to changefeed in ddl_puller, which can reduces the latency and memory consumption of replicating DDLs.

It is worth noting that before this PR, for rename table DDL, cdc will filter it with the new table name. After this PR, cdc will filter rename table by its old table name, which is more in line with the semantics of the filter rule.

For example:
----------------------------------
filter:  test.t1(white list)
DDL1:  rename test.t1 to test.t11
DDL2:  rename test.t11 to test.t1
----------------------------------
Before this pr:
CDC will ignore DDL1, sink DDL2.
It will cause upstream and downstream inconsistent.

Result: 
1. Upstream has table 't11',  but downstream still holds table 't1'.
2. When sink DDL2, if downstream don't have the table 't11' before, it may raise error in CDC. 
(This changefeed don't care about the 't11' actually)
----------------------------------
After this pr:
CDC will sink DDL1,  report error DDL2.

Result: 
1. Both upstream and downstream will have table 't11'.
2. CDC reports error for DDL2. More details in following explaination.
(This changefeed don't care about the table 't11' before. User need confirm the error before sinking DDL2)
------------------------------------

And, if the user wants to rename a table with an old table name that does not match filter.rule to a new table name that matches filter.rule, then an error will be reported directly.

For example, we have filter.rule = ['test.t1'], and there are tables name test.t11 in the upstream TiDB:

1. excute rename table 'test.t11' to 'test.t1' will report an error.

In addition, if you rename multiple tables in a single one rename table DDL, TiCDC will apply the rule mention above to every single subordinate rename table DDL jobs.

For example, we have filter.rule = ['test.t1', test.t3], and there are tables name test.t1, test.t2 in the upstream TiDB:

1. excute rename table 'test.t1' to 'test.t11', 'test.t2' to 'test.t22', the rename table 'test.t1' will be replicated and 'test.t2' to 'test.t22'will be filtered out sincetest.t1matchfilter.rulebuttest.t2` does not.
2. excuet rename table 'test.t2' to 'test.t3', 'test.t1' to 'test.t11' will report an error, since test.t2 does not match filter.rule but test.t3 does, TiCDC do not allow such behavior.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)

1. start cdc server
2. create a chengfeed with filter.rule = ['test1.t1', 'test1.t2', 'test1.t3', 'test1.t4']
3. create t1, t2, t3, t4, t5, t6 in upstream tidb.
4. exec : rename table t1 to t11 (replicated)
5. exec : rename table t11 to t1 (error)

➜  bin git:(filter_ddl) ✗ ./cdc cli changefeed list
[
  {
    "id": "test-1",
    "namespace": "default",
    "summary": {
      "state": "error",
      "tso": 435389327982985217,
      "checkpoint": "2022-08-19 11:07:58.939",
      "error": {
        "addr": "127.0.0.1:8300",
        "code": "CDC:ErrSyncRenameTablesFailed",
        "message": "[CDC:ErrSyncRenameTablesFailed]table's old name is not in filter rule, and its new name in filter rule table id '108', ddl query: [rename table t11 to t1], it's an unexpected behavior, if you want to replicate this table, please add its old name to filter rule."
      }
    }
  }
]

8. exec: rename table t2 to t22, t3 to t33 (replicated)
9. exec: rename table t22 to t222, t33 to t333 (discard)
10. exec: rename table t222 to t2, t333 to t3 (error)

➜  bin git:(filter_ddl) ✗ ./cdc cli changefeed list
[
  {
    "id": "test-1",
    "namespace": "default",
    "summary": {
      "state": "error",
      "tso": 435389397673967617,
      "checkpoint": "2022-08-19 11:12:24.789",
      "error": {
        "addr": "127.0.0.1:8300",
        "code": "CDC:ErrSyncRenameTablesFailed",
        "message": "[CDC:ErrSyncRenameTablesFailed]table's old name is not in filter rule, and its new name in filter rule table id '110', ddl query: [rename table t222 to t2, t333 to t3], it's an unexpected behavior, if you want to replicate this table, please add its old name to filter rule."
      }
    }
  }
]

11. exec: rename table t4 to t44, t5 to t55
    It will be splited into two parts, the rename table t4 to t44 part is replicated and t5 to t55 is ignored.

You can learn more discussion detail in this internal doc: https://pingcap.feishu.cn/wiki/wikcnJ6SaB5cGLeG5OrM4nf6MPg

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Improve replicating performance by discard DDLs that irrelevant with a changefeed.

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• maxshuang
• zhaoxinyu

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[asddongmen]

/run-integration-tests

[codecov-commenter]

Codecov Report

Merging #6446 (0d5272f) into master (7c5032c) will increase coverage by 0.0058%.
The diff coverage is 51.8737%.

Additional details and impacted files

Flag	Coverage Δ
cdc	67.2666% <62.9411%> (-0.1103%)	⬇️
dm	52.0500% <36.0184%> (+0.0514%)	⬆️
engine	62.6212% <64.9269%> (-0.0142%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             master      #6446        +/-   ##
================================================
+ Coverage   59.9224%   59.9283%   +0.0058%     
================================================
  Files           783        789         +6     
  Lines         89519      90121       +602     
================================================
+ Hits          53642      54008       +366     
- Misses        31168      31369       +201     
- Partials       4709       4744        +35     

[asddongmen]

/run-integration-tests

[asddongmen]

/run-integration-tests

[asddongmen]

/run-integration-tests

[asddongmen]

/run-integration-tests

[asddongmen]

/run-integration-tests

[asddongmen]

/run-all-tests

[asddongmen]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 0d5272f

[ti-chi-bot]

@asddongmen: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[asddongmen]

/run-leak-tests

[asddongmen]

/run-leak-tests

[asddongmen]

/run-leak-tests