Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sink(ticdc): limit the number of transactions cached in a mysql worker #10892

Merged
merged 10 commits into from
Apr 22, 2024
Merged

sink(ticdc): limit the number of transactions cached in a mysql worker #10892

merged 10 commits into from
Apr 22, 2024

Conversation

CharlesCheung96
Copy link
Contributor

@CharlesCheung96 CharlesCheung96 commented Apr 9, 2024 •
edited
Loading

What problem does this PR solve?

Issue Number: close #10896

What is changed and how it works?

  1. Remove the Add method of Txn.Worker to reduce interdependencies between modules.
  2. Use the method of conflictDetector to receive event in Txn.Worker.
  3. Implement causality.worker that support different batch control policies.

Check List

Tests

In the workload skew case, the maximum performance improvement after optimization is about 300% ~ 1000%.

Questions

Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?

Release note 

`Fix the issue that workload could not be evenly distributed among different mysql sink workers in some scenarios`.

@ti-chi-bot ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Apr 9, 2024

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@ti-chi-bot ti-chi-bot bot added do-not-merge/needs-linked-issue release-note Denotes a PR that will be considered when it comes time to generate release notes. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/needs-linked-issue release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Apr 9, 2024
@CharlesCheung96 CharlesCheung96 marked this pull request as ready for review April 9, 2024 08:54
@ti-chi-bot ti-chi-bot bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none Denotes a PR that doesn't merit a release note. labels Apr 9, 2024
@CharlesCheung96 CharlesCheung96 force-pushed the add-batch-limit branch 3 times, most recently from e159815 to c302346 Compare April 11, 2024 05:14
@ti-chi-bot ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Apr 11, 2024
lidezhu
lidezhu reviewed Apr 11, 2024
View reviewed changes
pkg/causality/worker.go Outdated Show resolved Hide resolved
@CharlesCheung96 CharlesCheung96 requested a review from lidezhu April 12, 2024 05:51
@ti-chi-bot ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Apr 16, 2024
@CharlesCheung96 CharlesCheung96 force-pushed the add-batch-limit branch 2 times, most recently from 51d2935 to 000f85d Compare April 17, 2024 05:59
@pingcap pingcap deleted a comment from ti-chi-bot bot Apr 17, 2024
@3AceShowHand
Copy link
Contributor

/retest

@ti-chi-bot ti-chi-bot mentioned this pull request Apr 22, 2024
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-7.5: #10941.

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Apr 22, 2024
@CharlesCheung96 @ti-chi-bot
This is an automated cherry-pick of pingcap#10892
77a3200 
Signed-off-by: ti-chi-bot <[email protected]>
@ti-chi-bot ti-chi-bot mentioned this pull request Apr 22, 2024
Merged
@CharlesCheung96 CharlesCheung96 added needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. needs-cherry-pick-release-8.1 Should cherry pick this PR to release-8.1 branch. labels Apr 24, 2024
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-8.1: #10958.

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Apr 24, 2024
@CharlesCheung96 @ti-chi-bot
This is an automated cherry-pick of pingcap#10892
5d3c922 
Signed-off-by: ti-chi-bot <[email protected]>
@ti-chi-bot ti-chi-bot mentioned this pull request Apr 24, 2024
Merged
ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Apr 24, 2024
@CharlesCheung96 @ti-chi-bot
This is an automated cherry-pick of pingcap#10892
7537db1 
Signed-off-by: ti-chi-bot <[email protected]>
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-6.5: #10959.

@ti-chi-bot ti-chi-bot mentioned this pull request Apr 24, 2024
Merged
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this pull request Apr 26, 2024
@CharlesCheung96
sink(ticdc): limit the number of transactions cached in a mysql worker (
7012f8f 
pingcap#10892)

close pingcap#10896
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this pull request Apr 26, 2024
@CharlesCheung96
sink(ticdc): limit the number of transactions cached in a mysql worker (
445a471 
pingcap#10892)

close pingcap#10896
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this pull request Apr 26, 2024
@CharlesCheung96
sink(ticdc): limit the number of transactions cached in a mysql worker (
30d8389 
pingcap#10892)

close pingcap#10896
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this pull request Apr 26, 2024
@CharlesCheung96
sink(ticdc): limit the number of transactions cached in a mysql worker (
c3f8823 
pingcap#10892)

close pingcap#10896
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this pull request Apr 26, 2024
@CharlesCheung96
sink(ticdc): limit the number of transactions cached in a mysql worker (
c68c96e 
pingcap#10892)

close pingcap#10896
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this pull request Apr 26, 2024
@CharlesCheung96
sink(ticdc): limit the number of transactions cached in a mysql worker (
febf443 
pingcap#10892)

close pingcap#10896
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this pull request Apr 26, 2024
@CharlesCheung96
sink(ticdc): limit the number of transactions cached in a mysql worker (
89e783b 
pingcap#10892)

close pingcap#10896
ti-chi-bot bot pushed a commit that referenced this pull request Apr 28, 2024
@ti-chi-bot
sink(ticdc): limit the number of transactions cached in a mysql worker (
c56e6ba 
#10892) (#10959)

close #10896
ti-chi-bot bot pushed a commit that referenced this pull request Apr 29, 2024
@ti-chi-bot
sink(ticdc): limit the number of transactions cached in a mysql worker (
687dea0 
#10892) (#10940)

close #10896
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this pull request Apr 29, 2024
@CharlesCheung96
sink(ticdc): limit the number of transactions cached in a mysql worker (
7fcc953 
pingcap#10892)

close pingcap#10896
ti-chi-bot bot pushed a commit that referenced this pull request Apr 30, 2024
@ti-chi-bot
sink(ticdc): limit the number of transactions cached in a mysql worker (
ceabaab 
#10892) (#10958)

close #10896
ti-chi-bot bot pushed a commit that referenced this pull request May 20, 2024
@ti-chi-bot
sink(ticdc): limit the number of transactions cached in a mysql worker (
764fdb4 
#10892) (#10941)

close #10896
CharlesCheung96 added a commit to CharlesCheung96/tiflow that referenced this pull request Jun 6, 2024
@CharlesCheung96
Revert "sink(ticdc): limit the number of transactions cached in a mys…
db84468 
…ql worker (pingcap#10892) (pingcap#10941)"

This reverts commit 764fdb4.
This was referenced Jun 6, 2024
Closed
Closed
hicqu added a commit to hicqu/ticdc that referenced this pull request Jun 12, 2024
@hicqu
Revert "sink(ticdc): limit the number of transactions cached in a mys…
cf48014 
…ql worker (pingcap#10892) (pingcap#10959)"

This reverts commit c56e6ba.
hicqu added a commit to ti-chi-bot/tiflow that referenced this pull request Jun 12, 2024
@hicqu
Revert "sink(ticdc): limit the number of transactions cached in a mys…
2d96d8a 
…ql worker (pingcap#10892) (pingcap#10959)"

This reverts commit c56e6ba.
hicqu added a commit to hicqu/ticdc that referenced this pull request Jun 13, 2024
@hicqu
Revert "sink(ticdc): limit the number of transactions cached in a mys…
235c155 
…ql worker (pingcap#10892) (pingcap#10959)"

This reverts commit c56e6ba.

Signed-off-by: qupeng <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lidezhu lidezhu lidezhu approved these changes

@asddongmen asddongmen asddongmen approved these changes

@hongyunyan hongyunyan Awaiting requested review from hongyunyan

@hicqu hicqu Awaiting requested review from hicqu

@3AceShowHand 3AceShowHand Awaiting requested review from 3AceShowHand

Assignees

@CharlesCheung96 CharlesCheung96

Labels
approved lgtm needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. needs-cherry-pick-release-8.1 Should cherry pick this PR to release-8.1 branch. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

limit the maximum number of cached txns in mysql worker
5 participants
@CharlesCheung96 @3AceShowHand @lidezhu @ti-chi-bot @asddongmen