Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mysql sink allocate a huge number of memory when build multi statement sql. #9935

Closed
3AceShowHand opened this issue Oct 20, 2023 · 1 comment · Fixed by #9936
Closed

mysql sink allocate a huge number of memory when build multi statement sql. #9935

3AceShowHand opened this issue Oct 20, 2023 · 1 comment · Fixed by #9936
Assignees
@3AceShowHand
Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. area/ticdc Issues or PRs related to TiCDC. component/mysql needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. type/enhancement The issue or PR belongs to an enhancement.

Comments

@3AceShowHand
Copy link
Contributor

Screenshot 2023-10-20 at 15 21 58

From the above image, we can see that the MySQL sink allocates a huge number of memory when trying to build the multi-statement SQL, this can be reduced by using a more efficient string concatenation method.
@3AceShowHand 3AceShowHand self-assigned this Oct 20, 2023
@3AceShowHand 3AceShowHand added type/enhancement The issue or PR belongs to an enhancement. area/ticdc Issues or PRs related to TiCDC. component/mysql labels Oct 20, 2023
@3AceShowHand 3AceShowHand mentioned this issue Oct 20, 2023
Merged
@3AceShowHand 3AceShowHand added needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. labels Oct 20, 2023
ti-chi-bot bot pushed a commit that referenced this issue Oct 20, 2023
@3AceShowHand
mysql(ticdc): use string join to build multi statement sql. (#9936)
e4ac90b 
close #9935
This was referenced Oct 20, 2023
Merged
Merged
Merged
@nongfushanquan
Copy link
Contributor

/label affects-6.5
/label affects-7.1
/label affects-7.5

@ti-chi-bot ti-chi-bot bot added affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. labels Oct 20, 2023
ti-chi-bot bot pushed a commit that referenced this issue Oct 20, 2023
@ti-chi-bot
mysql(ticdc): use string join to build multi statement sql. (#9936) (#…
4b2529f 
…9938)

close #9935
ti-chi-bot bot pushed a commit that referenced this issue Oct 20, 2023
@ti-chi-bot
mysql(ticdc): use string join to build multi statement sql. (#9936) (#…
4febedd 
…9939)

close #9935
@3AceShowHand 3AceShowHand mentioned this issue Oct 30, 2023
Closed
ti-chi-bot bot pushed a commit that referenced this issue Oct 30, 2023
@ti-chi-bot
mysql(ticdc): use string join to build multi statement sql. (#9936) (#…
4439ae4 
…9937)

close #9935
@nongfushanquan nongfushanquan mentioned this issue Dec 14, 2023
Merged
18 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@3AceShowHand 3AceShowHand

Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. area/ticdc Issues or PRs related to TiCDC. component/mysql needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

mysql(ticdc): use string join to build multi statement sql. 3AceShowHand/tiflow
2 participants
@3AceShowHand @nongfushanquan