Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Too many logs: "synchronize is taking too long, report a bug" occupied a lot disk capacity #2706

Closed
Tammyxia opened this issue Sep 1, 2021 · 4 comments · Fixed by #3775
Closed

Too many logs: "synchronize is taking too long, report a bug" occupied a lot disk capacity #2706

Tammyxia opened this issue Sep 1, 2021 · 4 comments · Fixed by #3775
Assignees
@overvenus
Labels
area/ticdc Issues or PRs related to TiCDC. priority/P0 The issue has P0 priority. severity/major type/bug The issue is confirmed as a bug.

Comments

@Tammyxia
Copy link

Tammyxia commented Sep 1, 2021 •
edited
Loading

Bug Report

Please answer these questions before submitting your issue. Thanks!

  1. What did you do? If possible, provide a recipe for reproducing the error.
  • Workload: DML + 20 new creating tables.
  • TiKV scale-in from 24 to 20. Total region 20w.

  1. What did you expect to see?

  2. What did you see instead?

  • Owner node new generated 230+ cdc log in 20hours, each log file is 300MB, which occupied too much disk capacity.

image
image

  1. Versions of the cluster

    • Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

      4.0.14

    • TiCDC version (execute cdc version):

      4.0.14 cloud
@Tammyxia Tammyxia added type/bug The issue is confirmed as a bug. severity/moderate labels Sep 1, 2021
@liuzix liuzix self-assigned this Sep 2, 2021
@liuzix liuzix added the priority/P0 The issue has P0 priority. label Sep 2, 2021
@liuzix
Copy link
Contributor

liuzix commented Sep 2, 2021

This seems to be the exact problem solved by #2078

@overvenus
Copy link
Member

@liuzix Can we set the level to debug or reduce frequency? In this case, it takes lots of CPU and IO and may block the entire TiCDC process.

image

@liuzix liuzix closed this as completed Sep 7, 2021
@cyliu0 cyliu0 reopened this Sep 9, 2021
@amyangfei
Copy link
Contributor

amyangfei commented Sep 9, 2021 •
edited
Loading

Should add this commit to master branch pingcap/ticdc@d6ac981

@overvenus
Copy link
Member

@liuzix Please cherry-pick pingcap/ticdc@d6ac981 to master branch, thanks!

@AkiraXie AkiraXie added the area/ticdc Issues or PRs related to TiCDC. label Dec 6, 2021
@overvenus overvenus assigned overvenus and unassigned liuzix Dec 7, 2021
@overvenus overvenus mentioned this issue Dec 7, 2021
Merged
This was referenced Dec 8, 2021
Merged
Merged
Merged
This was referenced Dec 8, 2021
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@overvenus overvenus

Labels
area/ticdc Issues or PRs related to TiCDC. priority/P0 The issue has P0 priority. severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

workerpool: limit the rate to output deadlock warning overvenus/ticdc
6 participants
@amyangfei @overvenus @liuzix @cyliu0 @AkiraXie @Tammyxia