Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

stuck check can still make some mistake reports #9736

Closed
hicqu opened this issue Sep 13, 2023 · 0 comments · Fixed by #9742
Closed

stuck check can still make some mistake reports #9736

hicqu opened this issue Sep 13, 2023 · 0 comments · Fixed by #9742
Assignees
@hicqu
Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. area/ticdc Issues or PRs related to TiCDC. severity/moderate type/bug The issue is confirmed as a bug.

Comments

@hicqu
Copy link
Contributor

hicqu commented Sep 13, 2023

What did you do?

Start one TiCDC instance, and run a changefeed with 30 tables. Then

  1. pause the changefeed, and prepare some data in the upstream cluster;
  2. inject a latency into the sink;
  3. resume the changefeed;
  4. after a while, we can get the log "stuck too long".

Why "stuck too long" happens?

It's because the table is the last one to finish incremental scan. There are already 29 tables scheduled into sink. After the last table gets scheduled, its resolvedTs has been updated, but its checkpoint can only be advanced after all other tables
pending data has been consumed. So, the stuck check is triggered, which is unexpected.

What did you expect to see?

No response

What did you see instead?

As decribed above.

Versions of the cluster

Master branch.
@hicqu hicqu added type/bug The issue is confirmed as a bug. area/ticdc Issues or PRs related to TiCDC. labels Sep 13, 2023
@hicqu hicqu self-assigned this Sep 13, 2023
@hicqu hicqu added affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. labels Sep 13, 2023
@hicqu hicqu mentioned this issue Sep 14, 2023
Merged
ti-chi-bot bot pushed a commit that referenced this issue Sep 14, 2023
@hicqu
sink(cdc): only check sink stuck for MQ sinks (#9742)
141c9a7 
close #9736
This was referenced Sep 14, 2023
Merged
Merged
ti-chi-bot bot pushed a commit that referenced this issue Sep 14, 2023
@ti-chi-bot
sink(cdc): only check sink stuck for MQ sinks (#9742) (#9747)
11b023a 
close #9736
@nongfushanquan nongfushanquan mentioned this issue Sep 15, 2023
Merged
18 tasks
@jayl1e jayl1e mentioned this issue Sep 21, 2023
Closed
11 tasks
hicqu added a commit to ti-chi-bot/tiflow that referenced this issue Sep 26, 2023
@hicqu
sink(cdc): only check sink stuck for MQ sinks (pingcap#9742)
eb670c0 
close pingcap#9736
ti-chi-bot bot pushed a commit that referenced this issue Sep 26, 2023
@ti-chi-bot
sink(cdc): only check sink stuck for MQ sinks (#9742) (#9748)
5e20eb6 
close #9736
@nongfushanquan nongfushanquan mentioned this issue Oct 18, 2023
Merged
18 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hicqu hicqu

Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. area/ticdc Issues or PRs related to TiCDC. severity/moderate type/bug The issue is confirmed as a bug.
Projects
Question and Bug Reports
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

sink(cdc): only check sink stuck for MQ sinks hicqu/ticdc
1 participant
@hicqu