Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TiFlash crashes because of out of threads #7747

Closed
Tracked by #6438
JaySon-Huang opened this issue Jul 5, 2023 · 2 comments
Closed
Tracked by #6438

TiFlash crashes because of out of threads #7747

JaySon-Huang opened this issue Jul 5, 2023 · 2 comments
Assignees
@windtalker
Labels
affects-5.3 affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. component/compute severity/major type/bug The issue is confirmed as a bug.

Comments

@JaySon-Huang
Copy link
Contributor

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  • TiDB use some incorrect plan that issue lots of coprocessor request to TiFlash
  • TiFlash can not handle the coprocessor request in time, resulting in request timeout
  • TiDB retry by sending more coprocessor request to TiFlash

2. What did you expect to see? (Required)

TiFlash reject the coprocessor request that it can not handle in time quickly

3. What did you see instead (Required)

Handling coprocessor requests reach more than 30k and TiFlash crashes because of out of threads

image

4. What is your TiFlash version? (Required)

v6.5.2
@JaySon-Huang JaySon-Huang added type/bug The issue is confirmed as a bug. severity/major component/compute affects-6.5 This bug affects the 6.5.x(LTS) versions. labels Jul 5, 2023
@JaySon-Huang JaySon-Huang mentioned this issue Jul 5, 2023
Merged
12 tasks
@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Jul 5, 2023 •
edited
Loading

It should be resolved by #6438. However, we still see this happens on v6.5.2.

Most changes of #6438 has been merged into the release-6.5 branch, just the changes in #6394 is not yet merged.

@JaySon-Huang JaySon-Huang mentioned this issue Jul 5, 2023
Open
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Jul 5, 2023
@ti-chi-bot
limit the queued task number and queued duration of coprocess task. (#…
72da328 
…6394) (#7739)

ref #6438, close #7747
@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Jul 5, 2023 •
edited
Loading

close as #7739 has been merged to release-6.5

@ti-chi-bot ti-chi-bot mentioned this issue Jul 11, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Jul 11, 2023
@ti-chi-bot
limit the queued task number and queued duration of coprocess task. (#…
676896d 
…6394) (#7739) (#7788)

ref #6438, close #7747
@ti-chi-bot ti-chi-bot added affects-5.3 affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.1 This bug affects the 6.1.x(LTS) versions. labels Aug 17, 2023
@windtalker windtalker mentioned this issue Aug 23, 2023
Merged
17 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@windtalker windtalker

Labels
affects-5.3 affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. component/compute severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@windtalker @JaySon-Huang @ti-chi-bot