Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support sharing RNFetchPagesWorker and PrepareStreamsWorker between queries #7878

Closed
JinheLin opened this issue Aug 1, 2023 · 1 comment
Closed

Support sharing RNFetchPagesWorker and PrepareStreamsWorker between queries #7878

JinheLin opened this issue Aug 1, 2023 · 1 comment
Assignees
@JinheLin
Labels
type/enhancement The issue or PR belongs to an enhancement.

Comments

@JinheLin
Copy link
Contributor

JinheLin commented Aug 1, 2023

Enhancement

Currently, TiFlash will create a RNFetchPagesWorker and PrepareStreamsWorker for each query. This introduces the cost of frequently creating and destroying threads, and at the same time, it is impossible to control the number of thread concurrency on a node, which can easily lead to OOM.

We can use RNFetchPagesWorker and PrepareStreamsWorker as global thread pools and shard these thread pools between queries.
@JinheLin JinheLin added the type/enhancement The issue or PR belongs to an enhancement. label Aug 1, 2023
@JinheLin JinheLin self-assigned this Aug 1, 2023
@JinheLin JinheLin mentioned this issue Aug 1, 2023
Closed
12 tasks
@JinheLin
Copy link
Contributor Author

JinheLin commented Nov 1, 2023

Replace by #6834

@JinheLin JinheLin closed this as completed Nov 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@JinheLin JinheLin

Labels
type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Support sharing RNFetchPagesWorker and PrepareStreamsWorker between queries. JinheLin/tiflash-1
1 participant
@JinheLin