Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevent over-allocation (and spills) on TopK queries #3596

Closed
isidentical opened this issue Sep 23, 2022 · 0 comments · Fixed by #3593
Closed

Prevent over-allocation (and spills) on TopK queries #3596

isidentical opened this issue Sep 23, 2022 · 0 comments · Fixed by #3593
Labels
enhancement New feature or request

Comments

@isidentical
Copy link
Contributor

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
During sorting, when we receive a new record batch we try to allocate space for it. This is done with the assumption that the result of this sort will still be around, and we don't want to accidentally overflow the memory so we have to keep track of it. But after the #3510, this assumption might not hold for all cases (particularly when you have a fetch limit set on your sorting operation) so we might be over-allocating memory and constantly spilling for no good reason.

Describe the solution you'd like
Avoiding over-allocations by instructing the memory manager to shrink after each partial sort with a limit.

Describe alternatives you've considered
Leaving it as is, which would mean tons of unnecessary spills under a heavy load of data with a fixed limit.

Additional context
Originally posted here: #3579 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant