You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
During sorting, when we receive a new record batch we try to allocate space for it. This is done with the assumption that the result of this sort will still be around, and we don't want to accidentally overflow the memory so we have to keep track of it. But after the #3510, this assumption might not hold for all cases (particularly when you have a fetch limit set on your sorting operation) so we might be over-allocating memory and constantly spilling for no good reason.
Describe the solution you'd like
Avoiding over-allocations by instructing the memory manager to shrink after each partial sort with a limit.
Describe alternatives you've considered
Leaving it as is, which would mean tons of unnecessary spills under a heavy load of data with a fixed limit.
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
During sorting, when we receive a new record batch we try to allocate space for it. This is done with the assumption that the result of this sort will still be around, and we don't want to accidentally overflow the memory so we have to keep track of it. But after the #3510, this assumption might not hold for all cases (particularly when you have a fetch limit set on your sorting operation) so we might be over-allocating memory and constantly spilling for no good reason.
Describe the solution you'd like
Avoiding over-allocations by instructing the memory manager to shrink after each partial sort with a limit.
Describe alternatives you've considered
Leaving it as is, which would mean tons of unnecessary spills under a heavy load of data with a fixed limit.
Additional context
Originally posted here: #3579 (comment)
The text was updated successfully, but these errors were encountered: