-
Notifications
You must be signed in to change notification settings - Fork 3.9k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
colexec: add "recursive" merging to external sort
Previously, external sorter was merging all partitions at once (i.e. there was a single merger). This is problematic because each partition uses some amount of RAM for its buffer, we want to make sure that all partitions together do not exceed the memory limit. Now this is addressed by splitting previous "merging" stage into three: 1. new "repeated merging" stage is now responsible for merging all current partitions and spilling new partitions to disk until only one is left. This might be performed while we're still consuming the input. 2. new "final merging" stage that sets up an emitter that can merge all the remaining partitions. This occurs only when the input has been fully consumed and we can merge all the partitions at once, without having to spill to disk. 3. new " emitting" stage that simply emits the output. Release note: None
- Loading branch information
1 parent
dd98a30
commit 3429c96
Showing
6 changed files
with
204 additions
and
69 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.