Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Browse files
Browse the repository at this point in the history
### Rationale for this change This is an implementation of a node that can merge N sorted inputs (only in ascending order for a first pass). Where possible I have shared components with `asof_join_node.cc`. Full description/use case is described in #38381 ### What changes are included in this PR? * Take out relevant guts of asofjoin to stream data top to bottom/consume in a non blocking manner * Implement a sorted merger ### Are these changes tested? Basic test added. Locally I have tested this on 100+ gigabytes of parquet, sharded across 50+ files. Happy to add a benchmark test on top of the basic test, but submitting now for code feedback. ### Are there any user-facing changes? Yes, `sorted_merge` is now an exposed declaration Lead-authored-by: Jeremy Aguilon <[email protected]> Co-authored-by: jeremy <[email protected]> Co-authored-by: Weston Pace <[email protected]> Signed-off-by: Weston Pace <[email protected]>
- Loading branch information