Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
apacheGH-34474: [C++] Detect and raise an error if a join will need t…
…oo much key data (apache#35087) ### Rationale for this change This fixes the test in apache#34474 though there are likely still other bad scenarios with large joins. I've fixed this one since the behavior (invalid data) is particularly bad. Most of the time if there is too much data I'm guessing we probably just crash. Still, I think a test suite of some kind stressing large joins would be good to have. Perhaps this could be added if someone finds time to work on join spilling. ### What changes are included in this PR? If the join will require more than 4GiB of key data it should now return an invalid status instead of invalid data. ### Are these changes tested? No. I created a unit test but it requires over 16GiB of RAM (Besides the input data itself (4GiB), by the time you get 4GiB of key data there are various other join state buffers that also grow. The test also took nearly a minute to run. I think investigation and creation of a test suite for large joins is probably a standalone effort. ### Are there any user-facing changes? No. * Closes: apache#34474 Authored-by: Weston Pace <[email protected]> Signed-off-by: Joris Van den Bossche <[email protected]>
- Loading branch information