[C++] Support hash-join on larger than memory datasets

[asfimport]

The current implementation of the hash-join node current queues in memory the hashtable, the entire build side input, and the entire probe side input (e.g. the entire dataset). This means the current implementation will run out of memory and crash if the input dataset is larger than the memory on the system.

By spilling to disk when memory starts to fill up we can allow the hash-join node to process datasets larger than the available memory on the machine.

Reporter: Weston Pace / @westonpace

Related issues:

• [C++] Naive spillover implementation for join (supercedes)

PRs and other links:

• GitHub Pull Request #13669

_{Note: This issue was originally created as ARROW-16389. Please see the migration documentation for further details.}

[asfimport]

Weston Pace / @westonpace:
Attaching a design for this created by @michalursa: https://docs.google.com/document/d/1hPxmfJYmWne_OyX3-CqXXyvDBoimi00M0WbS6bXcGYw/edit?usp=sharing

[asfimport]

Apache Arrow JIRA Bot:
This issue was last updated over 90 days ago, which may be an indication it is no longer being actively worked. To better reflect the current state, the issue is being unassigned per project policy. Please feel free to re-take assignment of the issue if it is being actively worked, or if you plan to start that work soon.

[asfimport]

Zane Wilbert Keeler:
dr jim
data sets on proper hdwe sfwr for your info.

z

On Thu, Jul 21, 2022 at 1:04 AM ASF GitHub Bot (Jira) [email protected]

[asfimport]

Zane Wilbert Keeler:
dr jim
changing server
hdwe update.
z