You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
In many cases a task is going to have a single batch to process. When we sort right now, we will sort the input batch, slice it into smaller batches, and then start to merge those batches together. But most of that ends up being a noop if there is only a single batch to sort.
I would like to propose that we modify GpuOutOfCoreSortIterator so that it takes the first batch and just makes it spillable. It then asks for a second batch. Only if it gets a second batch does it go through the complicated sort/merge process. If there is only one batch we sort it and go on. It is a little more complicated than that because of retry and splits with the retry, but I think it is totally doable.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
In many cases a task is going to have a single batch to process. When we sort right now, we will sort the input batch, slice it into smaller batches, and then start to merge those batches together. But most of that ends up being a noop if there is only a single batch to sort.
I would like to propose that we modify GpuOutOfCoreSortIterator so that it takes the first batch and just makes it spillable. It then asks for a second batch. Only if it gets a second batch does it go through the complicated sort/merge process. If there is only one batch we sort it and go on. It is a little more complicated than that because of retry and splits with the retry, but I think it is totally doable.
The text was updated successfully, but these errors were encountered: