Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make state spillable in partitioned writer [databricks] #8667

Merged
merged 38 commits into from
Jul 26, 2023
Merged

Make state spillable in partitioned writer [databricks] #8667

merged 38 commits into from
Jul 26, 2023

Commits on Jul 6, 2023

  1. Make state spillable in partitioned writer 

    Signed-off-by: Alessandro Bellina <[email protected]>
    @abellina
    abellina committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    13e3ab2 View commit details
    Browse the repository at this point in the history

  2. Fix import order

    @abellina
    abellina committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    e349864 View commit details
    Browse the repository at this point in the history

  3. Move withRestoreOnRetry outside of bufferBatchAndClose only on retry … 

    …path
    @abellina
    abellina committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    dce0f19 View commit details
    Browse the repository at this point in the history

  4. Remove extraneous withRetryNoSplit

    @abellina
    abellina committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    6eeb1a0 View commit details
    Browse the repository at this point in the history

  5. Address review comments in GpuFileFormatDataWriter

    @abellina
    abellina committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    7fd33d6 View commit details
    Browse the repository at this point in the history

  6. Add comment around the anythingWritten flag in ColumnarOutputWriter

    @abellina
    abellina committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    e6a739b View commit details
    Browse the repository at this point in the history

  7. Fix leak in my revisions

    @abellina
    abellina committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    f73790e View commit details
    Browse the repository at this point in the history

  8. Add withRetryNoSplit in GpuDeltaTaskStatisticsTracker.newBatch

    @abellina
    abellina committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    c056912 View commit details
    Browse the repository at this point in the history

  9. Fix import order

    @abellina
    abellina committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    9825536 View commit details
    Browse the repository at this point in the history

Commits on Jul 7, 2023

  1. Do not close spillable in GpuDeltaTaskStatisticsTracker

    @abellina
    abellina committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    a921bfd View commit details
    Browse the repository at this point in the history

  2. Pass template argument to withRetryNoSplit

    @abellina
    abellina committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    fbe0fb4 View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2023

  1. Upmerge

    @abellina
    abellina committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    e5fe2e6 View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2023

  1. Fix upmerge issues

    @abellina
    abellina committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    4922214 View commit details
    Browse the repository at this point in the history

  2. Add unit tests

    @abellina
    abellina committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    ab4633f View commit details
    Browse the repository at this point in the history

  3. Import order

    @abellina
    abellina committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    1420101 View commit details
    Browse the repository at this point in the history

  4. Make sure to pass TaskContext

    @abellina
    abellina committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    c989223 View commit details
    Browse the repository at this point in the history

  5. Remove debug statements

    @abellina
    abellina committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    57bd89d View commit details
    Browse the repository at this point in the history

  6. Remove more debug logic

    @abellina
    abellina committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    963a99a View commit details
    Browse the repository at this point in the history

Commits on Jul 18, 2023

  1. Address review comments

    @abellina
    abellina committed Jul 18, 2023
    Configuration menu
    Copy the full SHA
    9e3f5b3 View commit details
    Browse the repository at this point in the history

  2. Ensure newBatch is called inside closeOnExcept

    @abellina
    abellina committed Jul 18, 2023
    Configuration menu
    Copy the full SHA
    f9cc9ec View commit details
    Browse the repository at this point in the history

  3. Add scaladoc

    @abellina
    abellina committed Jul 18, 2023
    Configuration menu
    Copy the full SHA
    bef9335 View commit details
    Browse the repository at this point in the history

Commits on Jul 19, 2023

  1. Initialize RapidsBufferCatalog in FileCacheIntegrationSuite

    @abellina
    abellina committed Jul 19, 2023
    Configuration menu
    Copy the full SHA
    79ce06b View commit details
    Browse the repository at this point in the history

Commits on Jul 20, 2023

  1. Tweak method name and make sure writeSpillableAndClose doesnt call ne… 

    …wBatch more than once
    @abellina
    abellina committed Jul 20, 2023
    Configuration menu
    Copy the full SHA
    022e2d6 View commit details
    Browse the repository at this point in the history

Commits on Jul 21, 2023

  1. Adds a test and fix code around writeSpillableAndClose

    @abellina
    abellina committed Jul 21, 2023
    Configuration menu
    Copy the full SHA
    4fb645f View commit details
    Browse the repository at this point in the history

  2. Merge branch 'branch-23.08' of https://github.com/NVIDIA/spark-rapids … 

    …into spillable_splits_part_writer
    @abellina
    abellina committed Jul 21, 2023
    Configuration menu
    Copy the full SHA
    03d6105 View commit details
    Browse the repository at this point in the history

  3. Close spillable if we cant materialize the whole batch

    @abellina
    abellina committed Jul 21, 2023
    Configuration menu
    Copy the full SHA
    1ade1f6 View commit details
    Browse the repository at this point in the history

  4. Close batches here for now in the test

    @abellina
    abellina committed Jul 21, 2023
    Configuration menu
    Copy the full SHA
    3c903d3 View commit details
    Browse the repository at this point in the history

Commits on Jul 24, 2023

  1. We need to close batches in our mock to maintain expectations

    @abellina
    abellina committed Jul 24, 2023
    Configuration menu
    Copy the full SHA
    8c57ff2 View commit details
    Browse the repository at this point in the history

  2. Batches are now closed correctly from mock

    @abellina
    abellina committed Jul 24, 2023
    Configuration menu
    Copy the full SHA
    22bafa4 View commit details
    Browse the repository at this point in the history

  3. Make sure that we close existing sessions in SparkQueryCompreTestSuite

    @abellina
    abellina committed Jul 24, 2023
    Configuration menu
    Copy the full SHA
    8fe1d18 View commit details
    Browse the repository at this point in the history

  4. Handling of session cleanup is happening at superclass

    @abellina
    abellina committed Jul 24, 2023
    Configuration menu
    Copy the full SHA
    f556fbc View commit details
    Browse the repository at this point in the history

  5. Merge branch 'branch-23.08' of https://github.com/NVIDIA/spark-rapids … 

    …into spillable_splits_part_writer
    @abellina
    abellina committed Jul 24, 2023
    Configuration menu
    Copy the full SHA
    fd96e54 View commit details
    Browse the repository at this point in the history

Commits on Jul 25, 2023

  1. Fixes unit test issues where the catalog/semaphore were being left in… 

    …itialized
    @abellina
    abellina committed Jul 25, 2023
    Configuration menu
    Copy the full SHA
    df425ce View commit details
    Browse the repository at this point in the history

  2. Remove extra line

    @abellina
    abellina committed Jul 25, 2023
    Configuration menu
    Copy the full SHA
    f746816 View commit details
    Browse the repository at this point in the history

  3. Unused imports

    @abellina
    abellina committed Jul 25, 2023
    Configuration menu
    Copy the full SHA
    3e0445b View commit details
    Browse the repository at this point in the history

  4. Fix issues with DeviceMemoryEventHandlerSuite

    @abellina
    abellina committed Jul 25, 2023
    Configuration menu
    Copy the full SHA
    87f22cc View commit details
    Browse the repository at this point in the history

  5. Use RmmSparkRetrySuiteBase to reset rmm event handlers

    @abellina
    abellina committed Jul 25, 2023
    Configuration menu
    Copy the full SHA
    54c83b3 View commit details
    Browse the repository at this point in the history

Commits on Jul 26, 2023

  1. Apply code review comments

    @abellina
    abellina committed Jul 26, 2023
    Configuration menu
    Copy the full SHA
    a08e5a5 View commit details
    Browse the repository at this point in the history