Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support "WindowGroupLimit" optimization on GPU [databricks] #10500

Merged
merged 19 commits into from
Feb 29, 2024
Merged

Support "WindowGroupLimit" optimization on GPU [databricks] #10500

merged 19 commits into from
Feb 29, 2024

Commits on Feb 26, 2024

  1. Window group limit: Initial stab. 

    1. Rough idea of algo.
2. Not bothering with the 3.5 shim now.
3. NPE when running BasicWindowCalc.
    @mythrocks
    mythrocks committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    11814c3 View commit details
    Browse the repository at this point in the history

  2. Working with grouped window functions, RANK and DENSE_RANK.

    @mythrocks
    mythrocks committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    bc6e197 View commit details
    Browse the repository at this point in the history

  3. Support for un-partitioned queries.

    @mythrocks
    mythrocks committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    f7a3010 View commit details
    Browse the repository at this point in the history

  4. Now withRetry.

    @mythrocks
    mythrocks committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    9369006 View commit details
    Browse the repository at this point in the history

  5. With metrics.

    @mythrocks
    mythrocks committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    d11945d View commit details
    Browse the repository at this point in the history

  6. Moved to shims.

    @mythrocks
    mythrocks committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    bda6105 View commit details
    Browse the repository at this point in the history

  7. Widened input datatype scopes.

    @mythrocks
    mythrocks committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    e8099eb View commit details
    Browse the repository at this point in the history

  8. Add GPU Conversion for ranking function. 

    Without this, the runtime assumes that the ranking function is a CPU function,
even though the GpuWindowGroupLimitExec is on the GPU.

Signed-off-by: MithunR <[email protected]>
    @mythrocks
    mythrocks committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    421b3fc View commit details
    Browse the repository at this point in the history

  9. Added test for partitioned group limits.

    @mythrocks
    mythrocks committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    9069b3f View commit details
    Browse the repository at this point in the history

  10. Tests for rank, dense_rank, with partitioned/unpartitioned window cla… 

    …uses.
    @mythrocks
    mythrocks committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    3105f03 View commit details
    Browse the repository at this point in the history

  11. Skip test for Spark < 3.5.

    @mythrocks
    mythrocks committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    509b57b View commit details
    Browse the repository at this point in the history

  12. Updated to supportedExecs, etc.

    @mythrocks
    mythrocks committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    6b1842d View commit details
    Browse the repository at this point in the history

Commits on Feb 27, 2024

  1. Fixed skipif, test name.

    @mythrocks
    mythrocks committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    06e165e View commit details
    Browse the repository at this point in the history

  2. Test for CPU fallback for ROW_NUMBER.

    @mythrocks
    mythrocks committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    e5994f1 View commit details
    Browse the repository at this point in the history

  3. Verify that limit-mode is either Partial or Final .

    @mythrocks
    mythrocks committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    a956832 View commit details
    Browse the repository at this point in the history

  4. Removed code marker.

    @mythrocks
    mythrocks committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    96aeb74 View commit details
    Browse the repository at this point in the history

  5. Fixed formatting.

    @mythrocks
    mythrocks committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    12382bb View commit details
    Browse the repository at this point in the history

  6. Documented the approach in the code.

    @mythrocks
    mythrocks committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    02e23d4 View commit details
    Browse the repository at this point in the history

  7. More formatting .

    @mythrocks
    mythrocks committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    bd04495 View commit details
    Browse the repository at this point in the history