Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Streaming Aggregation: Do not break pipeline in aggregation if group by columns are ordered #6034

Closed
wants to merge 42 commits into from
Closed

Implement Streaming Aggregation: Do not break pipeline in aggregation if group by columns are ordered #6034

wants to merge 42 commits into from

Commits on Mar 22, 2023

  1. add starting code for experimenting

    @mustafasrepo
    mustafasrepo committed Mar 22, 2023
    Configuration menu
    Copy the full SHA
    6c1dfeb View commit details
    Browse the repository at this point in the history

Commits on Mar 28, 2023

  1. stream group by linear implementation

    @mustafasrepo
    mustafasrepo committed Mar 28, 2023
    Configuration menu
    Copy the full SHA
    1d8e6f5 View commit details
    Browse the repository at this point in the history

Commits on Mar 29, 2023

  1. sorted implementation

    @mustafasrepo
    mustafasrepo committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    e35703b View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into feature/stream_groupby

    @mustafasrepo
    mustafasrepo committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    7057106 View commit details
    Browse the repository at this point in the history

Commits on Mar 30, 2023

  1. minor changes

    @mustafasrepo
    mustafasrepo committed Mar 30, 2023
    Configuration menu
    Copy the full SHA
    16c52f8 View commit details
    Browse the repository at this point in the history

Commits on Mar 31, 2023

  1. Merge branch 'main' into feature/stream_groupby 

    # Conflicts:
#	datafusion/core/src/physical_optimizer/repartition.rs
#	datafusion/core/src/physical_plan/joins/sort_merge_join.rs
#	datafusion/core/src/physical_plan/joins/symmetric_hash_join.rs
#	datafusion/core/src/physical_plan/mod.rs
#	datafusion/core/src/physical_plan/sorts/sort_preserving_merge.rs
#	datafusion/core/src/physical_plan/windows/bounded_window_agg_exec.rs
#	datafusion/core/src/physical_plan/windows/window_agg_exec.rs
#	datafusion/physical-expr/src/utils.rs
    @mustafasrepo
    mustafasrepo committed Mar 31, 2023
    Configuration menu
    Copy the full SHA
    f67313d View commit details
    Browse the repository at this point in the history

  2. simplifications

    @mustafasrepo
    mustafasrepo committed Mar 31, 2023
    Configuration menu
    Copy the full SHA
    48c8085 View commit details
    Browse the repository at this point in the history

  3. Simplifications

    @mustafasrepo
    mustafasrepo committed Mar 31, 2023
    Configuration menu
    Copy the full SHA
    da7b2c6 View commit details
    Browse the repository at this point in the history

Commits on Apr 3, 2023

  1. convert vec to Option

    @mustafasrepo
    mustafasrepo committed Apr 3, 2023
    Configuration menu
    Copy the full SHA
    ab93bf3 View commit details
    Browse the repository at this point in the history

Commits on Apr 4, 2023

  1. minor changes

    @mustafasrepo
    mustafasrepo committed Apr 4, 2023
    Configuration menu
    Copy the full SHA
    6134751 View commit details
    Browse the repository at this point in the history

  2. minor changes

    @mustafasrepo
    mustafasrepo committed Apr 4, 2023
    Configuration menu
    Copy the full SHA
    2cf0180 View commit details
    Browse the repository at this point in the history

  3. minor changes

    @mustafasrepo
    mustafasrepo committed Apr 4, 2023
    Configuration menu
    Copy the full SHA
    2802685 View commit details
    Browse the repository at this point in the history

Commits on Apr 5, 2023

  1. simplifications

    @mustafasrepo
    mustafasrepo committed Apr 5, 2023
    Configuration menu
    Copy the full SHA
    786caef View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into feature/stream_groupby 

    # Conflicts:
#	datafusion/core/src/test_util/mod.rs
#	datafusion/physical-expr/src/utils.rs
    @mustafasrepo
    mustafasrepo committed Apr 5, 2023
    Configuration menu
    Copy the full SHA
    f04bd05 View commit details
    Browse the repository at this point in the history

  3. minor changes

    @mustafasrepo
    mustafasrepo committed Apr 5, 2023
    Configuration menu
    Copy the full SHA
    a9f78cb View commit details
    Browse the repository at this point in the history

Commits on Apr 6, 2023

  1. all tests pass

    @mustafasrepo
    mustafasrepo committed Apr 6, 2023
    Configuration menu
    Copy the full SHA
    a9f6d93 View commit details
    Browse the repository at this point in the history

Commits on Apr 7, 2023

  1. refactor

    @mustafasrepo
    mustafasrepo committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    4f49e55 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into feature/stream_groupby 

    # Conflicts:
#	datafusion/common/src/utils.rs
    @mustafasrepo
    mustafasrepo committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    0a0b496 View commit details
    Browse the repository at this point in the history

  3. simplifications

    @mustafasrepo
    mustafasrepo committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    ae29248 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into feature/output_order_vec 

    # Conflicts:
#	datafusion/core/src/physical_optimizer/repartition.rs
#	datafusion/core/src/physical_optimizer/sort_pushdown.rs
    @mustafasrepo
    mustafasrepo committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    8828aec View commit details
    Browse the repository at this point in the history

Commits on Apr 10, 2023

  1. Merge branch 'feature/output_order_vec' into feature/stream_groupby 

    # Conflicts:
#	datafusion/core/src/physical_plan/aggregates/mod.rs
    @mustafasrepo
    mustafasrepo committed Apr 10, 2023
    Configuration menu
    Copy the full SHA
    45a0aab View commit details
    Browse the repository at this point in the history

  2. remove unnecessary code

    @mustafasrepo
    mustafasrepo committed Apr 10, 2023
    Configuration menu
    Copy the full SHA
    c1872f6 View commit details
    Browse the repository at this point in the history

  3. simplifications

    @mustafasrepo
    mustafasrepo committed Apr 10, 2023
    Configuration menu
    Copy the full SHA
    b4c25ff View commit details
    Browse the repository at this point in the history

Commits on Apr 11, 2023

  1. Merge branch 'main' into feature/stream_groupby 

    # Conflicts:
#	datafusion/core/src/physical_plan/aggregates/row_hash.rs
    @mustafasrepo
    mustafasrepo committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    c6730c0 View commit details
    Browse the repository at this point in the history

Commits on Apr 12, 2023

  1. Merge branch 'main' into feature/stream_groupby 

    # Conflicts:
#	datafusion/core/src/physical_plan/aggregates/mod.rs
#	datafusion/core/src/physical_plan/aggregates/row_hash.rs
#	datafusion/core/src/test_util/mod.rs
#	datafusion/core/tests/sql/window.rs
    @mustafasrepo
    mustafasrepo committed Apr 12, 2023
    Configuration menu
    Copy the full SHA
    e321082 View commit details
    Browse the repository at this point in the history

  2. minor changes

    @mustafasrepo
    mustafasrepo committed Apr 12, 2023
    Configuration menu
    Copy the full SHA
    bb55f50 View commit details
    Browse the repository at this point in the history

  3. simplifications

    @mustafasrepo
    mustafasrepo committed Apr 12, 2023
    Configuration menu
    Copy the full SHA
    2eab0d0 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into feature/stream_groupby 

    # Conflicts:
#	datafusion/core/tests/sqllogictests/test_files/window.slt
    @mustafasrepo
    mustafasrepo committed Apr 12, 2023
    Configuration menu
    Copy the full SHA
    cfc86e4 View commit details
    Browse the repository at this point in the history

Commits on Apr 14, 2023

  1. Merge branch 'main' into feature/stream_groupby 

    # Conflicts:
#	datafusion/common/src/utils.rs
#	datafusion/core/src/physical_optimizer/sort_enforcement.rs
#	datafusion/core/src/physical_plan/streaming.rs
#	datafusion/core/src/physical_plan/windows/bounded_window_agg_exec.rs
#	datafusion/core/src/physical_plan/windows/window_agg_exec.rs
#	datafusion/physical-expr/src/utils.rs
    @mustafasrepo
    mustafasrepo committed Apr 14, 2023
    Configuration menu
    Copy the full SHA
    2fc47a8 View commit details
    Browse the repository at this point in the history

  2. minor changes

    @mustafasrepo
    mustafasrepo committed Apr 14, 2023
    Configuration menu
    Copy the full SHA
    0932f52 View commit details
    Browse the repository at this point in the history

Commits on Apr 15, 2023

  1. Simplify the GroupByOrderMode type

    @ozankabak
    ozankabak committed Apr 15, 2023
    Configuration menu
    Copy the full SHA
    4083422 View commit details
    Browse the repository at this point in the history

Commits on Apr 17, 2023

  1. Address reviews

    @mustafasrepo
    mustafasrepo committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    c17186a View commit details
    Browse the repository at this point in the history

  2. separate fully ordered case and remaining cases

    @mustafasrepo
    mustafasrepo committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    01dd18b View commit details
    Browse the repository at this point in the history

Commits on Apr 20, 2023

  1. Merge branch 'main' into feature/stream_groupby4 

    # Conflicts:
#	datafusion/core/src/physical_plan/aggregates/row_hash.rs
    @mustafasrepo
    mustafasrepo committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    e4f4347 View commit details
    Browse the repository at this point in the history

Commits on Apr 21, 2023

  1. change test data type

    @mustafasrepo
    mustafasrepo committed Apr 21, 2023
    Configuration menu
    Copy the full SHA
    479bc0c View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into feature/stream_groupby4 

    # Conflicts:
#	datafusion/core/src/physical_plan/union.rs
    @mustafasrepo
    mustafasrepo committed Apr 21, 2023
    Configuration menu
    Copy the full SHA
    19f82da View commit details
    Browse the repository at this point in the history

Commits on Apr 24, 2023

  1. address reviews

    @mustafasrepo
    mustafasrepo committed Apr 24, 2023
    Configuration menu
    Copy the full SHA
    6e70583 View commit details
    Browse the repository at this point in the history

Commits on Apr 25, 2023

  1. Convert to option

    @mustafasrepo
    mustafasrepo committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    e13742c View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into feature/stream_groupby4 

    # Conflicts:
#	datafusion/core/src/physical_plan/windows/bounded_window_agg_exec.rs
#	datafusion/core/tests/sql/group_by.rs
#	datafusion/core/tests/sqllogictests/test_files/window.slt
    @mustafasrepo
    mustafasrepo committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    feb9117 View commit details
    Browse the repository at this point in the history

  3. retract back to old API.

    @mustafasrepo
    mustafasrepo committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    0de426c View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into feature/stream_groupby4 

    # Conflicts:
#	datafusion/core/src/physical_plan/aggregates/mod.rs
    @mustafasrepo
    mustafasrepo committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    4a07c10 View commit details
    Browse the repository at this point in the history

  5. Code quality: stylistic changes

    @ozankabak
    ozankabak committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    70a13f4 View commit details
    Browse the repository at this point in the history