Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test: Port (last) repartition.rs query to sqllogictest #8936

Merged
merged 4 commits into from
Jan 25, 2024
Merged

test: Port (last) repartition.rs query to sqllogictest #8936

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
6492d28
Migrate last repartition query
simicd Jan 21, 2024
1088f7f
Add reference to issue
simicd Jan 21, 2024
4814d95
Fix missing statement
simicd Jan 21, 2024
614a4e5
Merge remote-tracking branch 'apache/main' into test/repartition-sqll…
alamb Jan 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion datafusion/core/tests/sql/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,6 @@ pub mod create_drop;
pub mod explain_analyze;
pub mod expr;
pub mod joins;
pub mod repartition;
pub mod select;
mod sql_api;

Expand Down
59 changes: 0 additions & 59 deletions datafusion/core/tests/sql/repartition.rs
View file
Open in desktop

This file was deleted.

56 changes: 56 additions & 0 deletions datafusion/sqllogictest/test_files/repartition.slt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,3 +71,59 @@ AggregateExec: mode=FinalPartitioned, gby=[column1@0 as column1], aggr=[SUM(parq
# Cleanup
statement ok
DROP TABLE parquet_table;



# Unbounded repartition
simicd marked this conversation as resolved.
Show resolved Hide resolved
# See https://github.com/apache/arrow-datafusion/issues/5278
# Set up unbounded table and run a query - the query plan should display a `RepartitionExec`
# and a `CoalescePartitionsExec`
statement ok
CREATE UNBOUNDED EXTERNAL TABLE sink_table (
c1 VARCHAR NOT NULL,
c2 TINYINT NOT NULL,
c3 SMALLINT NOT NULL,
c4 SMALLINT NOT NULL,
c5 INTEGER NOT NULL,
c6 BIGINT NOT NULL,
c7 SMALLINT NOT NULL,
c8 INT NOT NULL,
c9 INT UNSIGNED NOT NULL,
c10 BIGINT UNSIGNED NOT NULL,
c11 FLOAT NOT NULL,
c12 DOUBLE NOT NULL,
c13 VARCHAR NOT NULL
)
STORED AS CSV
WITH HEADER ROW
LOCATION '../../testing/data/csv/aggregate_test_100.csv';

query TII
SELECT c1, c2, c3 FROM sink_table WHERE c3 > 0 LIMIT 5;
----
c 2 1
b 1 29
e 3 104
a 3 13
d 1 38

statement ok
set datafusion.execution.target_partitions = 3;

statement ok
set datafusion.optimizer.enable_round_robin_repartition = true;

query TT
EXPLAIN SELECT c1, c2, c3 FROM sink_table WHERE c3 > 0 LIMIT 5;
----
logical_plan
Limit: skip=0, fetch=5
--Filter: sink_table.c3 > Int16(0)
----TableScan: sink_table projection=[c1, c2, c3]
physical_plan
GlobalLimitExec: skip=0, fetch=5
--CoalescePartitionsExec
----CoalesceBatchesExec: target_batch_size=8192
------FilterExec: c3@2 > 0
--------RepartitionExec: partitioning=RoundRobinBatch(3), input_partitions=1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This partitioning is round robin partitioning where the original test uses hash partitioning. I am not sure if that is equivalent.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see! Thanks for looking into it

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think there will be a difference. It's hard to reproduce the hash partitioning with unbounded tables. However, this would work well enough.

----------StreamingTableExec: partition_sizes=1, projection=[c1, c2, c3], infinite_source=true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Loading