Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Union in decoupled mode #15870

Merged
merged 81 commits into from
Feb 21, 2024
Merged
Show file tree
Hide file tree
Changes from 79 commits
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
ecd7099
Support Window operators in decoupled planning
kgyrtkirk Jan 31, 2024
37d11f3
no more need for workaround that `getTrait(T)` sometimes returns RelC…
kgyrtkirk Jan 31, 2024
ce3bd03
lets not do this now
kgyrtkirk Jan 31, 2024
8112d1d
describe stuff
kgyrtkirk Jan 31, 2024
5a1b740
add
kgyrtkirk Feb 1, 2024
274fbfa
add test to remove modes
kgyrtkirk Feb 1, 2024
443058b
some stuff
kgyrtkirk Feb 1, 2024
0ba43ac
add fixme
kgyrtkirk Feb 1, 2024
0c132df
some unrelated
kgyrtkirk Feb 5, 2024
aec32da
Revert "some unrelated"
kgyrtkirk Feb 5, 2024
d1b2dad
some stuff
kgyrtkirk Feb 5, 2024
2dd63ab
a different approach
kgyrtkirk Feb 5, 2024
a8e04db
remove stuff
kgyrtkirk Feb 5, 2024
7657e7d
cleanup
kgyrtkirk Feb 5, 2024
9775c1a
temp-crap
kgyrtkirk Feb 6, 2024
6e6c752
talk
kgyrtkirk Feb 6, 2024
407d4f3
Revert "talk"
kgyrtkirk Feb 6, 2024
c9dbe26
cleanup
kgyrtkirk Feb 6, 2024
ef01196
cleanup
kgyrtkirk Feb 6, 2024
1ebe421
fix/etc union
kgyrtkirk Feb 6, 2024
a83dd8d
updates
kgyrtkirk Feb 6, 2024
7a94653
cleanup
kgyrtkirk Feb 6, 2024
4f78e85
move org.reflections to depManagement
kgyrtkirk Feb 6, 2024
6e9e5ee
checkstyle
kgyrtkirk Feb 6, 2024
3bc9326
cleanup
kgyrtkirk Feb 6, 2024
7ff59fc
merge 2 enum entries
kgyrtkirk Feb 7, 2024
fd858b5
Merge remote-tracking branch 'apache/master' into decouple-support-wi…
kgyrtkirk Feb 7, 2024
df615e0
Merge branch 'decouple-support-window2' into decouple-support-union
kgyrtkirk Feb 7, 2024
cab8b8f
xinputprod
kgyrtkirk Feb 7, 2024
646559e
partially undo
kgyrtkirk Feb 7, 2024
aea1021
add interface
kgyrtkirk Feb 7, 2024
11c1d30
PDQVertexFactoy
kgyrtkirk Feb 7, 2024
04e6712
move pdqfactory to new file
kgyrtkirk Feb 7, 2024
8d363b2
move class
kgyrtkirk Feb 7, 2024
58d5344
rename/etc
kgyrtkirk Feb 7, 2024
9d2a061
extract common
kgyrtkirk Feb 7, 2024
2cce30e
move stuff around
kgyrtkirk Feb 7, 2024
84326a2
some stuff
kgyrtkirk Feb 7, 2024
56dd715
remove project field
kgyrtkirk Feb 7, 2024
2522d50
set druidTable for now
kgyrtkirk Feb 7, 2024
15c9e79
fix currentTable for scan
kgyrtkirk Feb 7, 2024
e333267
values
kgyrtkirk Feb 7, 2024
e9198b7
remove
kgyrtkirk Feb 7, 2024
40c02e2
cleanup
kgyrtkirk Feb 7, 2024
712191a
remove cTSCBV
kgyrtkirk Feb 7, 2024
e240bbf
format/etc
kgyrtkirk Feb 7, 2024
08235d9
format
kgyrtkirk Feb 8, 2024
19d455d
cleanup
kgyrtkirk Feb 8, 2024
2a723f1
rename
kgyrtkirk Feb 8, 2024
d50a491
rename/etc
kgyrtkirk Feb 8, 2024
03daf53
cleanup
kgyrtkirk Feb 8, 2024
71a2d18
make vertex/pdqvertexfactory an inner class
kgyrtkirk Feb 8, 2024
7ac5327
retire some createVertex methods
kgyrtkirk Feb 8, 2024
1ba7958
supller/etc
kgyrtkirk Feb 8, 2024
a9b9cf8
undo supplier
kgyrtkirk Feb 8, 2024
7694d6f
order interface methods
kgyrtkirk Feb 8, 2024
6cb2846
cleanup
kgyrtkirk Feb 8, 2024
00264db
refactor
kgyrtkirk Feb 8, 2024
c33122f
format
kgyrtkirk Feb 8, 2024
46c6299
cleanup
kgyrtkirk Feb 8, 2024
587de39
wiggle
kgyrtkirk Feb 8, 2024
9177573
cleanup
kgyrtkirk Feb 8, 2024
9690ae3
cleanup
kgyrtkirk Feb 8, 2024
6f678b4
cleanup
kgyrtkirk Feb 8, 2024
a18726b
add some apidoc
kgyrtkirk Feb 8, 2024
797e806
raise class security a bit
kgyrtkirk Feb 8, 2024
7bf90e5
fix err
kgyrtkirk Feb 9, 2024
b0d5f9e
fix bug
kgyrtkirk Feb 9, 2024
63630e3
remove debug on by default
kgyrtkirk Feb 9, 2024
4600378
remove processUnion method
kgyrtkirk Feb 9, 2024
84048a2
cleanup
kgyrtkirk Feb 14, 2024
c0307fa
add test
kgyrtkirk Feb 14, 2024
e3e9379
enable test for CalciteUnionQueryTest
kgyrtkirk Feb 14, 2024
87c345b
rename method
kgyrtkirk Feb 19, 2024
87f801c
change message
kgyrtkirk Feb 19, 2024
a7e04e2
update
kgyrtkirk Feb 20, 2024
7770e37
cleanup
kgyrtkirk Feb 20, 2024
3d5dcd7
fix
kgyrtkirk Feb 20, 2024
a7a1358
fix java8
kgyrtkirk Feb 20, 2024
65f2d1a
tries with reflections
kgyrtkirk Feb 21, 2024
98adcd0
restrict to org.apache.druid.sql
kgyrtkirk Feb 21, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -1262,6 +1262,12 @@
<version>${hamcrest.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.reflections</groupId>
<artifactId>reflections</artifactId>
<version>0.9.12</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>pl.pragmatists</groupId>
<artifactId>JUnitParams</artifactId>
Expand Down
1 change: 0 additions & 1 deletion processing/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -380,7 +380,6 @@
<dependency>
<groupId>org.reflections</groupId>
<artifactId>reflections</artifactId>
<version>0.9.12</version>
<scope>test</scope>
</dependency>
<dependency>
Expand Down
5 changes: 5 additions & 0 deletions sql/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -262,6 +262,11 @@
<artifactId>jdbi</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.reflections</groupId>
<artifactId>reflections</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

<build>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -248,20 +248,21 @@ public List<Program> programs(final PlannerContext plannerContext)
),
Programs.sequence(
druidPreProgram,
buildBaseRuleSetProgram(plannerContext),
new LoggingProgram("After baseRuleSet program", isDebug),
buildDecoupledLogicalOptimizationProgram(plannerContext),
new LoggingProgram("After DecoupledLogicalOptimizationProgram program", isDebug),
Programs.ofRules(logicalConventionRuleSet(plannerContext)),
new LoggingProgram("After logical volcano planner program", isDebug)
)
);
}

private Program buildBaseRuleSetProgram(PlannerContext plannerContext)
private Program buildDecoupledLogicalOptimizationProgram(PlannerContext plannerContext)
{
final HepProgramBuilder builder = HepProgram.builder();
builder.addMatchLimit(CalciteRulesManager.HEP_DEFAULT_MATCH_LIMIT);
builder.addGroupBegin();
builder.addRuleCollection(baseRuleSet(plannerContext));
builder.addRuleInstance(CoreRules.UNION_MERGE);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel the name of the method is no longer accurate (it's going beyond baseRuleSet). It appears this is only used in decoupled mode-- is that correct? If so some comments about that, or reflecting it in the name, would be useful.

Haven't read the rest of the patch as yet, so this is just a single comment rather than a review.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to separate basic optimiazation rules from the conversion rules - it will be usefull to be able to enable/tweak rules without altering the other planning rules.

It kinda already made one things easier: I was able to just throw in UNION_MERGE - without having to worry about disturbing existing plan differences.

there will be some more deeper tweaking will be needed as AGGREGATE_REMOVE and AGGREGATE_CASE_TO_FILTER doesn't play nicely together - which induces some plan differences

I can rename it to buildDecoupledLogicalOptimizationProgram - does that sound better?

builder.addGroupEnd();
return Programs.of(builder.build(), true, DefaultRelMetadataProvider.INSTANCE);
}
Expand Down
Loading
Loading