Fix Joining on ROWKEY #2735
Merged
Fix Joining on ROWKEY #2735
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…now have the correct keyfield - unnecessary repartition steps of streams now avoided in new queries, but maintained in existing queries.
agavra
approved these changes
Apr 25, 2019
There was a problem hiding this comment.
Really nice change @big-andy-coates. Took me a while to get my head wrapped around exactly what was changing despite it being, in essence, a pretty simple change... The data flow has such high complexity around this part of the code I wish there was a simpler way to model it!
ksql-engine/src/main/java/io/confluent/ksql/structured/SchemaKStream.java
Show resolved
Hide resolved
ksql-engine/src/main/java/io/confluent/ksql/planner/plan/JoinNode.java
Outdated
Show resolved
Hide resolved
ksql-engine/src/main/java/io/confluent/ksql/planner/plan/JoinNode.java
Outdated
Show resolved
Hide resolved
ksql-engine/src/main/java/io/confluent/ksql/structured/SchemaKStream.java
Show resolved
Hide resolved
ksql-engine/src/main/java/io/confluent/ksql/structured/SchemaKStream.java
Outdated
Show resolved
Hide resolved
ksql-engine/src/main/java/io/confluent/ksql/planner/plan/JoinNode.java
Outdated
Show resolved
Hide resolved
ksql-engine/src/test/java/io/confluent/ksql/physical/PhysicalPlanBuilderTest.java
Show resolved
Hide resolved
ksql-engine/src/test/java/io/confluent/ksql/physical/PhysicalPlanBuilderTest.java
Show resolved
Hide resolved
…rgs, as this is what they are. They are not KeyFields. Clean up some naming to make the code easier to understand.
This was referenced Apr 29, 2019
hjafarpour
approved these changes
Apr 29, 2019
| @@ -104,12 +107,38 @@ | |||
| private static final String FILTER_MAPVALUES_NODE = "KSTREAM-MAPVALUES-0000000004"; | |||
| private static final String FOREACH_NODE = "KSTREAM-FOREACH-0000000005"; | |||
|
|
|||
| private static final String createStream = "CREATE STREAM TEST1 (COL0 BIGINT, COL1 VARCHAR, COL2 DOUBLE) WITH ( " | |||
| + "KAFKA_TOPIC = 'test1', VALUE_FORMAT = 'JSON' );"; | |||
| private static final String CREATE_STREAM_TEST1 = "CREATE STREAM TEST1 " | |||
There was a problem hiding this comment.
For better readability you could include stream/table in the name. For instance, Create Stream Stream_test1 ...
There was a problem hiding this comment.
That's what I've already done...
Constant CREATE_STREAM_TEST1 CREATEs a STEAM called TEST1...
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Fixes #2636
Fixes #2525
Fixes #2470
This continues the work to clean up key field handling in the existing code.
This PR addresses issues when joining by ROWKEY.
emptyon the source added to the metastore when joining on ROWKEY. This is also now resolved.Review notes:
joins.jsonto see the tests that have been fixed and added.Analyzer, which now builds theKeyFields that theJoinNodeneeds.JoinNodefor the wiring. Note:getKeyFieldnow returnsleftKeyField, which may feel wrong, but this is refactored existing functionality. I'll be looking into this more soon, as there are issues around key fields where only the right join key is in the projection.SchemaKStream, specificallyselectKeyas this is where the magic happens. This will, if using new key fields i.e. if a new query, early out and not repartition is selecting rowkey. It still repartitions if using legacy key fields. This method also builds a new key field, replacing ROWKEY in the new key, and uses this to create newSchemaKStream, which fixes theemptykey issue.PhysicalPlanBuilderTestas this adds functional tests around the different combinations on join keys.Testing done
Extensive JSON and Unit tests added.
Reviewer checklist