Extend the PARTITIONED BY clause to accept string literals for the time partitioning #15836

zachjsh · 2024-02-06T00:15:01Z

Description

This PR contains a portion of the changes from the inactive draft PR for integrating the catalog with the Calcite planner #13686 from @paul-rogers, extending the PARTITION BY clause to accept string literals for the time partitioning

This PR has:

…me partitioning

sql/src/main/java/org/apache/druid/sql/calcite/parser/DruidSqlInsert.java

sql/src/main/java/org/apache/druid/sql/calcite/parser/DruidSqlParserUtils.java

sql/src/test/java/org/apache/druid/sql/calcite/parser/DruidSqlParserUtilsTest.java

sql/src/main/codegen/includes/common.ftl

sql/src/main/java/org/apache/druid/sql/calcite/parser/DruidSqlParserUtils.java

sql/src/main/java/org/apache/druid/sql/calcite/parser/DruidSqlIngest.java

sql/src/main/java/org/apache/druid/sql/calcite/parser/DruidSqlReplace.java

sql/src/main/java/org/apache/druid/sql/calcite/parser/DruidSqlInsert.java

sql/src/main/java/org/apache/druid/sql/calcite/parser/DruidSqlParserUtils.java

kgyrtkirk · 2024-02-06T10:16:50Z

sql/src/main/codegen/includes/common.ftl

      }
    ]
  |
    e = Expression(ExprContext.ACCEPT_SUB_QUERY)
    {
-      granularity = DruidSqlParserUtils.convertSqlNodeToGranularityThrowingParseExceptions(e);


I think some validation should happen here; I don't see a nice way to do it but I've found this:

result = new SqlLiteral(DruidSqlParserUtils.convertSqlNodeToGranularityThrowingParseExceptions(e), SYMBOL, getPos());

which might work (but the typeName must be supplied - other candidate could be: UNKNOWN )

Added validation here in another way, let me know if ok.

ok - but why not do the conversion here - I think you might also able to do similar thing on all the other branches

result = new SqlLiteral(DruidSqlParserUtils.convertSqlNodeToGranularityThrowingParseExceptions(e), SYMBOL, getPos());

you could shortcut the transient String + Symbol stuff

Added a new SqlNode type, let me know if good now.

sql/src/main/java/org/apache/druid/sql/calcite/parser/DruidSqlParserUtils.java

docs/multi-stage-query/reference.md

sql/src/main/codegen/includes/common.ftl

sql/src/main/java/org/apache/druid/sql/calcite/parser/DruidSqlParserUtils.java

sql/src/main/java/org/apache/druid/sql/calcite/parser/DruidSqlReplace.java

sql/src/test/java/org/apache/druid/sql/calcite/parser/DruidSqlParserUtilsTest.java

sql/src/test/java/org/apache/druid/sql/calcite/parser/DruidSqlUnparseTest.java

…s-string-literals

abhishekrb19

Had a few questions and suggestions. Also, to future proof this new functionality, I would recommend adding tests in different parts of the code in case things get lost with refactoring or something breaks between Calcite upgrades. For example, https://github.com/apache/druid/blob/master/sql/src/test/java/org/apache/druid/sql/calcite/CalciteInsertDmlTest.java#L608 could be extended to test the newly added literal support.

sql/src/main/codegen/includes/common.ftl

sql/src/main/codegen/includes/replace.ftl

sql/src/main/java/org/apache/druid/sql/calcite/parser/DruidSqlInsert.java

sql/src/main/java/org/apache/druid/sql/calcite/parser/DruidSqlReplace.java

sql/src/main/java/org/apache/druid/sql/calcite/parser/DruidSqlParserUtils.java

sql/src/test/resources/calcite/expected/ingest/insertWithClusteredBy-logicalPlan.txt

sql/src/main/java/org/apache/druid/sql/calcite/parser/SqlGranularityLiteral.java

abhishekrb19

Thanks for the changes @zachjsh!

kgyrtkirk · 2024-02-09T14:40:25Z

sql/src/main/java/org/apache/druid/sql/calcite/parser/SqlGranularityLiteral.java

+  private String unparseString;
+  private Granularity granularity;


note: these can be final

kgyrtkirk · 2024-02-09T14:41:17Z

sql/src/main/java/org/apache/druid/sql/calcite/parser/SqlGranularityLiteral.java

+  @Override
+  public void unparse(SqlWriter writer, int leftPrec, int rightPrec)
+  {
+    if (unparseString != null) {


note: not sure if you need to be defensive here; its @NonNull in the constructor

kgyrtkirk · 2024-02-09T14:48:50Z

sql/src/main/java/org/apache/druid/sql/calcite/parser/DruidSqlParserUtils.java

+    }
+  }
+
+  private static Granularity convertSqlIdentiferToGranularity(SqlIdentifier identifier)


these 2 methods are almost identical; the difference is how they get to the String ; can you leave just one with a String arg - and do the X2String conversion before calling that method?

…urce input expressions (#15962) * * address remaining comments from #15836 * * address remaining comments from #15908 * * add test that exposes relational algebra issue * * simplify test exposing issue * * fix * * add tests for sealed / non-sealed * * update test descriptions * * fix test failure when -Ddruid.generic.useDefaultValueForNull=true * * check type assignment based on natice Druid types * * add tests that cover missing jacoco coverage * * add replace tests * * add more tests and comments about column ordering * * simplify tests * * review comments * * remove commented line * * STRING family types should be validated as non-null

* Extend the PARTITION BY clause to accept string literals for the ti…

e511502

…me partitioning

github-actions bot added the Area - Querying label Feb 6, 2024

zachjsh requested a review from abhishekrb19 February 6, 2024 00:16

zachjsh added the Area - Ingestion label Feb 6, 2024

zachjsh added 2 commits February 5, 2024 19:24

* fix exception

a82730f

* update docs

f44e694

github-actions bot added the Area - Documentation label Feb 6, 2024

abhishekrb19 reviewed Feb 6, 2024

View reviewed changes

abhishekrb19 requested a review from LakshSingla February 6, 2024 02:26

kgyrtkirk reviewed Feb 6, 2024

View reviewed changes

zachjsh added 2 commits February 6, 2024 14:20

* address some review comments

1daeab8

* address more comments

1168cfd

zachjsh requested review from kgyrtkirk and abhishekrb19 February 6, 2024 20:05

github-advanced-security bot found potential problems Feb 6, 2024

View reviewed changes

sql/src/main/java/org/apache/druid/sql/calcite/parser/DruidSqlParserUtils.java Fixed Show resolved Hide resolved

abhishekrb19 reviewed Feb 7, 2024

View reviewed changes

zachjsh added 2 commits February 7, 2024 13:25

Merge remote-tracking branch 'apache/master' into partition-by-accept…

75c0510

…s-string-literals

* address review comments

96b3ac1

zachjsh requested a review from abhishekrb19 February 7, 2024 20:18

abhishekrb19 reviewed Feb 8, 2024

View reviewed changes

zachjsh added 4 commits February 8, 2024 14:20

* address review comments

265627e

* fix stuff

cead1f4

* allow for period form granularity in PARTITIONED BY clause

5e502ec

* address more review comments

8a5a2d8

zachjsh requested a review from abhishekrb19 February 8, 2024 23:24

github-advanced-security bot found potential problems Feb 8, 2024

View reviewed changes

sql/src/main/java/org/apache/druid/sql/calcite/parser/SqlGranularityLiteral.java Fixed Show fixed Hide fixed

* implement clone method on new SqlNode

1d7b80f

abhishekrb19 approved these changes Feb 9, 2024

View reviewed changes

kgyrtkirk approved these changes Feb 9, 2024

View reviewed changes

zachjsh merged commit f9ee2c3 into apache:master Feb 9, 2024
83 checks passed

zachjsh deleted the partition-by-accepts-string-literals branch February 9, 2024 16:45

zachjsh added a commit to zachjsh/druid that referenced this pull request Feb 24, 2024

* address remaining comments from apache#15836

acfa4ca

zachjsh mentioned this pull request Feb 24, 2024

INSERT/REPLACE dimension target column types are validated against source input expressions #15962

Merged

10 tasks

abhishekrb19 changed the title ~~Extend the PARTITION BY clause to accept string literals for the time partitioning~~ Extend the PARTITIONED BY clause to accept string literals for the time partitioning Apr 9, 2024

adarshsanjeev added this to the 30.0.0 milestone May 6, 2024

adarshsanjeev mentioned this pull request May 28, 2024

[DRAFT] 30.0.0 release notes #16505

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extend the PARTITIONED BY clause to accept string literals for the time partitioning #15836

Extend the PARTITIONED BY clause to accept string literals for the time partitioning #15836

zachjsh commented Feb 6, 2024 •

edited

Loading

kgyrtkirk Feb 6, 2024

zachjsh Feb 6, 2024

kgyrtkirk Feb 8, 2024

zachjsh Feb 8, 2024

abhishekrb19 left a comment

abhishekrb19 left a comment

kgyrtkirk Feb 9, 2024

kgyrtkirk Feb 9, 2024

kgyrtkirk Feb 9, 2024

		private String unparseString;
		private Granularity granularity;

Extend the PARTITIONED BY clause to accept string literals for the time partitioning #15836

Extend the PARTITIONED BY clause to accept string literals for the time partitioning #15836

Conversation

zachjsh commented Feb 6, 2024 • edited Loading

Description

kgyrtkirk Feb 6, 2024

Choose a reason for hiding this comment

zachjsh Feb 6, 2024

Choose a reason for hiding this comment

kgyrtkirk Feb 8, 2024

Choose a reason for hiding this comment

zachjsh Feb 8, 2024

Choose a reason for hiding this comment

abhishekrb19 left a comment

Choose a reason for hiding this comment

abhishekrb19 left a comment

Choose a reason for hiding this comment

kgyrtkirk Feb 9, 2024

Choose a reason for hiding this comment

kgyrtkirk Feb 9, 2024

Choose a reason for hiding this comment

kgyrtkirk Feb 9, 2024

Choose a reason for hiding this comment

zachjsh commented Feb 6, 2024 •

edited

Loading