Support range predicate pushdown for string columns with collation in PostgreSQL connector #9746

takezoe · 2021-10-23T00:40:44Z

No description provided.

kokosing · 2021-10-23T19:12:32Z

plugin/trino-postgresql/src/main/java/io/trino/plugin/postgresql/PostgreSqlClient.java

@@ -37,7 +37,6 @@
 import io.trino.plugin.jdbc.LongWriteFunction;


Please update io.trino.plugin.postgresql.TestPostgreSqlConnectorTest#hasBehavior to support SUPPORTS_PREDICATE_PUSHDOWN_WITH_VARCHAR_INEQUALITY

hashhar

The impl looks good. Please adjust the declaration in tests to make sure the tests pass with the new assumption.

takezoe · 2021-10-25T14:14:58Z

...n/trino-postgresql/src/test/java/io/trino/plugin/postgresql/TestPostgreSqlConnectorTest.java

@@ -81,13 +76,16 @@ public void setExtensions()
    protected boolean hasBehavior(TestingConnectorBehavior connectorBehavior)
    {
        switch (connectorBehavior) {
-            case SUPPORTS_PREDICATE_PUSHDOWN_WITH_VARCHAR_INEQUALITY:
+            case SUPPORTS_JOIN_PUSHDOWN_WITH_VARCHAR_INEQUALITY:
                return false;


Additional updates are necessary to support aggregation pushdown for varchar columns. Since I would focus on support range predicates pushdown in this pull request, I just introduced a new behavior that indicates whether aggregation pushdown is supported for varchar columns.

Let's create a GitHub issue and add a TODO so that we can remove the additional behaviour once it's no longer needed (and other people know that this is something they can work on).

Yeah, that makes sense (as long as this fix can be merged) and I would work on it once this pull request is completed. By the way, supporting type sensitive aggregation pushdown in JDBC plugins doesn't seem easy. Fundamental interface changes may be required.

Yeah, I ran into some similar issues in #7320.

There were some ideas floated in #7320 (comment) (which we didn't end up doing since it looked like a one-off need at that time).

If you already have some direction in your mind it might be helpful to discuss it on #dev on Slack too (if you think the changes will be large and touch the SPI).

Oh, will take a look at #7320 first. Thanks.

takezoe · 2021-10-25T14:30:58Z

TestPostgreSqlConnectorTest#testSystemTable() is still failing with the following error:

io.trino.spi.TrinoException: ERROR: collations are not supported by type name

I'm trying to find a way to fix it.

takezoe · 2021-10-26T05:55:02Z

Looks like collations have been supported by name type since PostgreSQL 12. I would allow pushdown only for column types that isCollatable() returns true for now.

plugin/trino-postgresql/src/main/java/io/trino/plugin/postgresql/PostgreSqlClient.java

takezoe · 2021-10-29T09:17:09Z

@kokosing @hashhar Could you take a look again?

plugin/trino-postgresql/src/main/java/io/trino/plugin/postgresql/PostgreSqlClient.java

testing/trino-testing/src/main/java/io/trino/testing/TestingConnectorBehavior.java

findepi · 2021-10-31T20:38:20Z

...n/trino-postgresql/src/test/java/io/trino/plugin/postgresql/TestPostgreSqlConnectorTest.java

-                                        tableHandle -> ((JdbcTableHandle) tableHandle).getConstraint().isAll(),
-                                        TupleDomain.all(),
-                                        ImmutableMap.of())));
+                                PlanMatchPattern.node(TableScanNode.class)));


why still isNotFullyPushedDown?

With the domain compaction, the original predicate seems to remain on Trino side although the compacted predicate is pushed down to PostgreSQL:

SQL on PostgreSQL:

SELECT "nationkey", "name", "regionkey" FROM "tpch"."nation" WHERE ("name" >= ? COLLATE "C" AND "name" <= ? COLLATE "C")

Plan on Trino:

Output[regionkey, nationkey, name] │ Layout: [regionkey:bigint, nationkey:bigint, name:varchar(25)] └─ RemoteExchange[GATHER] │ Layout: [nationkey:bigint, name:varchar(25), regionkey:bigint] └─ ScanFilter[table = postgresql:tpch.nation tpch.nation constraint on [name] columns=[nationkey:bigint:int8, name:varchar(25):varchar, regionkey:bigint:int8], filterPredicate = ("name" IN (CAST('POLAND' AS varchar(25)), CAST('ROMANIA' AS varchar(25)), CAST('VIETNAM' AS varchar(25))))] Layout: [nationkey:bigint, name:varchar(25), regionkey:bigint] nationkey := nationkey:bigint:int8 regionkey := regionkey:bigint:int8 name := name:varchar(25):varchar

...n/trino-postgresql/src/test/java/io/trino/plugin/postgresql/TestPostgreSqlConnectorTest.java

plugin/trino-postgresql/src/main/java/io/trino/plugin/postgresql/PostgreSqlClient.java

hashhar · 2021-11-01T05:20:12Z

...n/trino-postgresql/src/test/java/io/trino/plugin/postgresql/TestPostgreSqlConnectorTest.java

@@ -81,13 +76,16 @@ public void setExtensions()
    protected boolean hasBehavior(TestingConnectorBehavior connectorBehavior)
    {
        switch (connectorBehavior) {
-            case SUPPORTS_PREDICATE_PUSHDOWN_WITH_VARCHAR_INEQUALITY:
+            case SUPPORTS_JOIN_PUSHDOWN_WITH_VARCHAR_INEQUALITY:
                return false;


Let's create a GitHub issue and add a TODO so that we can remove the additional behaviour once it's no longer needed (and other people know that this is something they can work on).

takezoe · 2021-11-08T23:51:11Z

@findepi Finished updating for now. Could you take a look again?

hashhar

Some minor comments.

Can we detect (or apply some heuristics) to determine during runtime whether a given predicate needs the COLLATION applied or not? For example for equality predicates adding the collation will lead to a definite performance regression because no indexes can be used anymore.

If it's not possible to do dynamically during runtime then let's add an experimental. prefixed config property to enable this behaviour and keep it opt in for now?

WDYT @wendigo ?

plugin/trino-postgresql/src/main/java/io/trino/plugin/postgresql/PostgreSqlClient.java

...n/trino-postgresql/src/test/java/io/trino/plugin/postgresql/TestPostgreSqlConnectorTest.java

takezoe · 2021-11-11T16:29:41Z

For example for equality predicates adding the collation will lead to a definite performance regression because no indexes can be used anymore.

I think it's possible in some cases but 100% is impossible. But anyway it should be better than full-scan?

If it's not possible to do dynamically during runtime then let's add an experimental. prefixed config property to enable this behaviour and keep it opt in for now?

I'm fine to make this optional as long as we can enable this by configuration. Is adding experimental.postgresql.enable-string-pushdown-with-collate ok?

hashhar · 2021-11-15T06:07:49Z

For example for equality predicates adding the collation will lead to a definite performance regression because no indexes can be used anymore.

I think it's possible in some cases but 100% is impossible. But anyway it should be better than full-scan?

If it's not possible to do dynamically during runtime then let's add an experimental. prefixed config property to enable this behaviour and keep it opt in for now?

I'm fine to make this optional as long as we can enable this by configuration. Is adding experimental.postgresql.enable-string-pushdown-with-collate ok?

@takezoe You can add a session property with matching config property in PostgreSqlConfig. I think experimental.postgresql.string-pushdown-with-collate is fine.

cc: @findepi @kokosing @wendigo @ebyhr Any opinions? I'd prefer to have this behind a configuration toggle (at-least for now) until we can verify that there isn't a performance concern in actual practical usage?

hashhar

Looks good % confirmation about config from other maintainers.

...n/trino-postgresql/src/test/java/io/trino/plugin/postgresql/TestPostgreSqlConnectorTest.java

takezoe · 2021-11-16T09:01:25Z

Looks like postgresql.experimental.string-pushdown-with-collate is more consistent because PostgreSqlConfig used to have another experimental setting postgresql.experimental.array-mapping:

trino/plugin/trino-postgresql/src/main/java/io/trino/plugin/postgresql/PostgreSqlConfig.java

Line 40 in f7aac0d

@LegacyConfig("postgresql.experimental.array-mapping")

hashhar

LGTM % comments.

Mostly about possible simplifications.

plugin/trino-postgresql/src/main/java/io/trino/plugin/postgresql/PostgreSqlClient.java

plugin/trino-postgresql/src/main/java/io/trino/plugin/postgresql/PostgreSqlConfig.java

takezoe · 2021-11-27T15:50:07Z

@hashhar Finished updating the pull request. Could you take a look again?

hashhar

Looks good to me % comment. Thanks for working on this.

Also it looks the first two commits should be squashed together.

plugin/trino-postgresql/src/test/java/io/trino/plugin/postgresql/PostgreSqlQueryRunner.java

hashhar

LGTM % minor testing change.

Thanks for working on this @takezoe. You might also be interested in #10059 which we plan to use as the base for allowing join pushdown too using COLLATE.

...n/trino-postgresql/src/test/java/io/trino/plugin/postgresql/TestPostgreSqlConnectorTest.java

takezoe · 2021-11-30T01:28:32Z

Looks like hasBehavior() has to behave differently depending on whether enable_string_pushdown_with_collate is true or false. And it seems annoying to cover by a single test class.

For example, having a flag as a field and changing its value inside testStringPushdownWithCollate() is simple but I don't think it's an ideal way. Accepting Session at hasBehavior() might be smarter but this will require many changes even in other JDBC connectors' test cases.

Does splitting TestPostgreSqlConnectorTest to create another test class just for testing string pushdown with collate make sense?

takezoe · 2021-11-30T08:13:57Z

having a flag as a field and changing its value inside testStringPushdownWithCollate()

I adopted this way for now and added a case for join on VARCHAR columns.

hashhar · 2021-12-01T07:16:54Z

I adopted this way for now and added a case for join on VARCHAR columns.

Since only PostgreSQL has this behaviour today (and not enabled by default) I think it's fine to not add a new behaviour to TestingConnectorBehaviour just yet. That way you don't need the flag field (especially because we aren't using hasBehaviour in the new test at all).

In terms of future evolution I think once we prove it out with PostgreSQL we can lift the config to BaseJdbcConfig and then add a test in BaseJdbcConnectorTest which sets the session property - at that time we can revisit the best way to structure the TestingConnectorBehaviour (or maybe by that time we'll have more formal ways of handling collations).

@findepi @ebyhr @wendigo @takezoe Any thoughts?

findepi · 2021-12-01T08:00:51Z

@hashhar sounds good. except i am not convinced we need a config just yet, except for trying things out (a temporary kill switch)

takezoe · 2021-12-02T16:43:45Z

Ah, I see. That makes sense. I updated the test case.

hashhar · 2021-12-03T18:11:18Z

Closing and re-opening to get ci to run (some glitch on GitHub end).

hashhar · 2021-12-03T21:17:49Z

Unrelated failure. Thanks @takezoe for the feature. Merging it.

cla-bot bot added the cla-signed label Oct 23, 2021

kokosing reviewed Oct 23, 2021

View reviewed changes

hashhar reviewed Oct 25, 2021

View reviewed changes

takezoe force-pushed the postgresql_pushdown_collate branch 2 times, most recently from 69077fa to 968b6cd Compare October 25, 2021 14:09

takezoe commented Oct 25, 2021

View reviewed changes

takezoe force-pushed the postgresql_pushdown_collate branch from 968b6cd to ac742f8 Compare October 25, 2021 14:16

hashhar reviewed Oct 26, 2021

View reviewed changes

plugin/trino-postgresql/src/main/java/io/trino/plugin/postgresql/PostgreSqlClient.java Outdated Show resolved Hide resolved

takezoe force-pushed the postgresql_pushdown_collate branch from dc45a73 to d9c1e3a Compare October 29, 2021 07:27

hashhar requested review from wendigo and hashhar October 29, 2021 09:53

findepi reviewed Oct 31, 2021

View reviewed changes

hashhar reviewed Nov 1, 2021

View reviewed changes

takezoe force-pushed the postgresql_pushdown_collate branch from 9908b52 to ee99698 Compare November 2, 2021 04:49

findepi requested a review from hashhar November 9, 2021 10:06

hashhar reviewed Nov 10, 2021

View reviewed changes

hashhar reviewed Nov 15, 2021

View reviewed changes

...n/trino-postgresql/src/test/java/io/trino/plugin/postgresql/TestPostgreSqlConnectorTest.java Outdated Show resolved Hide resolved

hashhar mentioned this pull request Nov 16, 2021

Support predicate pushdown for text types for connectors whose default collation differs from Trino #7496

Open

hashhar reviewed Nov 16, 2021

View reviewed changes

takezoe force-pushed the postgresql_pushdown_collate branch 2 times, most recently from 587edbe to dbddd0a Compare November 20, 2021 03:46

takezoe force-pushed the postgresql_pushdown_collate branch 2 times, most recently from 28313bd to c59edbe Compare November 27, 2021 08:54

hashhar reviewed Nov 29, 2021

View reviewed changes

plugin/trino-postgresql/src/test/java/io/trino/plugin/postgresql/PostgreSqlQueryRunner.java Outdated Show resolved Hide resolved

takezoe force-pushed the postgresql_pushdown_collate branch 2 times, most recently from 005f947 to c5646cc Compare November 29, 2021 16:18

hashhar approved these changes Nov 29, 2021

View reviewed changes

...n/trino-postgresql/src/test/java/io/trino/plugin/postgresql/TestPostgreSqlConnectorTest.java Show resolved Hide resolved

hashhar mentioned this pull request Nov 30, 2021

Describe support of CHAR/VARCHAR predicate pushdown #10110

Merged

takezoe force-pushed the postgresql_pushdown_collate branch from c5646cc to 0721fe8 Compare November 30, 2021 08:12

takezoe force-pushed the postgresql_pushdown_collate branch 2 times, most recently from e5e96e6 to 7e5ae28 Compare November 30, 2021 08:51

takezoe force-pushed the postgresql_pushdown_collate branch from 7e5ae28 to 0a1f394 Compare December 2, 2021 16:43

takezoe force-pushed the postgresql_pushdown_collate branch 3 times, most recently from 6fd6890 to 16b3c0c Compare December 3, 2021 16:12

Support string pushdown with collation in PostgreSQL connector

5eb794a

takezoe force-pushed the postgresql_pushdown_collate branch from 16b3c0c to 5eb794a Compare December 3, 2021 17:08

hashhar closed this Dec 3, 2021

hashhar reopened this Dec 3, 2021

hashhar approved these changes Dec 3, 2021

View reviewed changes

hashhar merged commit fe1c6ec into trinodb:master Dec 3, 2021

github-actions bot added this to the 365 milestone Dec 3, 2021

takezoe mentioned this pull request Dec 30, 2021

Support JOIN pushdown for relational operators on varchar in PostgreSQL connector #10435

Closed

hashhar mentioned this pull request Jan 5, 2022

Refactor QueryBuilder usage and implement *char join pushdown in PostgreSQL #10059

Merged

findepi mentioned this pull request Aug 29, 2022

Push varchar non-equality comparisons into PostgreSQL #13894

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support range predicate pushdown for string columns with collation in PostgreSQL connector #9746

Support range predicate pushdown for string columns with collation in PostgreSQL connector #9746

takezoe commented Oct 23, 2021

kokosing Oct 23, 2021

takezoe Oct 25, 2021

hashhar left a comment

takezoe Oct 25, 2021 •

edited

Loading

hashhar Nov 1, 2021

takezoe Nov 1, 2021

hashhar Nov 1, 2021

takezoe Nov 1, 2021

takezoe commented Oct 25, 2021

takezoe commented Oct 26, 2021 •

edited

Loading

takezoe commented Oct 29, 2021

findepi Oct 31, 2021

takezoe Nov 1, 2021 •

edited

Loading

hashhar Nov 1, 2021

takezoe commented Nov 8, 2021

hashhar left a comment

takezoe commented Nov 11, 2021 •

edited

Loading

hashhar commented Nov 15, 2021

hashhar left a comment

takezoe commented Nov 16, 2021 •

edited

Loading

hashhar left a comment

takezoe commented Nov 27, 2021

hashhar left a comment

hashhar left a comment

takezoe commented Nov 30, 2021 •

edited

Loading

takezoe commented Nov 30, 2021 •

edited

Loading

hashhar commented Dec 1, 2021

findepi commented Dec 1, 2021

takezoe commented Dec 2, 2021

hashhar commented Dec 3, 2021

hashhar commented Dec 3, 2021

		@@ -37,7 +37,6 @@
		import io.trino.plugin.jdbc.LongWriteFunction;

Support range predicate pushdown for string columns with collation in PostgreSQL connector #9746

Support range predicate pushdown for string columns with collation in PostgreSQL connector #9746

Conversation

takezoe commented Oct 23, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hashhar left a comment

Choose a reason for hiding this comment

takezoe Oct 25, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

takezoe commented Oct 25, 2021

takezoe commented Oct 26, 2021 • edited Loading

takezoe commented Oct 29, 2021

Choose a reason for hiding this comment

takezoe Nov 1, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

takezoe commented Nov 8, 2021

hashhar left a comment

Choose a reason for hiding this comment

takezoe commented Nov 11, 2021 • edited Loading

hashhar commented Nov 15, 2021

hashhar left a comment

Choose a reason for hiding this comment

takezoe commented Nov 16, 2021 • edited Loading

hashhar left a comment

Choose a reason for hiding this comment

takezoe commented Nov 27, 2021

hashhar left a comment

Choose a reason for hiding this comment

hashhar left a comment

Choose a reason for hiding this comment

takezoe commented Nov 30, 2021 • edited Loading

takezoe commented Nov 30, 2021 • edited Loading

hashhar commented Dec 1, 2021

findepi commented Dec 1, 2021

takezoe commented Dec 2, 2021

hashhar commented Dec 3, 2021

hashhar commented Dec 3, 2021

takezoe Oct 25, 2021 •

edited

Loading

takezoe commented Oct 26, 2021 •

edited

Loading

takezoe Nov 1, 2021 •

edited

Loading

takezoe commented Nov 11, 2021 •

edited

Loading

takezoe commented Nov 16, 2021 •

edited

Loading

takezoe commented Nov 30, 2021 •

edited

Loading

takezoe commented Nov 30, 2021 •

edited

Loading