Subsume non-identity partition predicate in Iceberg #12795

findepi · 2022-06-10T13:42:34Z

Before the change, Iceberg connector accepted all predicates expressible
as TupleDomain on primitive columns and they were used to filter data
files during split generation. However, only predicates defined on
identity partitioning columns were subsumed into connector.

This commit extends Iceberg capabilities to subsume predicates on
partitioning columns. Besides subsuming predicates on identity
partitioning columns, it also subsumes predicates if they align with
partitioning boundaries. For example, for truncate(col, 2) (round to
100s) partitioning, predicates col >= 1200 OR col > 1199 are
subsumed, while col > 1200 or col > 1250 are not.

This change is especially important for Iceberg OPTIMIZE table
procedure, which requires the WHERE condition to be fully subsumed
into the connector. It is also helpful for DELETE, as it allows to do
metadata-only delete in more cases, where we don't really needing to do
a row-level delete.

Fixes #7905
For #12362

findepi · 2022-06-10T13:43:17Z

Still a draft, so just FYI @alexjo2144 @losipiuk @ebyhr @erichwang @phd3 @findinpath @homar

findepi · 2022-06-10T14:47:23Z

CI #12726

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergUtil.java

findepi · 2022-06-15T12:35:23Z

This is ready to review.
Test-only prefix is extracted to: #12865

@alexjo2144 @losipiuk @ebyhr @hashhar @erichwang @martint PTAL

alexjo2144

Neat way of implementing this. I want to give it another read

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergUtil.java

erichwang

Mostly nits here, but the logic looks sound.

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/TrinoTypes.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergUtil.java

erichwang · 2022-06-15T21:04:29Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/TrinoTypes.java

+     *
+     * @throws IllegalStateException if this type is not {@link Type#isOrderable() orderable}
+     */
+    public static Optional<Object> getPreviousValue(Type type, Object value)


if you follow the java Math library naming conventions, this would be called: previousBefore or nextDown.

I didn't mean to follow the java Math library here.
Which particular methods are you referring to?
("nextDown" still sounds odd to me)

anyway, i think this should be provided by the Type itself, so let's discuss best naming in #12797

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/TrinoTypes.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/PartitionTransforms.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/TrinoTypes.java

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergConnectorTest.java

losipiuk

looks good overall

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

findepi · 2022-06-20T10:53:27Z

AC

findepi · 2022-06-20T13:56:17Z

Build is green. Rebasing after #12865 merged.

Before the change, Iceberg connector accepted all predicates expressible as `TupleDomain` on primitive columns and they were used to filter data files during split generation. However, only predicates defined on identity partitioning columns were subsumed into connector. This commit extends Iceberg capabilities to subsume predicates on partitioning columns. Besides subsuming predicates on identity partitioning columns, it also subsumes predicates if they align with partitioning boundaries. For example, for `truncate(col, 2)` (round to 100s) partitioning, predicates `col >= 1200` OR `col > 1199` are subsumed, while `col > 1200` or `col > 1250` are not. This change is especially important for Iceberg OPTIMIZE table procedure, which requires the `WHERE` condition to be fully subsumed into the connector. It is also helpful for `DELETE`, as it allows to do metadata-only delete in more cases, where we don't really needing to do a row-level delete.

cla-bot bot added the cla-signed label Jun 10, 2022

findepi force-pushed the findepi/iceberg-expression-pushdown branch from 9f5c58d to 6316cc7 Compare June 10, 2022 13:44

findepi force-pushed the findepi/iceberg-expression-pushdown branch from 6316cc7 to 89ad178 Compare June 10, 2022 14:52

findepi mentioned this pull request Jun 10, 2022

Add value traversal methods to Type #12797

Merged

findepi force-pushed the findepi/iceberg-expression-pushdown branch 2 times, most recently from 03f49a9 to 997709f Compare June 10, 2022 15:19

findepi commented Jun 10, 2022

View reviewed changes

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergUtil.java Outdated Show resolved Hide resolved

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergUtil.java Show resolved Hide resolved

findepi force-pushed the findepi/iceberg-expression-pushdown branch from 86b162b to e609691 Compare June 10, 2022 20:22

findinpath reviewed Jun 13, 2022

View reviewed changes

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergUtil.java Outdated Show resolved Hide resolved

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergUtil.java Show resolved Hide resolved

findepi force-pushed the findepi/iceberg-expression-pushdown branch 2 times, most recently from f4275ea to 50b834d Compare June 14, 2022 13:10

findepi marked this pull request as ready for review June 14, 2022 13:11

This was referenced Jun 14, 2022

Expose Iceberg UUID filtering bug #12836

Merged

Move assertion back to the right test #12837

Merged

findepi force-pushed the findepi/iceberg-expression-pushdown branch 3 times, most recently from a0d139d to 09f133c Compare June 15, 2022 09:04

findepi mentioned this pull request Jun 15, 2022

Fix reading timestamptz before epoch from Parquet #12855

Merged

findepi force-pushed the findepi/iceberg-expression-pushdown branch from 09f133c to 69a2446 Compare June 15, 2022 09:18

github-actions bot added the tests:hive label Jun 15, 2022

findepi force-pushed the findepi/iceberg-expression-pushdown branch 2 times, most recently from 6bb399c to 7e23ddc Compare June 15, 2022 12:31

findepi mentioned this pull request Jun 15, 2022

More test coverage for Iceberg partitioning transforms #12865

Merged

findepi requested review from losipiuk, hashhar, erichwang and martint June 15, 2022 12:35

findepi requested a review from alexjo2144 June 15, 2022 12:35

alexjo2144 reviewed Jun 15, 2022

View reviewed changes

erichwang approved these changes Jun 15, 2022

View reviewed changes

losipiuk reviewed Jun 17, 2022

View reviewed changes

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/TrinoTypes.java Show resolved Hide resolved

losipiuk reviewed Jun 17, 2022

View reviewed changes

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergConnectorTest.java Outdated Show resolved Hide resolved

losipiuk reviewed Jun 17, 2022

View reviewed changes

homar reviewed Jun 20, 2022

View reviewed changes

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java Show resolved Hide resolved

homar approved these changes Jun 20, 2022

View reviewed changes

findepi force-pushed the findepi/iceberg-expression-pushdown branch from 7e23ddc to 15998a5 Compare June 20, 2022 10:53

findepi requested review from erichwang, losipiuk, alexjo2144 and homar June 20, 2022 10:53

losipiuk approved these changes Jun 20, 2022

View reviewed changes

findepi added 4 commits June 20, 2022 15:56

Subsume non-identity partition date predicate in Iceberg

d6eea22

Subsume non-identity partition timestamp predicate in Iceberg

10d3d8f

Subsume non-identity partition timestamptz predicate in Iceberg

4b54f26

findepi force-pushed the findepi/iceberg-expression-pushdown branch from 15998a5 to 4b54f26 Compare June 20, 2022 13:56

findepi merged commit 85c6c09 into trinodb:master Jun 20, 2022

findepi deleted the findepi/iceberg-expression-pushdown branch June 20, 2022 14:15

findepi mentioned this pull request Jun 20, 2022

Release notes for 387 #12870

Closed

github-actions bot added this to the 387 milestone Jun 20, 2022

findepi mentioned this pull request Jun 21, 2022

Support Iceberg OPTIMIZE with WHERE casting timestamp_tz column to a date #12918

Merged

colebow mentioned this pull request Jun 21, 2022

Add Trino 387 release notes #12881

Merged

findepi mentioned this pull request Jun 22, 2022

Derive predicate over base column for CAST(a_varchar AS date) > a_date_constant condition #12925

Closed

alexjo2144 mentioned this pull request Jul 19, 2022

Improve Iceberg partition predicate enforcement #13239

Merged

hantangwangd mentioned this pull request Feb 24, 2024

[Iceberg]Support non-identity partition predicate for thoroughly push down prestodb/presto#21999

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Subsume non-identity partition predicate in Iceberg #12795

Subsume non-identity partition predicate in Iceberg #12795

findepi commented Jun 10, 2022

findepi commented Jun 10, 2022

findepi commented Jun 10, 2022

findepi commented Jun 15, 2022

alexjo2144 left a comment

erichwang left a comment

erichwang Jun 15, 2022

findepi Jun 20, 2022

losipiuk left a comment

findepi commented Jun 20, 2022

findepi commented Jun 20, 2022

Subsume non-identity partition predicate in Iceberg #12795

Subsume non-identity partition predicate in Iceberg #12795

Conversation

findepi commented Jun 10, 2022

findepi commented Jun 10, 2022

findepi commented Jun 10, 2022

findepi commented Jun 15, 2022

alexjo2144 left a comment

Choose a reason for hiding this comment

erichwang left a comment

Choose a reason for hiding this comment

erichwang Jun 15, 2022

Choose a reason for hiding this comment

findepi Jun 20, 2022

Choose a reason for hiding this comment

losipiuk left a comment

Choose a reason for hiding this comment

findepi commented Jun 20, 2022

findepi commented Jun 20, 2022