Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for writer version 7 in Delta Lake #18423

Merged
merged 1 commit into from
Sep 14, 2023
Merged

Add support for writer version 7 in Delta Lake #18423

merged 1 commit into from
Sep 14, 2023

Conversation

ebyhr
Copy link
Member

@ebyhr ebyhr commented Jul 26, 2023 •
edited
Loading

Description

Fixes #15873

Release notes

(x) Release notes are required, with the following suggested text:

# Delta Lake
* Add support for writer version 7. ({issue}`15873`)

@cla-bot cla-bot bot added the cla-signed label Jul 26, 2023
@ebyhr ebyhr self-assigned this Jul 26, 2023
@github-actions github-actions bot added tests:hive delta-lake Delta Lake connector labels Jul 26, 2023
@ebyhr ebyhr force-pushed the ebi/delta-writer-7 branch 5 times, most recently from 8e63e0b to c6c4cad Compare July 31, 2023 01:17
@ebyhr ebyhr force-pushed the ebi/delta-writer-7 branch 4 times, most recently from 517a340 to 0557cbe Compare August 15, 2023 07:55
@ebyhr ebyhr marked this pull request as ready for review August 15, 2023 10:05
@ebyhr ebyhr force-pushed the ebi/delta-writer-7 branch 2 times, most recently from aa8b88d to 47e20fc Compare August 15, 2023 23:17
@findinpath
Copy link
Contributor

Build is 🔴

Error:  Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.11.0:compile (default-compile) on project trino-product-tests: Compilation failure
Error:  /home/runner/work/trino/trino/testing/trino-product-tests/src/main/java/io/trino/tests/product/deltalake/TestDeltaLakeColumnMappingMode.java:[131,80] cannot find symbol
Error:    symbol:   variable ENGLISH
Error:    location: class io.trino.tests.product.deltalake.TestDeltaLakeColumnMappingMode

@ebyhr ebyhr marked this pull request as draft August 22, 2023 23:05
@ebyhr ebyhr force-pushed the ebi/delta-writer-7 branch 5 times, most recently from 1b11c0a to 70b9faa Compare August 25, 2023 04:01
@ebyhr ebyhr marked this pull request as ready for review August 25, 2023 06:01
@ebyhr
Copy link
Member Author

ebyhr commented Sep 1, 2023

Addressed comments.

@ebyhr ebyhr mentioned this pull request Sep 4, 2023
Merged
@ebyhr
Copy link
Member Author

ebyhr commented Sep 4, 2023

Rebased on master to resolve conflicts.

findinpath
findinpath reviewed Sep 4, 2023
View reviewed changes
plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMetadata.java Show resolved Hide resolved
plugin/trino-delta-lake/src/test/resources/databricks/invariants_writer_feature/README.md Outdated Show resolved Hide resolved
...rc/main/java/io/trino/tests/product/deltalake/TestDeltaLakeCheckConstraintCompatibility.java Outdated Show resolved Hide resolved
...uct-tests/src/main/java/io/trino/tests/product/deltalake/TestDeltaLakeColumnMappingMode.java Outdated Show resolved Hide resolved
...t-tests/src/main/java/io/trino/tests/product/deltalake/TestDeltaLakeDeleteCompatibility.java Show resolved Hide resolved
...rc/main/java/io/trino/tests/product/deltalake/TestDeltaLakeWriteDatabricksCompatibility.java Show resolved Hide resolved
...n/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TableSnapshot.java Outdated Show resolved Hide resolved
...lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeMinioAndHmsConnectorSmokeTest.java Outdated Show resolved Hide resolved
@ebyhr
Copy link
Member Author

ebyhr commented Sep 4, 2023

Addressed comments.

@ebyhr ebyhr mentioned this pull request Sep 5, 2023
Merged
@ebyhr ebyhr mentioned this pull request Sep 5, 2023
Merged
@ebyhr ebyhr force-pushed the ebi/delta-writer-7 branch 2 times, most recently from 26f18f4 to 0b81e20 Compare September 5, 2023 11:19
@ebyhr
Copy link
Member Author

ebyhr commented Sep 5, 2023

@findepi @alexjo2144 Could you take another look when you have time?

@ebyhr ebyhr mentioned this pull request Sep 6, 2023
Merged
findepi
findepi reviewed Sep 6, 2023
View reviewed changes
plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMetadata.java
@@ -501,7 +502,7 @@ public LocatedTableHandle getTableHandle(ConnectorSession session, SchemaTableNa
LOG.debug("Skip %s because the table contains unsupported reader features: %s", tableName, unsupportedReaderFeatures);
return null;
}
verifySupportedColumnMapping(getColumnMappingMode(metadataEntry));
verifySupportedColumnMapping(getColumnMappingMode(metadataEntry, protocolEntry));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this change go as a separate prep commit?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even if we separate a commit, the preparatory commit will contain almost all changes except for tests.

@ebyhr
Copy link
Member Author

ebyhr commented Sep 13, 2023 •
edited
Loading

Rebased on master to resolve conflicts and added another commit to handle deletion vectors feature in protocol entry.

@ebyhr
Add support for writer version 7 in Delta Lake
dd46758 
Support the following writer features:
* appendOnly
* invariants
* checkConstraints
* changeDataFeed
* columnMapping
@ebyhr ebyhr merged commit 6f15559 into master Sep 14, 2023
53 of 54 checks passed
@ebyhr ebyhr deleted the ebi/delta-writer-7 branch September 14, 2023 22:02
@github-actions github-actions bot added this to the 427 milestone Sep 14, 2023
jkylling
jkylling reviewed Sep 22, 2023
View reviewed changes
...n/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TableSnapshot.java
session,
ImmutableSet.of(METADATA),
ImmutableSet.of(METADATA, PROTOCOL),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change likely breaks the checkpoint reading optimizations added in 06e1339

We should probably adopt the optimization to support or-ing multiple tuple domains.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please file an issue to https://github.com/trinodb/trino/issues/new

@colebow colebow mentioned this pull request Sep 25, 2023
Merged
@ebyhr ebyhr mentioned this pull request Sep 25, 2023
Closed
@findinpath findinpath mentioned this pull request Nov 6, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alexjo2144 alexjo2144 alexjo2144 left review comments

@jkylling jkylling jkylling left review comments

@findinpath findinpath findinpath approved these changes

@findepi findepi findepi approved these changes

@homar homar Awaiting requested review from homar

Assignees

@ebyhr ebyhr

Labels
cla-signed delta-lake Delta Lake connector
Milestone
427
Development

Successfully merging this pull request may close these issues.

Delta Lake: Writer version 7 support
5 participants
@ebyhr @findinpath @findepi @jkylling @alexjo2144