Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Databricks runtime 14.3 & 15.4 LTS #21700

Merged
merged 2 commits into from
Oct 1, 2024
Merged

Conversation

ebyhr
Copy link
Member

@ebyhr ebyhr commented Apr 25, 2024

Release notes

(x) This is not user-visible or is docs only, and no release notes are required.

@cla-bot cla-bot bot added the cla-signed label Apr 25, 2024
@github-actions github-actions bot added the docs label Apr 25, 2024
Copy link
Member

@findepi findepi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

@ebyhr ebyhr marked this pull request as ready for review April 25, 2024 23:00
@ebyhr
Copy link
Member Author

ebyhr commented Apr 26, 2024

It seems 14.3 LTS enabled deletion vectors by default: https://github.com/trinodb/trino/actions/runs/8840347798/job/24275694250?pr=21700. Some tests started failing due to unsupported writer feature.

@ebyhr ebyhr marked this pull request as draft April 26, 2024 02:57
@findinpath findinpath requested a review from pajaks May 3, 2024 09:01
@ebyhr ebyhr changed the base branch from master to ebi/delta-write-deletion-vector May 26, 2024 23:35
@github-actions github-actions bot added hudi Hudi connector iceberg Iceberg connector delta-lake Delta Lake connector hive Hive connector labels May 26, 2024
@ebyhr ebyhr changed the base branch from ebi/delta-write-deletion-vector to master May 26, 2024 23:35
@benbauer-draeger
Copy link

Hey, is someone workign on this? Support of deletion vectors would be highly appreciated

@ebyhr
Copy link
Member Author

ebyhr commented Aug 19, 2024

@benbauer-draeger This PR just adds a new CI environment of Databricks 14.3 LTS. #22102 to support writing deletion vectors was already merged in version 454.

@ebyhr ebyhr force-pushed the ebi/delta-143-lts branch 4 times, most recently from 00eb4ff to 762c709 Compare October 1, 2024 05:03
@ebyhr ebyhr changed the title Add support for Databricks runtime 14.3 LTS Add support for Databricks runtime 14.3 & 15.4 LTS Oct 1, 2024
@@ -330,7 +331,7 @@ public void testInsertingIntoDatabricksTableWithAddedNotNullConstraint()
}
}

@Test(groups = {DELTA_LAKE_DATABRICKS, PROFILE_SPECIFIC_TESTS})
@Test(groups = {DELTA_LAKE_DATABRICKS_104, PROFILE_SPECIFIC_TESTS})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this change made?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The connector doesn't support VACUUM with deletion vector, so I specified the old version here. Added a comment to revert this change in #22809.

@ebyhr ebyhr marked this pull request as ready for review October 1, 2024 08:57
@ebyhr ebyhr merged commit 94d1b50 into master Oct 1, 2024
107 of 108 checks passed
@ebyhr ebyhr deleted the ebi/delta-143-lts branch October 1, 2024 08:57
@github-actions github-actions bot added this to the 460 milestone Oct 1, 2024
@mosabua
Copy link
Member

mosabua commented Oct 1, 2024

Are you sure we dont want a release notes entry for this @ebyhr and @findinpath ?

@mosabua mosabua mentioned this pull request Oct 1, 2024
1 task
@ebyhr
Copy link
Member Author

ebyhr commented Oct 1, 2024

@mosabua We don't need a release note entry because this PR changed only tests.

@mosabua
Copy link
Member

mosabua commented Oct 1, 2024

ok .. thanks @ebyhr

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla-signed delta-lake Delta Lake connector docs hive Hive connector hudi Hudi connector iceberg Iceberg connector
Development

Successfully merging this pull request may close these issues.

5 participants