Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not run Hive and Delta compatibility tests on Databricks #18878

Merged
merged 1 commit into from
Aug 31, 2023

Conversation

findinpath
Copy link
Contributor

@findinpath findinpath commented Aug 31, 2023

Description

Test against Delta Lake OSS gives enough certainty for the accuracy
of the Delta Lake connector.

Relates to #18805

Additional context and related issues

Release notes

(x) This is not user-visible or is docs only, and no release notes are required.

@cla-bot cla-bot bot added the cla-signed label Aug 31, 2023
@findinpath findinpath added the no-release-notes This pull request does not require release notes entry label Aug 31, 2023
Copy link
Member

@ebyhr ebyhr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we still need hive.properties in AbstractSinglenodeDeltaLakeDatabricks & singlenode-delta-lake-databricks directory and also delta.hive-catalog-name=hive in delta.properties?

@findinpath findinpath changed the title Do not run Hive and Delta redirect tests on Databricks Do not run Hive and Delta compatibility tests on Databricks Aug 31, 2023
@findinpath
Copy link
Contributor Author

findinpath commented Aug 31, 2023

Do we still need hive.properties in AbstractSinglenodeDeltaLakeDatabricks & singlenode-delta-lake-databricks directory and also delta.hive-catalog-name=hive in delta.properties?

I'm trying to figure out whether we need

onTrino().executeQuery("SET SESSION hive.hive_views_legacy_translation = true");

Let's handle this in a follow-up PR to focus solely on this aspect.
In any case TestDatabricksWithGlueMetastoreCleanUp should be probably executed only once per build, not for every databricks product test.

Idea for smoke test to legitimate existence of hive.properties: We could have a hive and delta lake compatibility test where we can observe whether Hive tables (USING parquet) created via Databricks are readable through the hive connector.

Test against Delta Lake OSS gives enough certainty for the accuracy
of the Delta Lake connector.
@findinpath findinpath force-pushed the findinpath/delta-hive-redirect-pt branch from b642f33 to c999c2f Compare August 31, 2023 07:39
@ebyhr ebyhr merged commit c23c8f7 into trinodb:master Aug 31, 2023
40 checks passed
@github-actions github-actions bot added this to the 426 milestone Aug 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla-signed no-release-notes This pull request does not require release notes entry
Development

Successfully merging this pull request may close these issues.

2 participants