Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(DAT-18853): Add custom diff change generator for Databricks #203

Merged
merged 10 commits into from
Nov 1, 2024

Conversation

filipelautert
Copy link
Contributor

  • Added ChangedTableChangeGeneratorDatabricks class to handle Databricks-specific table property changes.
  • Implemented logic to compare and generate changes for table properties.
  • Updated META-INF/services/liquibase.diff.output.changelog.ChangeGenerator to include the new change generator.

- Added `ChangedTableChangeGeneratorDatabricks` class to handle Databricks-specific table property changes.
- Implemented logic to compare and generate changes for table properties.
- Updated `META-INF/services/liquibase.diff.output.changelog.ChangeGenerator` to include the new change generator.
@filipelautert filipelautert self-assigned this Oct 22, 2024
@filipelautert
Copy link
Contributor Author

@KushnirykOleh any idea on how to add tests here?

@KushnirykOleh
Copy link
Contributor

diff-changelog test from test-harness doesn't work properly in extensions. To check parsing I would only add unit test with Strings that looks like these

"name": "test_table",
"tblProperties" : "'delta.checkpoint.writeStatsAsJson'='false', 'delta.checkpoint.writeStatsAsStruct'='true', 'delta.columnMapping.maxColumnId'='2', 'delta.columnMapping.mode'='name', 'delta.enableDeletionVectors'='true', 'delta.feature.allowColumnDefaults'='supported', 'delta.feature.columnMapping'='supported', 'delta.feature.deletionVectors'='supported', 'delta.feature.invariants'='supported', 'delta.minReaderVersion'='3', 'delta.minWriterVersion'='7'"
}
},
{
"table": {
"name": "test_table_properties",
"Location" : "s3://databricks-th/test_table_properties",
"tblProperties" : "'delta.checkpoint.writeStatsAsJson'='false', 'delta.checkpoint.writeStatsAsStruct'='true', 'delta.enableDeletionVectors'='true', 'delta.feature.deletionVectors'='supported', 'delta.feature.invariants'='supported', 'delta.minReaderVersion'='3', 'delta.minWriterVersion'='7', 'this.is.my.key'='12', 'this.is.my.key2'='true'"
}
- actual tblProperties returned from databricks

@filipelautert
Copy link
Contributor Author

To check parsing I would only add unit test with Strings that looks like these

Thanks!

- Added filters to ensure only valid key-value pairs are processed.
- added `ChangedTblPropertiesUtilTest`
@filipelautert filipelautert marked this pull request as ready for review October 24, 2024 12:06
@filipelautert filipelautert changed the title feat: Add custom diff change generator for Databricks feat(DAT-18853): Add custom diff change generator for Databricks Oct 25, 2024
Copy link
Contributor

@KushnirykOleh KushnirykOleh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks legit

- Added filters to ensure only valid key-value pairs are processed.
- Updated `ChangedTblPropertiesUtilTest` to include the necessary imports and package declaration.
- Added a new test case to validate that internal delta properties are ignored
Copy link

sonarcloud bot commented Oct 31, 2024

@filipelautert filipelautert merged commit 311bc8b into main Nov 1, 2024
17 checks passed
@filipelautert filipelautert deleted the DAT-18853 branch November 1, 2024 12:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants