-
Notifications
You must be signed in to change notification settings - Fork 234
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for AtomicCreateTableAsSelect with Delta Lake [databricks] #9425
Conversation
Signed-off-by: Jason Lowe <[email protected]>
build |
build |
build |
@@ -92,6 +101,43 @@ object DeltaProviderImpl extends DeltaProviderImplBase { | |||
val cpuFormat = format.asInstanceOf[DeltaParquetFileFormat] | |||
GpuDeltaParquetFileFormat.convertToGpu(cpuFormat) | |||
} | |||
|
|||
override def isSupportedCatalog(catalogClass: Class[_ <: StagingTableCatalog]): Boolean = { | |||
catalogClass == classOf[DeltaCatalog] || catalogClass == classOf[UnityCatalogV2Proxy] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
question: are we guaranteed in this comparison that both catalogClass
and classOf[DeltaCatalog]
are loaded via the same ClassLoader instance, might get a false negative otherwise.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe so. The technique essentially matches the technique used for almost all overrides, as GpuOverrides is building a large hashmap with classes as keys and doing a lookup on that map based on what we find in the CPU plan. So I would think that if we have an issue with this code, we'd have a similar issue with how all overrides work in the plugin.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Fixes #7276. Extended ExternalSource to handle potential replacements for AtomicCreateTableAsSelect and added unit tests. A significant amount of the code change comes from the various GpuCreateDeltaTableCommand files which are small changes from the original versions of CreateDeltaTableCommand sources needed to perform the operation on the GPU. Rather than refactor these for commonality between the Delta Lake versions and make it hard to see the GPU diffs needed for each version, I added each separately to make diffing easier both for the current review and for auditing for changes in Delta Lake patch versions in the future.