Preserve tags on FileSourceScanExec [databricks] #10461
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Jason Lowe <[email protected]>
jlowe
changed the title
|
build |
ttnghia
reviewed
Feb 22, 2024
Comment on lines
+342
to
+344
| val cpu = wrapped.copy(partitionFilters = partitionFilters) | ||
| cpu.copyTagsFrom(wrapped) | ||
| cpu |
There was a problem hiding this comment.
We have the same code for all shims. Can we extract that to a base class/function and get rid of all these repeated ones?
There was a problem hiding this comment.
Is it OK if that's tracked as a followup? There's a lot more than this method duplicated between shims, and this is trying to unblock the 24.02 release.
There was a problem hiding this comment.
Sure, I'm fine with that.
revans2
approved these changes
Feb 22, 2024
ttnghia
approved these changes
Feb 22, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes a bug when falling back to the CPU for FileSourceScanExec. Unlike most CPU fallbacks where we simply preserve the original node or at most call
withNewChildrenon it, we are calling.copyon it to preserve the possibly GPU converted dynamic filters. Fix is relatively straightforward in that we need to callcopyTagsFromon the new instance to preserve the tags from the original instance.