-
Notifications
You must be signed in to change notification settings - Fork 1.7k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Allow schema pruning for delete first pass
Resolves #1411 Re-orders the delete find files to rewrite command to put the empty project (input_file_name) before the non-determinstic filter udf. This allows for top level schema pruning and it Spark 3.4 should allow for nested pruning as well. Also updated the formatting (and `Column(InputFileName())` -> `input_file_name()`) to match Update. New UTs. Performance improvement on delete on data condition. Closes #1412 Signed-off-by: Shixiong Zhu <[email protected]> GitOrigin-RevId: abfee4cf9f8d8ffaef9e397ad9c237c576b8b807
- Loading branch information
1 parent
6bca231
commit 9017ac0
Showing
2 changed files
with
50 additions
and
5 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters