You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Deletion vectors are a storage optimization feature that can be enabled on Delta Lake tables. By default, when a single row in a data file is deleted, the entire Parquet file containing the record must be rewritten. With deletion vectors enabled for the table, some Delta operations use deletion vectors to mark existing rows as removed without rewriting the Parquet file. Subsequent reads on the table resolve current table state by applying the deletions noted by deletion vectors to the most recent table version.
Describe the solution you'd like
We need to ensure that we respect deletion vectors in Delta Lake 2.3.0 and later when performing reads. We should also add support for writing deletion vectors when performing writes.
andygrove
changed the title
[FEA] [Delta Lake] Add support for deletion vectors
[FEA] [Delta Lake] Add support for deletion vectors in OSS Delta Lake
Jul 3, 2023
Is your feature request related to a problem? Please describe.
Deletion vectors are a storage optimization feature that can be enabled on Delta Lake tables. By default, when a single row in a data file is deleted, the entire Parquet file containing the record must be rewritten. With deletion vectors enabled for the table, some Delta operations use deletion vectors to mark existing rows as removed without rewriting the Parquet file. Subsequent reads on the table resolve current table state by applying the deletions noted by deletion vectors to the most recent table version.
Describe the solution you'd like
We need to ensure that we respect deletion vectors in Delta Lake 2.3.0 and later when performing reads. We should also add support for writing deletion vectors when performing writes.
Describe alternatives you've considered
None
Additional context
https://docs.delta.io/2.4.0/delta-deletion-vectors.html
The text was updated successfully, but these errors were encountered: