Improve Position Deletes in V3

[aokolnychyi]

Proposed Change

This proposal aims to enhance the handling of position deletes in Iceberg. It builds on lessons learned from deploying the current approach at scale and addresses all unresolved questions from past community discussions and proposals. This effort primarily targets the V3 spec but some parts can be completed independently.

Shortcomings of position deletes we observe today:

• Choosing between fewer delete files on disk (partition granularity) or targeted deletes (file granularity).
• Dependence on external maintenance for consistent write and read performance.
• Writing and reading overhead as in-memory and on-disk representations differ.

Proposal document

https://docs.google.com/document/d/18Bqhr-vnzFfQk1S4AgRISkA_5_m5m32Nnc2Cw0zn2XM

Specifications

• Table
• View
• REST
• Puffin
• Encryption
• Other