Merge: Filtering on partitions

[emcake]

Description

I'd like merge to offer a partitions argument to reduce table churn and processing.

Use Case

Currently the merge operation consumes the whole table and merges in new source data. Currently this re-writes every file in the table: https://gist.github.com/emcake/acc1aa233339a5b3534e2f54702dd46e

For large tables where the updated surface area is small, this is inefficient. I'd propose we have a new parameter partitions: Optional[Union[List[PartitionValues], Literal['auto']]] = None which can provide a list of partitions for the merge operation to be restricted to.

Usage:

# current behaviour
table.merge(source_data, partitions=None) 

# restrict merge to files with the listed PartitionValues. If data in source_data is outside these partitions, it's dropped.
table.merge(source_data, partitions=[{ 'col_a' : 1, 'col_b' : 'foo' }, {'col_a' : 1, 'col_b' : 'bar'}, ...]) 

# use the table partition columns and find all distinct tuples of values in source_data
table.merge(source_data, partitions='auto') 

Related Issue(s)

[thomasfrederikhoeck]

Related to #1846 . I think ideally that the predicate is used like in the example instead of having a seperate keyword.

[ion-elgreco]

@thomasfrederikhoeck I agree, the predicate can be used together with statistics to do file pruning. @Blajda mentions this also in that issue.

Closing this as duplicate.