Upsert into delta table

[moria97]

Description

Currently write to delta table interface implements both append and overwrite mode. But seems overwrite mode can only remove all data and append new data. A typical Upsert scenario is not supported here. Do we have any plans to support overwrite based on some predicates?

Use Case
Update existing data. Not just append.

Related Issue(s)
None

[roeap]

Yes, the plan is to support this. I want to get to implement some more higher level write APIs soon-ish, but there are some priorities that need to get done first.

[moria97]

@roeap Thank you!
Do we have any timeline for the feature? Or do we have any tech docs/wikis for reference? Maybe I can start with my own version first:)

[roeap]

are you using the python APIs, or the rust API directly.

[moria97]

I'm using the rust API, actually I'm writing dotnet and plan to interop rust code from .net.
By the way, is there any plan for delete operation? Maybe this is also related to Upsert as the logic seems similar

[davidgasquez]

Hey there! Curious about the state/priority of this issue. 👀

[wjones127]

Operations like this (that rewrite data) are in our backlog right now. We are currently refactoring our writing APIs for now, and once that's done this should be ready for someone to take on. But I don't think there is yet a contributor who has volunteered to work on this.

[davidgasquez]

Awesome! Thanks for the info @wjones127. Sadly, I don't know Rust to help here.

[jordandakota]

@Blajda @wjones127
Now that delete operation seems to be working, any thoughts on this next?

[Blajda]

Hi @jordandakota to me Upserts are a special case of the Merge operation which is being tracked at #850.
Is there anything that merge will miss once it is implemented?

I plan to work on the update command next and will start to focus more on merge afterwards.

[jordandakota]

No, merge probably covers a larger use case. Insert Overwrite and replaceWhere are important, but mostly so for very large datasets. We can already obtain upsets in two transactions.

[ion-elgreco]

@wjones127 @rtyler maybe close this one since its a duplicate of: #850