The written delta file has corrupted structure

[Anna050689]

Environment

Delta-rs version:0.6.3

Binding: python 3.9.13

Environment:

• Cloud provider: Azure
• OS: Windows
• Other:

---

Bug

What happened:
The dataframe was written in invalid format to AWS S3. The _delta_log has the next structure:

What you expected to happen:

How to reproduce it:

More details:

[djouallah]

same issue with gcp !!!

[wjones127]

What do you mean by corrupted? Does it cause another reader to error?

This looks like a temporary file written in a failed transaction. We create temporary files and them atomically rename them to create the final commit. It should be removed by a vacuum (but let me know if it doesn’t—-that would be a bug.)

[djouallah]

@wjones127 same error here, #878

[djouallah]

same error with 0.6.4, it is the same bug reported both in aws and GCP, can you guys have a look !!!

[wjones127]

The behavior described here is not a bug. Those temporary files are created intentionally and should not interfere with any reader.

If you do encounter a bug where a temporary file breaks something, please file an issue that includes the error you encounter.

[djouallah]

@wjones127 yes I did already, see here #878 , the json is not written into the cloud storage, delta table can not be read, it is a corrupted table !!!

[djouallah]

@wjones127 can you please reopen this bug report the issue is not solved, the json is not written at all in the delta_log folder

[wjones127]

@djouallah if there is already a ticket open that contains the specific error, then please ping me there, not in a separate ticket.