Delta-rs writer hangs with to many file handles open (Azure)

[ion-elgreco]

Environment

Delta-rs version: 0.13.0

Binding: Python

Environment:

• Cloud provider: azure
• OS:
• Other:

---

Bug

What happened:
When writing a delta table with approx 10m records but around 35 partitions it hangs randomly due to many file handlers open.

I was able to narrow it down to the file handles since I set the max_files_open to 2. And then I looped the writing by 25 times which all worked fine.

When i removed this parameter it would randomly fail to write.

Some additional weird behavior, I could see some empty partition folders being created and then it hangs and doesn't do anything for many hours until I manually killed the pipeline.

What you expected to happen:
Write also with many file handles open at same time.

How to reproduce it:
Difficult, but the integration test @roeap triggered todat, since it didn't run for a while, was showing the same behaviour: https://github.com/delta-io/delta-rs/actions/runs/6812593991/job/18525273592

More details:

[stefnba]

Sounds like it’s the same issue reported by me #1770

[ion-elgreco]

@stefnba indeed seems to be the same issue! Sorry I missed that and also forget to respond there

[ion-elgreco]

Closing it in favor of discussion in #1770