deltalake 0.11.0 fails to read delta table with 416 Range Not Satisfiable error

[pdebelak]

Environment

Delta-rs version:

0.11.0

Binding:

python

Environment:

• Cloud provider: AWS
• OS: linux or macos
• Other:

---

Bug

What happened:

Trying to read a delta table from S3 fails with an error mentioning range size:

OSError: Generic S3 error: response error "<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidRange</Code><Message>The requested range is not satisfiable</Message><RangeRequested>bytes=25004280-25069815</RangeRequested><ActualObjectSize>24436361</ActualObjectSize><RequestId>XXX</RequestId><HostId>XXX</HostId></Error>", after 0 retries: HTTP status client error (416 Range Not Satisfiable) for url (https://s3.us-east-1.amazonaws.com/mybucket/delta/mytable/file.parquet)

What you expected to happen:

I would expect it to read the table since it can do so with 0.10.2.

How to reproduce it:

Create a delta table in s3 using the delta standalone library. Try to read the table with deltalake==0.11.0 out of s3.

More details:

It's possible that I'm doing something wrong writing the data out, but the error message isn't particularly helpful here and I'm not sure what changed in version 0.11.0 that might be causing this..

[pdebelak]

Looks like it's likely a problem on our side that is exposed by #1630. Presumably the size we're writing to the delta is larger than the real file size, causing this issue.