load_with_datetime(datetime_string) throw error message when delta table is created for more than 60 days(twice the default value of logRetentionDuration)

[zhujunyong]

Environment

Delta-rs version:
0.5.4
Binding:
python 3.9.7
Environment:

• Cloud provider: IBM
• OS: CentOS
• Other:

---

Bug

What happened:
load_with_datetime(datetime_string) function always throws error message when the specific detla table is created for more than 60 days.

>>> dt = DeltaTable('s3://mybucket/mypath/mydeltatable')
>>> dt.version()
469
>>> dt.load_with_datetime('2021-11-22T12:00:00Z')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/miniconda3/envs/delta-rs/lib/python3.9/site-packages/deltalake/table.py", line 196, in load_with_datetime
    self._table.load_with_datetime(datetime_string)
deltalake.PyDeltaTableError: Failed to read delta log object: Object not found
>>> 

What you expected to happen:
load the corresponding version and don't throw error message.

How to reproduce it:
create an delta table with specific logRetentionDuration value(small enough), and use load_with_datetime() function.

More details:
I believe it was because this method performs a binary search on all delta transaction logs. in the above case, the latest version no is 469, then it tries to get the timestamp of version 234 at the first time of the loop. but the transaction log of version 234 has been cleaned up(earlier than 30 days), so it throws an error message.

In this file, https://github.com/delta-io/delta-rs/blob/main/rust/src/delta.rs#L1223-L1261 line no 1240 will throw error message when the above conditions are met.

[fvaleye]

Hello @zhujunyong,

Thanks for submitting your issue and the detailled report.

I will need more time to investigate what is the root cause and I keep you updated!

[houqp]

I think @zhujunyong is likely right. We can't do binary search starting from version 0, instead we should probably fetch all available versions through object list API, then do a binary search within that list restul.