to_pyarrow_table() on a table in S3 kept getting "Generic S3 error: error decoding response body"

[k-ye]

Environment

Delta-rs version: deltalake==0.18.1

Binding: Python

Environment:

• Cloud provider: AWS S3
• OS: Linux
• Other:
  • Python 3.10.4
  • pyarrow==16.1.0
  • pyarrow-hotfix==0.6

---

Bug

What happened:

Trying to do a simple table loading from S3, but kept getting this OSError: Generic S3 error: error decoding response body

table = DeltaTable(table_uri, storage_options=storage_options)
print(f"version: {table.version()}")
print(f"schema: {table.schema()}")
print(table.files())

ts = time.time()
df = table.to_pyarrow_table()

version: 0
schema: Schema([Field(id, PrimitiveType("string"), nullable=True), Field(path, PrimitiveType("string"), nullable=True)])
['0-e03dac34-16a0-4b6e-82c8-fd1098d1bf45-0.parquet']
Traceback (most recent call last):
  File "test.py", line 32, in <module>
    df = table.to_pyarrow_table()
  File "***/lib/python3.10/site-packages/deltalake/table.py", line 1161, in to_pyarrow_table
    return self.to_pyarrow_dataset(
  File "pyarrow/_dataset.pyx", line 562, in pyarrow._dataset.Dataset.to_table
  File "pyarrow/_dataset.pyx", line 3804, in pyarrow._dataset.Scanner.to_table
  File "pyarrow/error.pxi", line 154, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 88, in pyarrow.lib.check_status
OSError: Generic S3 error: error decoding response body

Stack shows that this is actually in pyarrow. Not sure if it possible to tweak pyarrow's behavior with S3 from deltalake.

What you expected to happen:

I can get the pyarrow table.

How to reproduce it:

More details:

I have verified the integrity of this table with these methods:

1. Cloning the table locally, then load from there. to_pyarrow_table() runs fine.
2. Reading the S3 table with duckdb (and its delta extension). Worked fine, too.

[k-ye]

Seems related to #2301 and #2592

[ion-elgreco]

@k-ye can you try deltalake 0.19.1 please and report back if you see improvements

[k-ye]

@k-ye can you try deltalake 0.19.1 please and report back if you see improvements

I've done a few tests:

• On a U.S. machine (but not on AWS), this generally works now. Occasionally it was able to finish the script execution, but the process ended with

terminate called without an active exception
Aborted

• On a CN machine behind a VPN, I could still bump into this error, though.

[ion-elgreco]

@k-ye can you try deltalake 0.19.1 please and report back if you see improvements

I've done a few tests:

• On a U.S. machine (but not on AWS), this generally works now. Occasionally it was able to finish the script execution, but the process ended with

terminate called without an active exception
Aborted

• On a CN machine behind a VPN, I could still bump into this error, though.

Hmm that's a C++ error

Regarding VPN's, I would defer to what Tustvold said: apache/arrow-rs#5882 (comment)

[shriram-louisa]

I have the same issue,
@k-ye were you able to solve this?

[k-ye]

I have the same issue, @k-ye were you able to solve this?

Hi @shriram-louisa ,

Sorry I didn't follow up on this. We were just testing the water when I filed this issue, and have sinced moved on to other solutions...

[shriram-louisa]

Thank you for the response 🙌 @k-ye

[ion-elgreco]

Thank you for the response 🙌 @k-ye

What delta-rs version are you using?

[shriram-louisa]

0.19.2
I am able to avoid this error by using older version i.e 0.15.3

[sim-san]

I am getting the same error. Do know what causes the error?

delta-rs 0.19.1

[ion-elgreco]

@shriram-louisa @sim-san how stable is your connection, are you running this behind a VPN, what's the throughput, how big is your table?

I need more info guys

[ion-elgreco]

Have any of you tired increasing the timeout to 60s or 120s? storage_options = {"timeout":"120s"}

[sim-san]

Have any of you tired increasing the timeout to 60s or 120s? storage_options = {"timeout":"120s"}

The delta table has a file size of around 700MB. The increase of the timeout solved the problem.
But where can I find further documentation about the possible storage_options ?

[ion-elgreco]

Have any of you tired increasing the timeout to 60s or 120s? storage_options = {"timeout":"120s"}

The delta table has a file size of around 700MB. The increase of the timeout solved the problem.
But where can I find further documentation about the possible storage_options ?

You will have to check the object_store crate documentation, and then choose your storage aws S3, it shows the config keys

[junhl]

This seems to also happen when querying a longer data column, probably with same cause where during operation, the network connection is interrupted.

table = DeltaTable("...") # WITHOUT timeout addition
df = table.to_pandas(columns=["some_id"], filters=[("some_id", "in", ["id1", "id2"])]) # works fine in 1 second
df = table.to_pandas(columns=["some_id", "json_data"], filters=[("some_id", "in", ["id1", "id2"])]) # gives decoding error after few mins - json_data is JSON string about few kb per row.

table = DeltaTable("...", storage_options = {"timeout":"120s"}) # WITH timeout addition
df = table.to_pandas(columns=["some_id"], filters=[("some_id", "in", ["id1", "id2"])]) # works fine in 1 second
df = table.to_pandas(columns=["some_id", "json_data"], filters=[("some_id", "in", ["id1", "id2"])]) # works after few mins

[Tom-Newton]

We've been having some similar-ish problems recently on 0.19.0 with Azure. I thought it might be interesting to mention because in our case we only use the object store based filesystem for parsing the delta transaction log and we use the native Azure filesystem built into pyarrow apache/arrow#39968 for the big data reads. Maybe switching to pyarrow filesystem could help some people here? I've also found it to be significantly faster.

It looks like the original issue report was having issues when pyarrow tries to make use of the object store based filesystem implementation.

I'm going to try upgrading us to the latest delta-rs and hopefully #2789 will solve it.