AsyncChunkReader::get_bytes error: Generic MicrosoftAzure error: error decoding response body

[thomasfrederikhoeck]

Environment

Delta-rs version: 0.18.1

Binding: Python

Environment:

• Cloud provider: Azure
• OS: WIndows
• Other:

---

Bug

What happened:
After 0.18.1 was released it fixes the inital issue with #2301 for me but instead I started hitting this. The Z-order operations start and I can see that there is usage of network, CPU and memory but after 30 secs-ish I'm hit with the following. The Rust logs doens't show anything strange:

metrics = self.table._table.z_order_optimize(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_internal.DeltaError: Failed to parse parquet: Parquet error: Z-order failed while scanning data: ParquetError(General("AsyncChunkReader::get_bytes error: Generic MicrosoftAzure error: error decoding response body"))

What you expected to happen:
That the Z-order completes.

How to reproduce it:

import os
os.environ["RUST_LOG"]="debug"

from deltalake import DeltaTable
 
blob_path = "az://<redacted path>"
storage_options = {"AZURE_STORAGE_ACCOUNT_NAME": "<redacted sa>", "AZURE_CONTAINER_NAME":'<redacted container>', 'use_azure_cli': 'true'}

dt = DeltaTable(blob_path, storage_options=storage_options)
dt.optimize.z_order(["StatusDateTime"])

More details:

[abhiaagarwal]

I'm reasonably confident the error is orignating from here, based on my read of various error messages:

delta-rs/crates/core/src/operations/optimize.rs

Lines 500 to 511 in f041692

	while let Some(maybe_batch) = read_stream.next().await {
	let mut batch = maybe_batch?;
	batch = super::cast::cast_record_batch(
	&batch,
	task_parameters.file_schema.clone(),
	false,
	true,
	)?;
	partial_metrics.num_batches += 1;
	writer.write(&batch).await.map_err(DeltaTableError::from)?;
	}

Since it's run in a blocking context in the python side, I'm wondering if that's causing any weirdness (it shouldn't).

[thomasfrederikhoeck]

@abhiaagarwal I wish I could assist but my Rust knowledge is very limited. But let me know if I need to test something.

[Josh-Hiz]

This same issue is happening occasionally when also reading from a deltatable in Azure Gen 2:

  File "pyarrow\\_dataset.pyx", line 562, in pyarrow._dataset.Dataset.to_table
  File "pyarrow\\_dataset.pyx", line 3804, in pyarrow._dataset.Scanner.to_table
  File "pyarrow\\error.pxi", line 154, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow\\error.pxi", line 88, in pyarrow.lib.check_status
OSError: Generic MicrosoftAzure error: error decoding response body

In which to_table is causing this.

[thomasfrederikhoeck]

@Josh-Hiz what happens if try benchmarking with azcopy like I have done here: apache/arrow-rs#5882 (comment) maybe you can add a data point as a comment?

[thomasfrederikhoeck]

@Josh-Hiz what happens if try benchmarking with azcopy like I have done here: apache/arrow-rs#5882 (comment) maybe you can add a data point as a comment?

Very gentle ping @Josh-Hiz :-)

[ion-elgreco]

@thomasfrederikhoeck try to create a reproducible example that mimics the size and characteristics of your table on Azure. Otherwise no one can properly replicate

[ion-elgreco]

@thomasfrederikhoeck can you test in your environment if I provide you a custom wheel?

[thomasfrederikhoeck]

@ion-elgreco if you have a branch I can build that and try tmw?

[ion-elgreco]

@thomasfrederikhoeck https://github.com/ion-elgreco/delta-rs/tree/chore/fs_debug

Ah wait this is just for reading through pyarrow : P

[thomasfrederikhoeck]

@ion-elgreco Hmm I'm not seeing that debug print you added if I run:

import os
os.environ["RUST_LOG"]="debug"
os.environ["RUST_BACKTRACE"]="1"

from deltalake import DeltaTable


blob_path = "az://<redacted>"
storage_options = {"AZURE_STORAGE_ACCOUNT_NAME": "<redacted>",
                   "AZURE_CONTAINER_NAME":'l<redacted>', 
                   'azure_use_azure_cli': 'true',

                   }
dt = DeltaTable(blob_path, storage_options=storage_options)
dt.optimize.z_order(["<redacted>"])

Do I need to build with certain maturin args?

[ion-elgreco]

@ion-elgreco Hmm I'm not seeing that debug print you added if I run:

import os
os.environ["RUST_LOG"]="debug"
os.environ["RUST_BACKTRACE"]="1"

from deltalake import DeltaTable


blob_path = "az://<redacted>"
storage_options = {"AZURE_STORAGE_ACCOUNT_NAME": "<redacted>",
                   "AZURE_CONTAINER_NAME":'l<redacted>', 
                   'azure_use_azure_cli': 'true',

                   }
dt = DeltaTable(blob_path, storage_options=storage_options)
dt.optimize.z_order(["<redacted>"])

Do I need to build with certain maturin args?

Yeah it's only added in reading through pyarrow dataset interface, the issues you see requires some refactoring in delta-rs 😞

[ion-elgreco]

@thomasfrederikhoeck can you try this one please, it uses an additional runtime for the writing part of optimize: https://github.com/ion-elgreco/delta-rs/tree/fix/use_different_write_rt

[ion-elgreco]

@thomasfrederikhoeck I introduced a separate runtime for IO, can you try 0.19.1 please and let me know if things are improved

[thomasfrederikhoeck]

It worked!! Nice job @ion-elgreco!

[ion-elgreco]

@thomasfrederikhoeck Yey :D