Compression errors with feather/parquet export (lz4/zstd)

[alexander-beedie]

Versions

Python 3.9 / Polars 0.10.27 / Windows 10

Describe your bug.

Exports to compressed feather/parquet cannot be read back if use_pyarrow=True (succeed only if use_pyarrow=False).
Errors include:

• OSError: ZSTD decompression failed: Src size is incorrect.
• OSError: Lz4 compressed input contains more than one frame.
• OSError: Corrupt Lz4 compressed data.

What are the steps to reproduce the behavior?

# setup simple test data/frame
import polars as pl
from datetime import date

test_data = {
    'key':['abc','mno','xyz'], 
    'date':[date(1973,3,5),date(2000,8,20),date(2022,12,31)],
    'value':[-10.5,None,9999.0],
}
df = pl.DataFrame( data=test_data )

# export to compressed feather/parquet files
df.to_ipc( 'test1.feather', compression='lz4' )
df.to_ipc( 'test2.feather', compression='zstd' )
df.to_parquet( 'test1.parquet', compression='lz4' )
df.to_parquet( 'test2.parquet', compression='zstd' )

What is the actual behavior?

pl.read_ipc( 'test1.feather' )
# OSError: Lz4 compressed input contains more than one frame

pl.read_ipc( 'test2.feather' )
# OSError: ZSTD decompression failed: Src size is incorrect

pl.read_parquet( 'test1.parquet' )
# OSError: Corrupt Lz4 compressed data.

pl.read_parquet( 'test2.parquet' )
# succeeds for parquet with zstd compression

What is the expected behavior?

Successful load into DataFrame for round-trip import/export to compressed feather/parquet from Polars with default settings.

[ritchie46]

Could you open this issue upstream in arrow2? The parquet and IPC implementations are from arrow2.

[jorgecarleitao]

Oh, sorry, I missed this one. Looking into it

[jorgecarleitao]

wrt to the parquet, I concluded that the likely cause is on pyarrow itself, as parquet files written by pyarrow with LZ4 and ZSTD compression are un-readable by (py)spark. Filled bug upstream https://issues.apache.org/jira/browse/ARROW-15073

Will now look into the feather one.

[alexander-beedie]

Many thanks all - I'll try all other possible permutations and -for now- hide compression options that aren't bidirectionally successful on our end, and report any other interesting results (upstream, as requested ;).