Saving to compressed zarr yields lower or negative compression

[mattphysics]

• xbitinfo version:
  xbitinfo 0.0.2 pypi_0 pypi

• Python version:
  python 3.10.5 hdaaf3db_0_cpython conda-forge

• Operating System:
  macOS Big Sur

Description

I ran the tutorial notebook for saving to .nc and .zarr (https://xbitinfo.readthedocs.io/en/latest/quick-start.html). I expected progressively smaller file sizes, with "original" > "compressed" > "bitrounded". For .nc, this is what happened, while for .zarr

1. the file size of the "original" is 30% smaller than for .nc (2.8 MB).
2. Doing "ds.to_compressed_zarr") increases the size by 60% (4.5 MB).
3. Saving the bitrounded data to compressed_zarr yields (as expected) a smaller file size (804 KB), which is 64% larger than the equivalent .nc file
   
   

I have uploaded the notebook with results at https://github.com/mattphysics/xbitinfo/blob/main/tests/nb_xbitinfo_export.ipynb

[observingClouds]

Hi @mattphysics,
Thanks for trying out xbitinfo and opening this PR. The tricky part here is that to_zarr compresses by default. This explains why the xbitinfo/original.zarr is much smaller than xbitinfo/original.nc. The to_compressed_zarr method that xbitinfo provides is rather optimised for bit rounded data and might have some overhead for a small file without bit rounding.

Check out

xbitinfo/tests/test_save_compressed.py

Lines 59 to 71 in 4d5563e

	def test_to_compressed_zarr(rasm):
	"""Test to_compressed_zarr reduces size on disk."""
	ds = rasm
	label = "file"
	# save
	encoding = {
	var: {"compressor": None} for var in ds.data_vars
	} # deactivate default compression
	ds.to_zarr(f"./tmp_testdir/{label}.zarr", mode="w", encoding=encoding)
	ds.to_compressed_zarr(f"./tmp_testdir/{label}_compressed.zarr", mode="w")
	# check size reduction
	ori_size = get_zarr_size(f"./tmp_testdir/{label}.zarr")
	compressed_size = get_zarr_size(f"./tmp_testdir/{label}_compressed.zarr")

Here we explicitly set the compression to None for the to_zarr method.

We should adapt this in the notebook as well.