Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Codec buffer limit error during reconstruction #460

Closed
dsundarraman opened this issue Nov 28, 2023 · 1 comment · Fixed by #463 · May be fixed by #461
Closed

Codec buffer limit error during reconstruction #460

dsundarraman opened this issue Nov 28, 2023 · 1 comment · Fixed by #463 · May be fixed by #461

Comments

@dsundarraman
Copy link
Collaborator

Hi @edyoshikun @talonchandler @ziw-liu ,

I'm running into the error below on using recorder CLI to reconstruct my zebrafish datasets.


(recordernew) [deepika.sundarraman@gpu-sm01-13 fish_1]$ recOrder reconstruct -i ./label_free_initial.zarr/0/0/0 -c /hpc/projects/balla_group/imaging/zebrafish_infection/falcon/20231117_caax_mpeg_gfp_coro1a_mcherry_9dpf/fish_1/birefringence-and-phase.yml -o ./reconstruction.zarr
Generating transfer functions and storing in transfer_function_birefringence-and-phase.zarr

Generating birefringence transfer function with settings:
swing: 0.1

Generating phase transfer function with settings:
wavelength_illumination: 0.532
yx_pixel_size: 0.209
z_pixel_size: 2.0
z_padding: 0
index_of_refraction_media: 1.3
numerical_aperture_detection: 0.75
numerical_aperture_illumination: 0.2
invert_phase_contrast: false

/home/deepika.sundarraman/.conda/envs/recordernew/lib/python3.9/site-packages/waveorder/optics.py:321: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  * torch.tensor(z_position_list)[:, None, None]
/home/deepika.sundarraman/.conda/envs/recordernew/lib/python3.9/site-packages/waveorder/optics.py:370: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  * torch.tensor(z_position_list)[:, None, None]
Traceback (most recent call last):
  File "/home/deepika.sundarraman/.conda/envs/recordernew/bin/recOrder", line 8, in <module>
    sys.exit(cli())
  File "/home/deepika.sundarraman/.conda/envs/recordernew/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/deepika.sundarraman/.conda/envs/recordernew/lib/python3.9/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/deepika.sundarraman/.conda/envs/recordernew/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/deepika.sundarraman/.conda/envs/recordernew/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/deepika.sundarraman/.conda/envs/recordernew/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/deepika.sundarraman/Documents/Githubs/recOrder/recOrder/cli/reconstruct.py", line 48, in reconstruct
    compute_transfer_function_cli(
  File "/home/deepika.sundarraman/Documents/Githubs/recOrder/recOrder/cli/compute_transfer_function.py", line 184, in compute_transfer_function_cli
    generate_and_save_phase_transfer_function(
  File "/home/deepika.sundarraman/Documents/Githubs/recOrder/recOrder/cli/compute_transfer_function.py", line 101, in generate_and_save_phase_transfer_function
    dataset[
  File "/home/deepika.sundarraman/.conda/envs/recordernew/lib/python3.9/site-packages/iohub/ngff.py", line 617, in __setitem__
    self.create_image(key, value)
  File "/home/deepika.sundarraman/.conda/envs/recordernew/lib/python3.9/site-packages/iohub/ngff.py", line 666, in create_image
    self._group.array(
  File "/home/deepika.sundarraman/.local/lib/python3.9/site-packages/zarr/hierarchy.py", line 1187, in array
    return self._write_op(self._array_nosync, name, data, **kwargs)
  File "/home/deepika.sundarraman/.local/lib/python3.9/site-packages/zarr/hierarchy.py", line 895, in _write_op
    return f(*args, **kwargs)
  File "/home/deepika.sundarraman/.local/lib/python3.9/site-packages/zarr/hierarchy.py", line 1193, in _array_nosync
    return array(data, store=self._store, path=path, chunk_store=self._chunk_store,
  File "/home/deepika.sundarraman/.local/lib/python3.9/site-packages/zarr/creation.py", line 397, in array
    z[...] = data
  File "/home/deepika.sundarraman/.local/lib/python3.9/site-packages/zarr/core.py", line 1388, in __setitem__
    self.set_basic_selection(pure_selection, value, fields=fields)
  File "/home/deepika.sundarraman/.local/lib/python3.9/site-packages/zarr/core.py", line 1483, in set_basic_selection
    return self._set_basic_selection_nd(selection, value, fields=fields)
  File "/home/deepika.sundarraman/.local/lib/python3.9/site-packages/zarr/core.py", line 1787, in _set_basic_selection_nd
    self._set_selection(indexer, value, fields=fields)
  File "/home/deepika.sundarraman/.local/lib/python3.9/site-packages/zarr/core.py", line 1839, in _set_selection
    self._chunk_setitem(chunk_coords, chunk_selection, chunk_value, fields=fields)
  File "/home/deepika.sundarraman/.local/lib/python3.9/site-packages/zarr/core.py", line 2106, in _chunk_setitem
    self._chunk_setitem_nosync(chunk_coords, chunk_selection, value,
  File "/home/deepika.sundarraman/.local/lib/python3.9/site-packages/zarr/core.py", line 2117, in _chunk_setitem_nosync
    self.chunk_store[ckey] = self._encode_chunk(cdata)
  File "/home/deepika.sundarraman/.local/lib/python3.9/site-packages/zarr/core.py", line 2244, in _encode_chunk
    cdata = self._compressor.encode(chunk)
  File "numcodecs/blosc.pyx", line 559, in numcodecs.blosc.Blosc.encode
  File "/home/deepika.sundarraman/.local/lib/python3.9/site-packages/numcodecs/compat.py", line 155, in ensure_contiguous_ndarray
    ensure_contiguous_ndarray_like(
  File "/home/deepika.sundarraman/.local/lib/python3.9/site-packages/numcodecs/compat.py", line 121, in ensure_contiguous_ndarray_like
    raise ValueError(msg)
ValueError: Codec does not support buffers of > 2147483647 bytes

The parameters I'm using, if relevant are:

Screenshot from 2023-11-28 13-36-15

Operating system: HPC
Python version: 3.9.18
Python environment (command line, IDE, Jupyter notebook, etc): command line

@talonchandler
Copy link
Collaborator

Thanks @dsundarraman.

@edyoshikun you said that you think this might be related to a fix that you made to the mantis repo. Was it this PR? czbiohub-sf/shrimPy#97

@edyoshikun could I request your help with unifying the create_empty_zarr functions between recOrder and mantis?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants