Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adding conversion to tensorstore #242

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 22 additions & 1 deletion iohub/ngff/nodes.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
import math
import os
from copy import deepcopy
from pathlib import Path
from typing import TYPE_CHECKING, Generator, Literal, Sequence, Type

import numpy as np
Expand Down Expand Up @@ -344,7 +345,27 @@ def downscale(self):
raise NotImplementedError

def tensorstore(self):
raise NotImplementedError
import tensorstore as ts
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs a docstring.


metadata = {
"dtype": self.dtype.str,
"shape": self.shape,
"chunks": self.chunks,
}
ts_spec = {
"driver": "zarr",
"kvstore": {
"driver": "file",
"path": str((Path(self._store.path) / self.path).resolve()),
},
"metadata": metadata,
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There could be a **ts_kwargs here to allow passing in additional configuration, for example a global cache context.

try:
zarr_dataset = ts.open(ts_spec, open=True).result()
except ValueError as e:
print(f"Error opening Zarr store: {e}")
raise
Comment on lines +365 to +367
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the purpose of this?

return zarr_dataset


class TiledImageArray(ImageArray):
Expand Down
6 changes: 5 additions & 1 deletion setup.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,9 @@ install_requires =
xarray>=2024.1.1

[options.extras_require]

tensorstore=
tensorstore>=0.1.64
dev =
black
flake8
Expand All @@ -53,7 +56,8 @@ dev =
hypothesis>=6.61.0
requests>=2.22.0
wget>=3.2
ome-zarr>=0.9.0
ome-zarr>=0.9.
ziw-liu marked this conversation as resolved.
Show resolved Hide resolved

doc =
matplotlib
numpydoc>=1.1.0
Expand Down
22 changes: 22 additions & 0 deletions tests/ngff/test_ngff.py
Original file line number Diff line number Diff line change
Expand Up @@ -250,6 +250,28 @@ def test_position_data(channels_and_random_5d, arr_name):
_ = dataset.data


@given(
channels_and_random_5d=_channels_and_random_5d(),
arr_name=short_alpha_numeric,
)
@settings(
max_examples=16,
deadline=2000,
suppress_health_check=[HealthCheck.data_too_large],
)
def test_ome_zarr_to_tensorstore(channels_and_random_5d, arr_name):
"""Test `iohub.ngff.Position.data` to tensortore"""
pytest.importorskip("tensorstore")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add tensorstore to dev requirements and not skip this?

channel_names, random_5d = channels_and_random_5d
with _temp_ome_zarr(random_5d, channel_names, "0") as dataset:
assert_array_almost_equal(dataset.data.numpy(), random_5d)
Comment on lines +266 to +267
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this redundant with another test case?

with pytest.raises(KeyError):
with _temp_ome_zarr(random_5d, channel_names, arr_name) as dataset:
t = dataset.data.tensorstore()
t.read().result()
del t


@given(
channels_and_random_5d=_channels_and_random_5d(),
arr_name=short_alpha_numeric,
Expand Down
Loading