Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reading nd datasets #966

Merged
merged 50 commits into from
Apr 2, 2024
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
6269a11
Started debuging for reading nd datasets.
markbader Nov 7, 2023
f891606
Implementation of VecNInt and NDBoundingBox.
markbader Nov 23, 2023
0681252
Merge branch 'master' into reading_nd_datasets
markbader Nov 23, 2023
7cb395e
Rename VecInt and fix some issues.
markbader Nov 27, 2023
b3c972e
Work on import of existing zarr dataset.
markbader Dec 21, 2023
fdf9c03
Merge branch 'master' into reading_nd_datasets
markbader Dec 21, 2023
89a38e6
Add nd_bounding_box to properties of Layer.
markbader Dec 21, 2023
5580631
Update hooks in properties.py to make import of 4d zarr datasets poss…
markbader Jan 4, 2024
c4d2838
Add axis_order and index of additional axes to nd_bounding box creation.
markbader Jan 8, 2024
cf36c58
Working on reading nd data with pims images.
markbader Jan 11, 2024
4033eb0
Modify pims images to support more _iter_dims.
markbader Jan 16, 2024
ae29105
Add method for expected bounding box instead of expected shape in pim…
markbader Jan 18, 2024
f7ffb6b
Adding functions for editing ndboundingbox in 3d and update import fr…
markbader Jan 18, 2024
21bb0ea
Propagade nd-bounding box to different methods that take care of writ…
markbader Jan 22, 2024
51631c0
Update object unstructuring for json and start implementing nd array …
markbader Jan 23, 2024
2a755d3
Adapt array classes and add axes information to ArrayInfo.
markbader Jan 25, 2024
58c728b
Updated zarr writing for nd arrays. Axes order still get mixed up bet…
markbader Jan 29, 2024
af249e7
Adapted buffered_slice_reader and test behaviour with different tif i…
markbader Jan 30, 2024
ad3b22e
Adding testdata and fix bugs with axes operation for writing zarr array.
markbader Jan 31, 2024
e975f08
Fixing issues with axis order.
markbader Feb 1, 2024
61146c9
Rewrite buffered_slice_writer and fix bugs to get old tests working.
markbader Feb 5, 2024
6fff638
Fix chunking in buffered slice writer.
markbader Feb 5, 2024
6e9efd6
Fix issues with old tests.
markbader Feb 6, 2024
0cb8241
Working on fixing all tests.
markbader Feb 8, 2024
bb16e83
Merge branch 'master' into reading_nd_datasets
markbader Feb 12, 2024
0d8a4f7
Fixing issues with alignment, writing with offsets and different mags.
markbader Feb 15, 2024
9fc0253
Fix reading without parameters for nd datasets and addint test.
markbader Feb 15, 2024
48f9586
Fix issue with empty additionalAxes in json and some minor issues.
markbader Feb 16, 2024
51f173c
Move script reading_nd_data to examples in docs and implement some fe…
markbader Feb 20, 2024
7340073
Do some formatting and implement requested changes.
markbader Feb 20, 2024
1a94714
Update naming for accessing attributes of nd bounding boxes.
markbader Feb 21, 2024
e1d522f
Fix buffered_slice_writer for different axis and typechecking.
markbader Feb 26, 2024
e614525
Fix issue with ensure_size in initialization.
markbader Feb 26, 2024
23b992a
Merge branch 'master' into reading_nd_datasets
markbader Feb 26, 2024
27d3a3d
Adapt pims_images to support xyz images with channels and timepoint.
markbader Feb 26, 2024
8a1ca27
run formatter
markbader Feb 26, 2024
4167b9b
Fix issues with failed tests and add comments.
markbader Mar 7, 2024
2f69c30
Fix statement with wrong VecInt initialization.
markbader Mar 8, 2024
31255e3
Insert previously deleted assertions.
markbader Mar 8, 2024
8bedec5
Merge branch 'master' into reading_nd_datasets
markbader Mar 11, 2024
c354dd6
Add docstrings and use bounding box for read in nd case instead of Ve…
markbader Mar 11, 2024
ffbe7f7
Implement requested changes.
markbader Mar 18, 2024
36fef79
Merge branch 'master' into reading_nd_datasets
markbader Mar 18, 2024
2bfe961
Changes init of NDBoundingBoxes.
markbader Mar 18, 2024
0575eb9
Add converter for VecInt attributes of nd bounding box to pass typech…
markbader Mar 18, 2024
2ce18df
Merge branch 'master' into reading_nd_datasets
markbader Mar 19, 2024
e6781d7
Merge branch 'master' into reading_nd_datasets
markbader Mar 28, 2024
1a51e9b
Enhance documentation.
markbader Mar 28, 2024
f21ac09
Update Changelog.md
markbader Mar 28, 2024
092af26
Merge branch 'master' into reading_nd_datasets
markbader Apr 2, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions webknossos/script_collection/reading_nd_data.py
markbader marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
from pathlib import Path

import pims

import webknossos as wk


def main(input_path: Path, output_path: Path) -> None:
"""Imports a dataset with more than 3 dimensions."""

wk.Dataset.from_images(
input_path, output_path, voxel_size=(10, 10, 10), data_format="zarr3"
)
# reader = pims.Bioformats(input_path)
# print(reader.sizes)
# print(isinstance(reader, pims.FramesSequenceND))


if __name__ == "__main__":
input_path = Path(".") / "webknossos" / "testdata" / "4D" # / "4D-series.ome.tif"
output_path = Path(".") / "webknossos" / "testoutput" / "4D"
main(input_path, output_path)
Binary file added webknossos/testdata/4D/4D-series.ome.tif
Binary file not shown.
1 change: 1 addition & 0 deletions webknossos/webknossos/dataset/_utils/pims_images.py
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,7 @@ def __init__(
self.dtype = images.dtype

if isinstance(images, pims.FramesSequenceND):
# TODO: assertion that phrohibites nd data
assert all(
axis in "xyzct" for axis in images.axes
), f"Found unknown axes {set(images.axes) - set('xyzct')}"
Expand Down
1 change: 1 addition & 0 deletions webknossos/webknossos/dataset/dataset.py
Original file line number Diff line number Diff line change
Expand Up @@ -1235,6 +1235,7 @@ def add_layer_from_images(
z_end = min(z_start + batch_size, pims_images.expected_shape.z)
# return shapes and set to union when using --pad
args.append((z_start, z_end))
print(args)
with warnings.catch_warnings():
# Block alignmnent within the dataset should not be a problem, since shard-wise chunking is enforced.
# However, dataset borders might change between different parallelized writes, when sizes differ.
Expand Down
69 changes: 17 additions & 52 deletions webknossos/webknossos/geometry/bounding_box.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,14 @@
import numpy as np

from .mag import Mag
from .nd_bounding_box import NDBoundingBox
from .vec3_int import Vec3Int, Vec3IntLike

_DEFAULT_BBOX_NAME = "Unnamed Bounding Box"


@attr.frozen
class BoundingBox:
class BoundingBox(NDBoundingBox):
"""
This class is used to represent an axis-aligned cuboid in 3D.
The top-left coordinate is inclusive and the bottom-right coordinate is exclusive.
Expand All @@ -32,6 +33,7 @@ class BoundingBox:

topleft: Vec3Int = attr.field(converter=Vec3Int)
size: Vec3Int = attr.field(converter=Vec3Int)
axes: Tuple[str, str, str] = attr.field(default=("x", "y", "z"))
markbader marked this conversation as resolved.
Show resolved Hide resolved
bottomright: Vec3Int = attr.field(init=False)
name: Optional[str] = _DEFAULT_BBOX_NAME
is_visible: bool = True
Expand All @@ -51,61 +53,38 @@ def __attrs_post_init__(self) -> None:
# it is needed.
object.__setattr__(self, "bottomright", self.topleft + self.size)

def with_topleft(self, new_topleft: Vec3IntLike) -> "BoundingBox":
return attr.evolve(self, topleft=new_topleft)
def with_additional_axis(self, name: str, extent: Tuple[int, int]) -> "NDBoundingBox":
assert (name not in ["x", "y", "z"]), f"The name '{name}' of the axis is already taken."
new_topleft = min(extent)
new_size = max(extent) - new_topleft

def with_size(self, new_size: Vec3IntLike) -> "BoundingBox":
return attr.evolve(self, size=new_size)

def with_name(self, name: Optional[str]) -> "BoundingBox":
return attr.evolve(self, name=name)

def with_is_visible(self, is_visible: bool) -> "BoundingBox":
return attr.evolve(self, is_visible=is_visible)

def with_color(
self, color: Optional[Tuple[float, float, float, float]]
) -> "BoundingBox":
return attr.evolve(self, color=color)
return NDBoundingBox(
topleft=(*self.topleft, new_topleft),
size=(*self.size, new_size),
axes=(*self.axes, name)
)

def with_bounds_x(
self, new_topleft_x: Optional[int] = None, new_size_x: Optional[int] = None
) -> "BoundingBox":
"""Returns a copy of the bounding box with topleft.x optionally replaced and size.x optionally replaced."""

new_topleft = (
self.topleft.with_x(new_topleft_x)
if new_topleft_x is not None
else self.topleft
)
new_size = self.size.with_x(new_size_x) if new_size_x is not None else self.size
return attr.evolve(self, topleft=new_topleft, size=new_size)
return cast(BoundingBox, self.with_bounds("x", new_topleft_x, new_size_x))

def with_bounds_y(
self, new_topleft_y: Optional[int] = None, new_size_y: Optional[int] = None
) -> "BoundingBox":
"""Returns a copy of the bounding box with topleft.y optionally replaced and size.y optionally replaced."""

new_topleft = (
self.topleft.with_y(new_topleft_y)
if new_topleft_y is not None
else self.topleft
)
new_size = self.size.with_y(new_size_y) if new_size_y is not None else self.size
return attr.evolve(self, topleft=new_topleft, size=new_size)
return cast(BoundingBox, self.with_bounds("y", new_topleft_y, new_size_y))

def with_bounds_z(
self, new_topleft_z: Optional[int] = None, new_size_z: Optional[int] = None
) -> "BoundingBox":
"""Returns a copy of the bounding box with topleft.z optionally replaced and size.z optionally replaced."""

new_topleft = (
self.topleft.with_z(new_topleft_z)
if new_topleft_z is not None
else self.topleft
)
new_size = self.size.with_z(new_size_z) if new_size_z is not None else self.size
return attr.evolve(self, topleft=new_topleft, size=new_size)
return cast(BoundingBox, self.with_bounds("z", new_topleft_z, new_size_z))


@classmethod
def from_wkw_dict(cls, bbox: Dict) -> "BoundingBox":
Expand Down Expand Up @@ -231,18 +210,7 @@ def to_csv(self) -> str:
return ",".join(map(str, self.to_tuple6()))

def __repr__(self) -> str:
return "BoundingBox(topleft={}, size={})".format(
str(tuple(self.topleft)), str(tuple(self.size))
)

def __str__(self) -> str:
return self.__repr__()

def __eq__(self, other: object) -> bool:
if isinstance(other, BoundingBox):
return self.topleft == other.topleft and self.size == other.size
else:
raise NotImplementedError()
return f"BoundingBox(topleft={self.topleft.to_tuple()}, size={self.size.to_tuple()})"

def padded_with_margins(
self, margins_left: Vec3IntLike, margins_right: Optional[Vec3IntLike] = None
Expand Down Expand Up @@ -427,9 +395,6 @@ def chunk(
):
yield BoundingBox([x, y, z], chunk_shape).intersected_with(self)

def volume(self) -> int:
return self.size.prod()

def slice_array(self, array: np.ndarray) -> np.ndarray:
return array[
self.topleft.x : self.bottomright.x,
Expand Down
Loading
Loading