Provide "NULL" shards in annotations

[erjel]

Problem

We have again a problem with the merging of annotations with the corresponding fallback layer:

If you delete a complete shard while creating a volume layer annotation in webknossos, there is currently no way of knowing whether this shard should be
a) deleted
b) was untouched by the annotation

Example

I.e. if you create a dataset like this:

import webknossos as wk
import numpy as np

data = np.ones((64, 64, 64), dtype=np.uint8)
wk.Dataset('/tmp/ds_wkw', voxel_size=(1,)*3)\
    .add_layer('seg', wk.SEGMENTATION_CATEGORY, largest_segment_id=1, dtype_per_channel=np.uint8)\
    .add_mag(1, chunk_shape=(32, 32, 32), chunks_per_shard=(1,1,1))\
    .write(data)

and create an annotation in webknossos, where you set all pixels in the first shard to zero
659bef24010000ab03ebc433.zip
the information is (afaik) not stored anywhere in the downloaded zip file.

This becomes also obvious in the provided gist for file merging
https://gist.github.com/philippotto/fcba429c646d34733d045a14b390a17a#file-merge-annotation-with-fallback-layer-py-L40-L42
where you iterate over the available annotation shards on disk, to detect shards which need to be updated.

If I do this on the provided annotation however

with wk.Annotation.load('659bef24010000ab03ebc433.zip').temporary_volume_layer_copy() as annot_layer:
    bboxes = list(annot_layer.get_finest_mag().get_bounding_boxes_on_disk())

I see that only 7 out of 8 shards are present. This makes perfectly sense if we talk about datasets where all empty chunks can be skipped. For annotations however, we need the information preserved for the merging with the fallback layer. Otherwise we cannot distinguish between shards which have been deleted, or not been modified at all. In our case this results in shard-shaped artifacts in the merging process

import matplotlib.pyplot as plt
plt.imshow(data[0, :, :, 2])

Possible Solutions

If a shard has been deleted completely during an annotation, include the corresponding shard in the download process (or a special NULL shard).

[fm3]

Thanks for your detailed report! I think you are completely right, this information is lost in the zip download.

Since this needs a server-side change in webknossos, I created a separate issue at scalableminds/webknossos#7532

[fm3]

I just deployed a new server version at webknossos.org where zeroed chunks/shards should now be included in volume annotation downloads. For reference: scalableminds/webknossos#7576

Please re-download the zipfiles where you had this problem previously and try again.

I’m closing this issue, feel free to reopen if you have more problems with this :-)

[erjel]

Hi @fm3,

thanks for quickly fixing this. However I can still reproduce the error on our own instance (Version 25799).
I cannot reopen this issue since I am not a maintainer.

Eric

[fm3]

Hi @erjel I see, I had assumed you’d run this against webknossos.org.

The fix is only included in server version 26354 or newer. It is also included in the latest release 24.02.0.

Maybe your server could be upgraded?

[erjel]

Ok I see, I just assumed
a) That if I see this on the server

this is also the release number of the server...
b) That version 25799 is newer than 24.02.0

[erjel]

Btw: How can I map from server version to release in general?

[fm3]

Btw: How can I map from server version to release in general?

If a release version is deployed, the version in the menu will report the release number. The case that the raw build number is displayed only happens if versions are deployed that do not directly correspond to a release. I just checked with norman and this was the case for your webknossos server instance because of another bugfix that came in-between release schedules.

I’m afraid there is not currently an easy available way to find out how a build number fits in our version history. However, I can offer /api/buildinfo. Visit that uri on your webknossos instance to see the commitDate, which provides info about how recent a deployed version is. (Disclaimer: do not rely on this route programmatically, its format may change in the future).

Hope this helps!