Bfoptions

[will-moore]

Adds bfoptions command and --bfoptions arg to other commands to include creation of bfoptions file.

NB: contents of the file are hard-coded for now. Could be made more flexible in future if needed.

[will-moore]

With the last change above we want to test on idr0125-pilot with idr0004 data where we can test with/without the performance optimisations.

export IDRID=idr0004
for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  omero mkngff bfoptions /data/OMERO/ManagedRepository/ $fsid "/bia-integrator-data/$biapath/$uuid.zarr"
done

Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-49-38.885
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-49-38.885_mkngff/00d88a93-8d21-4a50-b8b5-60f11bcae0d3.zarr.bfoptions
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-34-48.864
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-34-48.864_mkngff/02c5d63f-36f5-4862-9682-ec3a2702a1e5.zarr.bfoptions
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-37-19.100
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-37-19.100_mkngff/06e3fba2-825a-441d-a3cb-2084515b1b14.zarr.bfoptions
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-43-41.108
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-43-41.108_mkngff/0bb5992f-e8d8-45b1-9e5d-d0fb8325aabb.zarr.bfoptions
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-10/01/11-45-16.436
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2015-10/01/11-45-16.436_mkngff/0d3e6be1-0c0a-42ef-8775-e3557c359b2d.zarr.bfoptions
...

$ cat /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-21-54.551_mkngff/f5ce45be-0b8c-4539-ae29-66978555f0ec.zarr.bfoptions
omezarr.list_pixels=false
omezarr.quick_read=true

[will-moore]

Updated ZarrReader on idr0125-pilot to todays merge build, => OMEZarrReader_p703_b625.jar.

Viewed Plate P105, image became viewable quite quickly... memo file generated - 66531 ms is 66 secs.

(base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ grep -A 2 97fde7f221dd /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A2 "saved memo"
2023-12-06 07:12:55,722 DEBUG [                   loci.formats.Memoizer] (l.Server-1) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2015-10/01/07-31-49.657_mkngff/1d37d3c1-08f2-42a9-8c61-97fde7f221dd.zarr/..zattrs.bfmemo (197423 bytes)
2023-12-06 07:12:55,723 DEBUG [                   loci.formats.Memoizer] (l.Server-1) start[1701846709191] time[66531] tag[loci.formats.Memoizer.setId]
2023-12-06 07:12:55,723 INFO  [                ome.io.nio.PixelsService] (l.Server-1) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-31-49.657_mkngff/1d37d3c1-08f2-42a9-8c61-97fde7f221dd.zarr/.zattrs Series: 6

For Plates P115 and P124 we want to disable the omezarr.quick_read=true option.
Edited...

(base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ sudo vi /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-46-42.965_mkngff/35cfc0db-7795-497c-aed5-1ae591b2d9f1.zarr.bfoptions 
(base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ sudo vi /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-57-40.271_mkngff/ee8872c8-e4b1-41fa-aa4f-a9e3e200c540.zarr.bfoptions 

(base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ cat /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-46-42.965_mkngff/35cfc0db-7795-497c-aed5-1ae591b2d9f1.zarr.bfoptions
omezarr.list_pixels=false

Viewing http://localhost:1040/webclient/?show=image-692975 was successful, taking 83939 ms or 84 secs.

(base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ grep -A 2 1ae591b2d9f1 /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A2 "saved memo"
2023-12-06 07:27:53,428 DEBUG [                   loci.formats.Memoizer] (l.Server-5) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2015-10/01/07-46-42.965_mkngff/35cfc0db-7795-497c-aed5-1ae591b2d9f1.zarr/..zattrs.bfmemo (216349 bytes)
2023-12-06 07:27:53,428 DEBUG [                   loci.formats.Memoizer] (l.Server-5) start[1701847589488] time[83939] tag[loci.formats.Memoizer.setId]
2023-12-06 07:27:53,428 INFO  [                ome.io.nio.PixelsService] (l.Server-5) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-46-42.965_mkngff/35cfc0db-7795-497c-aed5-1ae591b2d9f1.zarr/.zattrs Series: 28

[will-moore]

Build error due to python 3.6 not supporting Hatchling:

      ERROR: Could not find a version that satisfies the requirement hatchling>=1.8.0 (from versions: 0.8.0, 0.8.1, 0.8.2, 0.9.0, 0.10.0, 0.11.0, 0.11.1, 0.11.2, 0.11.3, 0.12.0, 0.13.0, 0.14.0, 0.15.0, 0.16.0, 0.17.0, 0.18.0, 0.19.0, 0.20.0, 0.20.1, 0.21.0, 0.21.1, 0.22.0, 0.23.0, 0.24.0, 0.25.0, 0.25.1)
      ERROR: No matching distribution found for hatchling>=1.8.0

In fact, none of the releases I tried at https://pypi.org/project/hatchling/#history supported python 3.6!

[joshmoore]

Nice!

[joshmoore]

Do you ever run this without "--bfoptions"?

[will-moore]

Hmmm - good point... probably not! I think we always need the .bfoptions file alongside the symlinks since we currently always exclude the chunks from new Filesets.
So, I guess I'll remove this option from the symlink command then...