idr-next steps for NGFF upgrade #675

will-moore · 2023-10-10T06:07:57Z

will-moore · 2023-10-10T11:25:00Z

Delete on idr-testing...

(venv3) [wmoore@test120-omeroreadwrite ~]$ omero delete Plate:4801 --report > delete_report.log
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public

omero.cmd.Delete2 Plate:4801 ok
Steps: 6
Elapsed time: 129.884 secs.
Flags: []
Deleted objects
  CommentAnnotation:1502861
  FilesetAnnotationLink:20901
  MapAnnotation:8368853
  PlateAnnotationLink:3493
  Channel:4731551-4733530
  Image:1958101-1958496
  LogicalChannel:38501
  OriginalFile:6948775-6988376
  Pixels:1958101-1958496
  ChannelBinding:4269391-4271370
  QuantumDef:1750202-1750597
  RenderingDef:1750202-1750597
  Thumbnail:1738526-1738921,4222134,4222136-4222139,4222144,4222146,4222165-
....
4222496,4222499,4222500,4255156-4255161,4255163-4255199,4255203
  Fileset:21101
  FilesetEntry:6810401-6848218,6848228,6848241
.....
6861554,6861559,6861564,6861569,6861574,6861579,6861584,6861590
  FilesetJobLink:99401-99405
  IndexingJob:106219
  JobOriginalFileLink:33515
  MetadataImportJob:106216
  PixelDataJob:106217
  ThumbnailGenerationJob:106218
  UploadJob:106215
  Plate:4801
  ScreenPlateLink:5001
  Well:1056601-1056996
  WellSample:1762101-1762496
  StatsInfo:3976732-3978711

will-moore · 2023-10-10T11:31:25Z

sudo -u root -s
source /opt/omero/server/venv3/bin/activate
pip install 'omero-mkngff @ git+https://github.com/IDR/omero-mkngff@main'

will-moore · 2023-10-10T11:36:05Z

[wmoore@test120-omeroreadwrite ~]$ cd /uod/idr/metadata/idr-utils/
[wmoore@test120-omeroreadwrite idr-utils]$ sudo -Es git fetch will
[wmoore@test120-omeroreadwrite idr-utils]$ sudo -Es git checkout will/filesets_to_swap

will-moore · 2023-10-10T11:51:44Z

Latest Bio-Formats push: https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-push/643/console

2023-10-10 00:05:33,744 [   scc.merge] INFO  Repository: ome/ZarrReader
2023-10-10 00:05:33,744 [   scc.merge] INFO  Excluded PRs:
2023-10-10 00:05:33,744 [   scc.merge] INFO    - PR 53 dgault 'Reorder group keys' (exclude comment)
2023-10-10 00:05:33,744 [   scc.merge] INFO    - PR 20 dgault 'ZarrReader: Use chunk API' (exclude comment)
2023-10-10 00:05:33,744 [   scc.merge] INFO  Already up-to-date.
2023-10-10 00:05:33,744 [   scc.merge] INFO  
2023-10-10 00:05:33,744 [   scc.merge] INFO  Merged PRs:
2023-10-10 00:05:33,745 [   scc.merge] INFO    - PR 62 joshmoore 'Update README.md'
2023-10-10 00:05:33,745 [   scc.merge] INFO    - PR 64 dgault 'ZarrReader performance improvements'
2023-10-10 00:05:33,745 [   scc.merge] INFO    - PR 65 dgault 'Bumping Bio-Formats version to 7.0.0'

Download from https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/565/default/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar

Run on omeroreadwrite, omeroreadonly-1 - 4...

sudo -u omero-server -s
cd
wget https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/565/default/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar
mv OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar OMEZarrReader_p643_b565.jar
rm /opt/omero/server/OMERO.server/lib/client/OMEZarrReader.jar && rm /opt/omero/server/OMERO.server/lib/server/OMEZarrReader.jar
cp OMEZarrReader_p643_b565.jar /opt/omero/server/OMERO.server/lib/client/ && cp OMEZarrReader_p643_b565.jar /opt/omero/server/OMERO.server/lib/server/
# as wmoore
sudo service omero-server restart

will-moore · 2023-10-10T13:41:11Z

On omeroreadwrite...

as omero-server, screen -S mkngff... - get SECRET
as wmoore...

cd /uod/idr/metadata/idr-utils/scripts/ngff_filesets

for dir in $(ls -d */); do
  cd $dir
  for i in $(ls); do sudo sed -i 's/SECRETUUID/1def0359-65xyz-443f-b14a-69f04f569f4d/g' $i; done
  cd ../;
done

as omero-server, screen -S mkngff...

omero login
cd /uod/idr/metadata/idr-utils/scripts/ngff_filesets

export IDRID=idr0004
for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
  omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/bia-integrator-data/$biapath/$uuid.zarr"
done
...
UPDATE 95
BEGIN
 mkngff_fileset 
----------------
        6312382
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-49-38.885
Creating dir at /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-49-38.885_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-49-38.885_mkngff/00d88a93-8d21-4a50-b8b5-60f11bcae0d3.zarr -> /bia-integrator-data/S-BIAD867/00d88a93-8d21-4a50-b8b5-60f11bcae0d3/00d88a93-8d21-4a50-b8b5-60f11bcae0d3.zarr
UPDATE 98
BEGIN
 mkngff_fileset 
----------------
        6312383
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-34-48.864
...

will-moore · 2023-10-11T09:30:08Z

Ran for all studies up to idr0016 yesterday...
Today...

idr0025 9:47
idr0026 9:48
idr0033 9:52
idr0036 10:12
idr0051 10:23
idr0054 10:24
idr0090 10:25
idr0091 10:29 (done at 10:50)

error from idr0054 because idr0054.csv has whitespace in rows - but shouldn't cause any errors as the correct IDs were still found?
/opt/omero/server/venv3/bin/omero mkngff symlink: error: argument fileset_id: invalid int value: 'idr0054/Tonsil'

will-moore · 2023-10-15T21:09:21Z

Saved memo times for the first Fileset in each study...

e.g.

[wmoore@test120-omeroreadwrite ~]$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log.4 | grep -A 2 "81ba6c774fdd"
2023-10-11 14:17:21,940 DEBUG [                   loci.formats.Memoizer] (l.Server-3) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-20/2020-10/02/14-41-08.138_mkngff/971f2809-c748-4259-8044-81ba6c774fdd.zarr/..zattrs.bfmemo (91291 bytes)
2023-10-11 14:17:21,940 DEBUG [                   loci.formats.Memoizer] (l.Server-3) start[1697033824100] time[17840] tag[loci.formats.Memoizer.setId]
2023-10-11 14:17:21,941 INFO  [                ome.io.nio.PixelsService] (l.Server-3) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-20/2020-10/02/14-41-08.138_mkngff/971f2809-c748-4259-8044-81ba6c774fdd.zarr/.zattrs Series: 0

idr0004 139197ms (2.3 mins)
idr0010 694251ms (11.5 mins)
idr0011 68187ms (1.1 mins)
idr0012 1248415ms (20.8 mins)
idr0013 1125667ms (18.7 mins)
idr0015 1163426ms (19.3 mins)
idr0016 3221519ms (53.7 mins)
idr0025 543319ms (9.1 mins)
idr0026 22904ms (0.4 mins)
idr0033 7688852ms (128 mins)
idr0035 451787ms (7.5 mins)
idr0036 7228099ms (120 mins)
idr0051 578885ms (9.6 mins)
idr0054 2476ms (0.04 mins)
idr0090 6702958ms (111 mins)
idr0091 17840ms (0.3 mins)

will-moore · 2023-11-01T15:56:44Z

As omero-server...
With IDR/omero-mkngff@7dc8222 installed
And the ZarrReader that supports bfoptions updated ome/ZarrReader#64 (comment)

omero login
cd /uod/idr/metadata/idr-utils/scripts/ngff_filesets

export IDRID=idr0004
for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  omero mkngff bfoptions /data/OMERO/ManagedRepository $fsid "/bia-integrator-data/$biapath/$uuid.zarr"
done
...
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-33-48.168
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-33-48.168_mkngff/f791de8d-cd01-4303-8b08-67cbbbb45b64.zarr.bfoptions
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-02-58.826
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-02-58.826_mkngff/fcc7c91a-f0e6-43f4-93a6-220e9224eda5.zarr.bfoptions

This now allows viewing of all idr0004.
Repeat for others...

All completed!

will-moore · 2023-11-03T16:36:53Z

Using IDR/idr-utils#55, let's validate a bunch of Plates etc.. 1st plate from each study...

First plate IDs from idr0004 etc.

omero login
cd /uod/idr/metadata/idr-utils/scripts

for i in 1751 4501 5387 4287 3451 4295 6208 6001 5958 6301 4403 9301; do 
echo Plate:$i
python check_pixels.py Plate:$i /tmp/check_pix_$i.log --max-planes=1; done
...

tail -f /tmp/check_pix_4287.log 
73/672 Check Image:1811320 HT01 [Well M21, Field 1]
74/672 Check Image:1811321 HT01 [Well M21, Field 2]
Error: Mismatch for Image: 1811321 at plane (z, c, t): (0, 0, 0)
75/672 Check Image:1811322 HT01 [Well P5, Field 1]
76/672 Check Image:1811323 HT01 [Well P5, Field 2]
Error: Mismatch for Image: 1811323 at plane (z, c, t): (0, 0, 0)
77/672 Check Image:1811324 HT01 [Well M15, Field 1]
78/672 Check Image:1811325 HT01 [Well M15, Field 2]
Error: Mismatch for Image: 1811325 at plane (z, c, t): (0, 0, 0)
79/672 Check Image:1811326 HT01 [Well E20, Field 1]
80/672 Check Image:1811327 HT01 [Well E20, Field 2]
Error: Mismatch for Image: 1811327 at plane (z, c, t): (0, 0, 0)

will-moore · 2023-11-06T13:26:23Z

for i in 1751 4501 5387 4287 3451 4295 6208 6001 5958 6301 4403 9301; do 
echo Plate:$i
grep -c Mismatch /tmp/check_pix_$i.log; done

Plate:1751 (idr0004 1-field)
0
Plate:4501 (idr0010 1-field)
0
Plate:5387 (idr0011 1-field)
0
Plate:4287 (idr0012 2-fields)
336
Plate:3451 (idr0013 1-field)
0
Plate:4295 (idr0015 1-field)
0
Plate:6208 (idr0016 6-fields) omero-cli-zarr
0
Plate:6001 (idr0025 4-fields)
288
Plate:5958 (idr0033 9-fields)
3072
Plate:6301 (id0035 4-fields)
180
Plate:4403 (idr0036 9-fields) omero-cli-zarr
0
Plate:9301. (idr0090 32-fields) omero-cli-zarr
0

The Errors come from all Fields except the first field in multi-field Plates. Only with bioformats2raw Plates. No issues with omero-cli-zarr.
All the error counts for the 4 plates above exactly match Wells x (fields - 1).
When viewing images in these plates, all appear fine.

For a single Image - on idr-testing, connect to localhost or idr.openmicroscopy.org.
Compare same images...

http://localhost:1080/webclient/?show=image-1811321
https://idr.openmicroscopy.org/webclient/?show=image-1811321

>>> conn = BlitzGateway("public", "public", host="localhost", port=4064)
>>> client = omero.client("idr.openmicroscopy.org", 4064)
>>> sf = client.createSession("public", "public")
>>> conn2 = BlitzGateway(client_obj=client)
>>> image = conn.getObject('Image', 1811321)
>>> plane = image.getPrimaryPixels().getPlane(0, 0, 0)
>>> image2 = conn2.getObject('Image', 1811321)
>>> plane2 = image2.getPrimaryPixels().getPlane(0, 0, 0)
>>> plane
array([[1596, 1602, 1625, ..., 1738, 1738,  586],
       [1634, 1641, 1614, ..., 1689, 1758,  586],
       [1746, 1717, 1723, ..., 1631, 1654,  586],
       ...,
       [1288, 1351, 1375, ..., 1243, 1275,  602],
       [1276, 1272, 1290, ..., 1225, 1250,  600],
       [ 618,  620,  618, ...,  594,  596,  600]], dtype=uint16)
>>> plane2
array([[1260, 1416, 1529, ...,  666,  671,  475],
       [ 993, 1205, 1362, ...,  647,  648,  475],
       [ 944,  955, 1048, ...,  644,  673,  475],
       ...,
       [1969, 2061, 2049, ..., 1375, 1409,  486],
       [1928, 1964, 2039, ..., 1468, 1455,  485],
       [ 498,  499,  498, ...,  481,  482,  485]], dtype=uint16)

Pixel values are different!

With the same rendering settings, these images do appear a little different:

Need to check the raw NGFF data...

will-moore · 2023-11-06T13:52:52Z

>>> url = "https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD845/409227a9-2f3f-421c-8086-b14253283e42/409227a9-2f3f-421c-8086-b14253283e42.zarr/M/21/1/"
>>> from ome_zarr.io import parse_url
>>> from ome_zarr.reader import Reader
>>> store = parse_url(url, mode="r").store
>>> reader = Reader(parse_url(url))
>>> nodes = list(reader())
>>> len(nodes)
1
>>> image_node = nodes[0]
>>> dask_data = image_node.data
>>> data = dask_data[0].compute()
>>> data.shape
(1, 3, 1, 1024, 1344)
>>> data[0,0,0]
array([[1260, 1416, 1529, ...,  666,  671,  475],
       [ 993, 1205, 1362, ...,  647,  648,  475],
       [ 944,  955, 1048, ...,  644,  673,  475],
       ...,
       [1969, 2061, 2049, ..., 1375, 1409,  486],
       [1928, 1964, 2039, ..., 1468, 1455,  485],
       [ 498,  499,  498, ...,  481,  482,  485]], dtype='>u2')

These pixel values appear identical to those loaded from idr.openmicroscopy.org, so it looks like the NGFF data is correct and the differences we're seeing on idr-testing are likely due to the ZarrReader.

will-moore · 2023-11-06T15:02:32Z

Updated ZarrReader on idr-testing - see ome/ZarrReader#64 (comment)

Generating memo file for previously un-viewed HT03 Plate:4289 from idr0012:
http://localhost:1080/webclient/?show=image-1813262 ...

will-moore · 2023-11-06T21:19:57Z

python check_pixels.py Plate:4289 /tmp/check_pix_4289.log

No images have pixel mismatch (checking all 3 planes for all images).
Issue above is fixed!

Known issue - all the C row of images have sizeC=1

$ grep -c Mismatch /tmp/check_pix_4289.log
0

grep  Error /tmp/check_pix_4289.log
...
Error: Image:1813207 Z '1' greater than sizeZ '1'.
Error: Image:1813212 Z '1' greater than sizeZ '1'.
Error: Image:1813213 Z '1' greater than sizeZ '1'.

grep -c Error /tmp/check_pix_4289.log
42

will-moore · 2023-11-06T21:43:22Z

Since images without chunks didn't import above, test import of idr0054 image from ome/ZarrReader#64 (comment)

$ omero import -d 18153 --depth=100 a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr --file /tmp/idr0054_20231106.log --errs /tmp/idr0054_20231106.err

...
-4600-b114-7b4697900e39.zarr/OME/.zgroup
2023-11-06 21:40:04,010 9144       [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /opt/omero/server/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/.zgroup
2023-11-06 21:40:04,340 9474       [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
2023-11-06 21:40:04,466 9600       [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 61520472
2023-11-06 21:40:05,084 10218      [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_IMPORTED Step: 1 of 5  Logfile: 61520472
2023-11-06 21:40:12,432 17566      [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - PIXELDATA_PROCESSED Step: 2 of 5  Logfile: 61520472
2023-11-06 21:40:12,959 18093      [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - THUMBNAILS_GENERATED Step: 3 of 5  Logfile: 61520472
2023-11-06 21:40:13,001 18135      [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_PROCESSED Step: 4 of 5  Logfile: 61520472
2023-11-06 21:40:13,042 18176      [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5  Logfile: 61520472
2023-11-06 21:40:13,165 18299      [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /opt/omero/server/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml
Other imported objects:
Fileset:6314645

==> Summary
12 files uploaded, 1 fileset created, 1 image imported, 0 errors in 0:00:15.205

So this still works 👍

will-moore · 2023-12-06T09:45:38Z

idr-next (prod120) steps...

Install mkngff from latest bfoptions branch...

ssh idr-next.openmicroscopy.org -At ssh omeroreadwrite -A
[wmoore@prod120-omeroreadwrite ~]$ sudo -u root -s
[root@prod120-omeroreadwrite wmoore]# source /opt/omero/server/venv3/bin/activate
(venv3) [root@prod120-omeroreadwrite wmoore]# pip install 'omero-mkngff @ git+https://github.com/will-moore/omero-mkngff@bfoptions'

Before running sql scripts on idr-next, let's remove all the known Filesets that are not ready (being re-submitted to EBI)...
18 removed from csv files... IDR/idr-utils@125c4e5

$ git diff | grep '\-idr'
-idr0004/P132.ome.zarr,S-BIAD867/3df515f9-75c8-4cc4-8c29-480ae9817880,13125
-idr0011/Plate15-Yellow-A.ome.zarr,S-BIAD866/0e5e1147-449f-4a89-96c6-3a3c85f8c1a7,21851
-idr0011/Plate13-Red-B.ome.zarr,S-BIAD866/36e53731-585a-4b73-9500-8e8b087b1022,21842
-idr0011/Plate3-Yellow-B.ome.zarr,S-BIAD866/8ba28efa-377f-4602-8c09-41737075ff08,21789
-idr0011/Plate11-Blue-A.ome.zarr,S-BIAD866/bd01548b-da85-4a66-8422-7b16995fabba,21834
-idr0013/LT0080_37.ome.zarr,S-BIAD865/8387705b-16bf-4b14-8884-426b0c16dfff,18655
-idr0013/LT0066_23.ome.zarr,S-BIAD865/dab29e5a-d36f-430a-a9ff-7a1d6e4ce299,18568
-idr0013/LT0103_13.ome.zarr,S-BIAD865/df947dfe-ed8f-4dda-a20a-fb9f3a717b47,18728
-idr0015/TARA_HCS1_H5_G100008060_G100008062--2013_11_03_22_51_32_chamber--U00--V01.ome.zarr,S-BIAD861/d69df538-4684-4b32-8ded-d2f2af43af9f,21118
-idr0015/TARA_HCS1_H5_G100010241_G100010731--2013_09_29_19_14_59_chamber--U00--V01.ome.zarr,S-BIAD861/0cc5dbe3-444a-4ea2-a335-b51cf89c1c53,21160
-idr0015/TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.ome.zarr,S-BIAD861/1a29207c-d50b-48b7-a7c0-54c6252bfd9c,21112
-idr0016/25585.ome.zarr,S-BIAD851/1b63ff26-391a-4b44-995c-83b7cbf5372f,21365
-idr0016/24634.ome.zarr,S-BIAD851/22803be2-9732-41ab-b9bf-fcd37c3b3b84,21278
-idr0016/24667.ome.zarr,S-BIAD851/2c49b893-ec6d-4329-9cc3-569b820075f2,23603
-idr0016/25909.ome.zarr,S-BIAD851/382abcd4-404b-4796-bbe2-b02fedcbb015,21476
-idr0016/25592.ome.zarr,S-BIAD851/481a1ae0-a492-451f-bf7d-fa2c106c7cf5,21369
-idr0016/25675.ome.zarr,S-BIAD851/5aec8bec-8573-44ec-9e9e-24fb81623fbe,21385
-idr0016/25740.ome.zarr,S-BIAD851/f8583882-cfa6-4409-b452-50b849b1397e,21454

NB: Also 1 plate from idr0011 that was never created

idr0011/LT0080_27

Checkout idr-utils filesets_to_swap branch (NB: this will eventually be moved to a separate repo and archived, instead of being merged into idr-utils)...

[wmoore@prod120-omeroreadwrite ~]$ cd /uod/idr/metadata/idr-utils/
[wmoore@prod120-omeroreadwrite idr-utils]$ sudo -Es git fetch will
[wmoore@prod120-omeroreadwrite idr-utils]$ sudo -Es git checkout will/filesets_to_swap
...
HEAD is now at 125c4e5... Remove Filesets from ngff_filesets/ csv files if not ready

Delete duplicate plate on idr0015: Log looks exactly the same as above: idr-next steps for NGFF upgrade #675 (comment)

(venv3) bash-4.2$ omero delete Plate:4801 --report > /tmp/idr0015_delete_report.log
...
omero.cmd.Delete2 Plate:4801 ok
Steps: 6
Elapsed time: 133.92 secs.
Flags: []
Deleted objects
  CommentAnnotation:1502861
  FilesetAnnotationLink:20901
  MapAnnotation:8368853
  PlateAnnotationLink:3493
  Channel:4731551-4733530
  Image:1958101-1958496
...

Run sql scripts... as omero-server user...

(venv3) bash-4.2$ omero config get | grep omero.db.host
(venv3) bash-4.2$ export DBHOST=***
(venv3) bash-4.2$ omero config get --show-password | grep omero.db.pass
omero.db.pass=*****
(venv3) bash-4.2$ export PGPASSWORD=****
***

as wmoore - repeat for all studiess

cd /uod/idr/metadata/idr-utils/scripts/ngff_filesets/idr0004
for i in $(ls); do sudo sed -i 's/SECRETUUID/****/g' $i; done

as omero-server in screen -S mkngff

$ omero mkngff setup > setup.sql
$ psql -U omero -d idr -h $DBHOST -f setup.sql 
CREATE FUNCTION

$ export IDRID=idr0004
$ cd /uod/idr/metadata/idr-utils/scripts/ngff_filesets
$ for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
  omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/bia-integrator-data/$biapath/$uuid.zarr" --bfoptions
done

gff symlink /data/OMERO/ManagedRepository $fsid "/bia-integrator-data/$biapath/$uuid.zarr" --bfoptions
> done^C
(venv3) bash-4.2$
(venv3) bash-4.2$
(venv3) bash-4.2$ 
(venv3) bash-4.2$ omero login
Previous session expired for demo on localhost:4064
Server: [localhost:4064]
Username: [demo]
Password:
Created session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
(venv3) bash-4.2$ 
(venv3) bash-4.2$ pwd
/uod/idr/metadata/idr-utils/scripts/ngff_filesets
(venv3) bash-4.2$ 
(venv3) bash-4.2$ for r in $(cat $IDRID.csv); do
>   biapath=$(echo $r | cut -d',' -f2)
>   uuid=$(echo $biapath | cut -d'/' -f2)
>   fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
>   psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
>   omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/bia-integrator-data/$biapath/$uuid.zarr" --bfoptions
> done
UPDATE 95
BEGIN
 mkngff_fileset 
----------------
        6312382
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-49-38.885
Creating dir at /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-49-38.885_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-49-38.885_mkngff/00d88a93-8d21-4a50-b8b5-60f11bcae0d3.zarr -> /bia-integrator-data/S-BIAD867/00d88a93-8d21-4a50-b8b5-60f11bcae0d3/00d88a93-8d21-4a50-b8b5-60f11bcae0d3.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-49-38.885
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-49-38.885_mkngff/00d88a93-8d21-4a50-b8b5-60f11bcae0d3.zarr.bfoptions
UPDATE 98
BEGIN
 mkngff_fileset 
----------------
        6312383
(1 row)

COMMIT
...

Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-33-48.168
Creating dir at /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-33-48.168_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-33-48.168_mkngff/f791de8d-cd01-4303-8b08-67cbbbb45b64.zarr -> /bia-integrator-data/S-BIAD867/f791de8d-cd01-4303-8b08-67cbbbb45b64/f791de8d-cd01-4303-8b08-67cbbbb45b64.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-33-48.168
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-33-48.168_mkngff/f791de8d-cd01-4303-8b08-67cbbbb45b64.zarr.bfoptions
UPDATE 88
BEGIN
 mkngff_fileset 
----------------
        6312426
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-02-58.826
Creating dir at /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-02-58.826_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-02-58.826_mkngff/fcc7c91a-f0e6-43f4-93a6-220e9224eda5.zarr -> /bia-integrator-data/S-BIAD867/fcc7c91a-f0e6-43f4-93a6-220e9224eda5/fcc7c91a-f0e6-43f4-93a6-220e9224eda5.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-02-58.826
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-02-58.826_mkngff/fcc7c91a-f0e6-43f4-93a6-220e9224eda5.zarr.bfoptions

manually updated bfoptions for idr0004 plates P115 and P124 to remove read_fast option

vi /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-46-42.965_mkngff/35cfc0db-7795-497c-aed5-1ae591b2d9f1.zarr.bfoptions
vi /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-57-40.271_mkngff/ee8872c8-e4b1-41fa-aa4f-a9e3e200c540.zarr.bfoptions

Repeat for:
- idr0010
- ...
- idr0036...

(venv3) bash-4.2$ export IDRID=idr0036
(venv3) bash-4.2$ 
(venv3) bash-4.2$ for r in $(cat $IDRID.csv); do   biapath=$(echo $r | cut -d',' -f2);   uuid=$(echo $biapath | cut -d'/' -f2);   fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]');   psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql";   omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/bia-integrator-data/$biapath/$uuid.zarr" --bfoptions; done
UPDATE 3456
BEGIN
psql:idr0036/21190.sql:21533: ERROR:  cannot set original repo property without secret key
CONTEXT:  PL/pgSQL function _protect_originalfile_repo_insert() line 28 at RAISE
SQL statement "insert into originalfile
          (id, permissions, creation_id, group_id, owner_id, update_id, mimetype, repo, path, name)
          values (nextval('seq_originalfile'), old_perms, new_event, old_group, old_owner, new_event,
            info[i][3], repo, info[i][1], uuid || info[i][2])
          returning id"
PL/pgSQL function mkngff_fileset(bigint,character varying,character varying,character varying,text[]) line 42 at SQL statement
ROLLBACK
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
^CCancelled
UPDATE 3456
BEGIN
psql:idr0036/21183.sql:21533: ERROR:  cannot set original repo property without secret key
CONTEXT:  PL/pgSQL function _protect_originalfile_repo_insert() line 28 at RAISE
SQL statement "insert into originalfile
          (id, permissions, creation_id, group_id, owner_id, update_id, mimetype, repo, path, name)
          values (nextval('seq_originalfile'), old_perms, new_event, old_group, old_owner, new_event,
            info[i][3], repo, info[i][1], uuid || info[i][2])
          returning id"
PL/pgSQL function mkngff_fileset(bigint,character varying,character varying,character varying,text[]) line 42 at SQL statement
ROLLBACK

Ooops - missed SECRET key for idr0036...
Cancelled after 1st ROLLBACK above. Re-run that one command:

omero mkngff symlink /data/OMERO/ManagedRepository 21190 "/bia-integrator-data/S-BIAD855/02dc024f-2228-4d19-923e-0d55c1224ba3/02dc024f-2228-4d19-923e-0d55c1224ba3.zarr" --bfoptions
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/14/09-46-41.391
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-06/14/09-46-41.391_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-06/14/09-46-41.391_mkngff/02dc024f-2228-4d19-923e-0d55c1224ba3.zarr -> /bia-integrator-data/S-BIAD855/02dc024f-2228-4d19-923e-0d55c1224ba3/02dc024f-2228-4d19-923e-0d55c1224ba3.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/14/09-46-41.391
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2016-06/14/09-46-41.391_mkngff/02dc024f-2228-4d19-923e-0d55c1224ba3.zarr.bfoptions

Need to re-run psql commands ONLY, after updating SECRET.. looks god...

$ for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
done

UPDATE 3456
BEGIN
 mkngff_fileset 
----------------
        6314042
(1 row)

COMMIT
UPDATE 3456
BEGIN
 mkngff_fileset 
----------------
        6314043
(1 row)

COMMIT
UPDATE 3456
BEGIN
 mkngff_fileset 
----------------
        6314044
(1 row)

COMMIT
UPDATE 3456
BEGIN
 mkngff_fileset 
----------------
        631404
...

ALL studies done!!!

will-moore · 2023-12-11T19:49:19Z

Memo file generation progress (running since 9th Dec - Now 11th Dec):
853 Filesets done... 9 errors so far... Known issues, waiting on EBI resubmission...

EDIT: updated 14th Dec after stopped running

[wmoore@prod120-proxy ~]$ ls /tmp/cache-20231209/1/ | wc
   1067    1067   14892

[wmoore@prod120-proxy ~]$ grep Error /tmp/cache-20231209/1/*/*
/tmp/cache-20231209/1/Image:1556033/stdout:fail: 1556033 1.7507498264312744 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:1573071/stdout:fail: 1573071 1.6358556747436523 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:1600787/stdout:fail: 1600787 1.5452053546905518 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:1949093/stdout:fail: 1949093 0.49617934226989746 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:1962455/stdout:fail: 1962455 2.030329465866089 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:1964831/stdout:fail: 1964831 1.3607375621795654 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:1972041/stdout:fail: 1972041 1.2656726837158203 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2131185/stdout:fail: 2131185 0.8565192222595215 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2330212/stdout:fail: 2330212 1.0435378551483154 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2340843/stdout:fail: 2340843 1.059697151184082 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2376573/stdout:fail: 2376573 0.600783109664917 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2435591/stdout:fail: 2435591 0.6575438976287842 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2486279/stdout:fail: 2486279 1.1877374649047852 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2592263/stdout:fill: 2592263 56292.620924949646 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2594567/stdout:fill: 2594567 46556.722032547 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2596871/stdout:fill: 2596871 43117.009229421616 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2599175/stdout:fill: 2599175 41285.22442698479 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2601479/stdout:fill: 2601479 41891.6873960495 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2603783/stdout:fill: 2603783 41889.01078701019 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2608391/stdout:fill: 2608391 41466.79130220413 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2668193/stdout:fail: 2668193 3547.1275115013123 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2677559/stdout:fail: 2677559 4.968778610229492 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2678921/stdout:fail: 2678921 1.9497594833374023 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:2680583/stdout:fail: 2680583 1.4392280578613281 exception ::omero::ResourceError
/tmp/cache-20231209/1/Image:797729/stdout:fail: 797729 0.8322896957397461 exception ::omero::ResourceError

1556033, 1573071, 1600787 (idr0013)

1949093 is idr0015 broken plate.

1962455, 1964831, 1972041 (idr0015) are actually all ready to go. Ran them #645 (comment)

2131185, 2330212, 2340843, 2376573, 2435591, 2486279 are idr0016 - all known

New errors from idr0016:
2592263 Plate name 26071 Data looks valid
2594567 data valid
2596871 validator
2599175
2601479
2603783
2608391
2668193
2677559
2678921
2680583

797729 is idr0004

13th Dec: 9:00...

[wmoore@prod120-proxy ~]$  ls /tmp/cache-20231209/1/ | wc
    992     992   13842
[wmoore@prod120-proxy ~]$ grep Error /tmp/cache-20231209/1/*/* | wc
     14      70    1522

14th Dec (after memo file generation was paused and restarted with different log location below)
Checking old logs...

[wmoore@prod120-proxy ~]$ ls /tmp/cache-20231209/1/ | wc
   1067    1067   14892
[wmoore@prod120-proxy ~]$ grep Error /tmp/cache-20231209/1/*/* | wc
     25     125    2712

Update the list above...

will-moore · 2023-12-14T11:06:19Z

Seb "NGFF memo file regeneration process restarted earlier this morning (results under /tmp/cache-20231214)"

[sbesson@prod120-proxy ~]$ grep fail /tmp/cache-20231214/1/*/* | wc
     14      61    1402
[sbesson@prod120-proxy ~]$ grep ok /tmp/cache-20231214/1/*/* | wc
   1043    3129   77501

Some fails are Authentication failed. All others are know errors (idr0015 3 plates now fixed).

[wmoore@prod120-proxy ~]$ grep fail /tmp/cache-20231214/1/*/*
/tmp/cache-20231214/1/Image:1556033/stdout:fail: 1556033 0.3516662120819092 exception ::omero::ResourceError
/tmp/cache-20231214/1/Image:1573071/stdout:fail: 1573071 0.1804647445678711 exception ::omero::ResourceError
/tmp/cache-20231214/1/Image:1600787/stdout:fail: 1600787 0.14262938499450684 exception ::omero::ResourceError
/tmp/cache-20231214/1/Image:1949093/stdout:fail: 1949093 0.06853318214416504 exception ::omero::ResourceError
/tmp/cache-20231214/1/Image:2131185/stdout:fail: 2131185 0.2601206302642822 exception ::omero::ResourceError
/tmp/cache-20231214/1/Image:2330212/stdout:fail: 2330212 0.2782464027404785 exception ::omero::ResourceError
/tmp/cache-20231214/1/Image:2340843/stdout:fail: 2340843 0.22100067138671875 exception ::omero::ResourceError
/tmp/cache-20231214/1/Image:2376573/stdout:fail: 2376573 0.2599215507507324 exception ::omero::ResourceError
/tmp/cache-20231214/1/Image:2435591/stdout:fail: 2435591 0.2041921615600586 exception ::omero::ResourceError
/tmp/cache-20231214/1/Image:2486279/stdout:fail: 2486279 0.16040253639221191 exception ::omero::ResourceError
/tmp/cache-20231214/1/Image:2850036/stderr:Authentication failed.
/tmp/cache-20231214/1/Image:2850084/stderr:Authentication failed.
/tmp/cache-20231214/1/Image:2850132/stderr:Authentication failed.
/tmp/cache-20231214/1/Image:797729/stdout:fail: 797729 0.7897646427154541 exception ::omero::ResourceError

will-moore · 2024-01-05T15:12:36Z

idr-next uk1s3 steps

idr0011, idr0013, idr0016

Grab all the latest sql scripts, now at https://github.com/IDR/mkngff_upgrade_scripts

[wmoore@prod120-omeroreadwrite ~]$ cd /uod/idr/metadata/
[wmoore@prod120-omeroreadwrite metadata]$ sudo -Es git clone https://github.com/IDR/mkngff_upgrade_scripts.git
[wmoore@prod120-omeroreadwrite mkngff_upgrade_scripts]$ git log
commit 551f04a4abfeed073e15dcf02c3f9c7e301f5adc
Author: William Moore <[email protected]>
Date:   Fri Jan 5 12:14:44 2024 +0000

    Add idr0011_four.csv idr0013_three.csv idr0016_seven.csv
...

Update sql with SECRET...

cd idr0011  # idr0013, idr0016
for i in $(ls); do sudo sed -i 's/SECRETUUID/e3e1ac30-7b69-473b-98b2-428780578b1c/g' $i; done

Mount buckets with goofys:

sudo mkdir /idr0011 && sudo /usr/bin/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other idr0011 /idr0011
sudo mkdir /idr0013 && sudo /usr/bin/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other idr0013 /idr0013
sudo mkdir /idr0016 && sudo /usr/bin/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other idr0016 /idr0016

See steps at #642 (comment)
As omero-server...

cd /uod/idr/metadata/mkngff_upgrade_scripts/ngff_filesets

omero login
export IDRID=idr0011
for r in $(cat idr0011_four.csv); do
  platename=$(echo $r | cut -d',' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
  omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/idr0011/zarr/$platename" --bfoptions
done

Images from all 4 Plates are viewable 👍

idr0013....

export IDRID=idr0013
for r in $(cat idr0013_three.csv); do
  platename=$(echo $r | cut -d',' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
  omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/idr0011/zarr/$platename" --bfoptions
done

Ooops - copy/paste error - still had idr0011 in symlink, and symlink doesn't have /zarr for idr0013...
Fix for all 3:

rm /data/OMERO/ManagedRepository/demo_2/2016-05/08/17-02-05.805_mkngff/LT0103_13--ex2006_11_22--sp2005_08_16--tt19--c4.ome.zarr
ln -s /idr0013/LT0103_13--ex2006_11_22--sp2005_08_16--tt19--c4.ome.zarr /data/OMERO/ManagedRepository/demo_2/2016-05/08/17-02-05.805_mkngff/LT0103_13--ex2006_11_22--sp2005_08_16--tt19--c4.ome.zarr

rm /data/OMERO/ManagedRepository/demo_2/2016-05/07/02-36-52.924_mkngff/LT0080_37--ex2005_07_20--sp2005_07_04--tt17--c4.ome.zarr
ln -s /idr0013/LT0080_37--ex2005_07_20--sp2005_07_04--tt17--c4.ome.zarr /data/OMERO/ManagedRepository/demo_2/2016-05/07/02-36-52.924_mkngff/LT0080_37--ex2005_07_20--sp2005_07_04--tt17--c4.ome.zarr

rm /data/OMERO/ManagedRepository/demo_2/2016-05/03/23-33-31.705_mkngff/LT0066_23--ex2005_08_03--sp2005_06_07--tt17--c3.ome.zarr
ln -s /idr0013/LT0066_23--ex2005_08_03--sp2005_06_07--tt17--c3.ome.zarr /data/OMERO/ManagedRepository/demo_2/2016-05/03/23-33-31.705_mkngff/LT0066_23--ex2005_08_03--sp2005_06_07--tt17--c3.ome.zarr

Images from all 3 Plates are viewable 👍

idr0016...

for r in $(cat idr0016_seven.csv); do
  platename=$(echo $r | cut -d',' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
  omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/idr0016/zarr/$platename" --bfoptions
done

UPDATE 2304
BEGIN
 mkngff_fileset 
----------------
        6317451
(1 row)

COMMIT
Previous session expired for demo on localhost:4064
Server: [localhost:4064]
Username: [demo]
Password:
Created session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/20/05-04-12.866
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-06/20/05-04-12.866_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-06/20/05-04-12.866_mkngff/25585.ome.zarr -> /idr0016/zarr/25585.ome.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/20/05-04-12.866
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2016-06/20/05-04-12.866_mkngff/25585.ome.zarr.bfoptions
UPDATE 2304
BEGIN
 mkngff_fileset 
----------------
        6317452
(1 row)
...

Images from all 7 Plates are viewable 👍

idr0064

See #682 (comment)

sudo mkdir /idr0064 && sudo /usr/bin/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other idr0064 /idr0064

export IDRID=idr0064
for r in $(cat $IDRID.csv); do
  platename=$(echo $r | cut -d',' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
  omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/idr0064/$platename.ome.zarr" --bfoptions
done

UPDATE 50
BEGIN
 mkngff_fileset 
----------------
        6317458
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-3/2020-04/23/14-44-25.357
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-3/2020-04/23/14-44-25.357_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-3/2020-04/23/14-44-25.357_mkngff/1.1.ome.zarr -> /idr0064/1.1.ome.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-3/2020-04/23/14-44-25.357
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-3/2020-04/23/14-44-25.357_mkngff/1.1.ome.zarr.bfoptions
UPDATE 50
BEGIN
 mkngff_fileset 
----------------
        6317459
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-12/2020-04/23/14-46-43.199
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-12/2020-04/23/14-46-43.199_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-12/2020-04/23/14-46-43.199_mkngff/1.2.ome.zarr -> /idr0064/1.2.ome.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-12/2020-04/23/14-46-43.199
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-12/2020-04/23/14-46-43.199_mkngff/1.2.ome.zarr.bfoptions
UPDATE 50
BEGIN
 mkngff_fileset 
----------------
        6317460
(1 row)
...

Images from all 9 Plates are visible in webclient 👍

idr0091

See #650 (comment)

sudo mkdir /idr0091 && sudo /usr/bin/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other idr0091 /idr0091

for r in $(cat idr0091_imageids.csv); do
  python /uod/idr/metadata/idr-utils/scripts/managed_repo_symlinks.py Image:$r /idr0091/zarr/ --repo /data/OMERO/ManagedRepository --fileset-mappings idr0091_symlinks.csv --report
done

...
Fileset: 6314117 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-10/2020-10/02/15-02-19.393_mkngff/
Render Image 10648050
fs_contents ['0c17cfda-6727-401f-8bdf-f0a9981403b6.zarr', '0c17cfda-6727-401f-8bdf-f0a9981403b6.zarr.bfoptions']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-10/2020-10/02/15-02-19.393_mkngff/0c17cfda-6727-401f-8bdf-f0a9981403b6.zarr to /idr0091/zarr/20151204_switch6h_pos0_GL06.ome.zarr
Symlink target not found: /idr0091/zarr/0c17cfda-6727-401f-8bdf-f0a9981403b6.zarr.bfoptions
...
Fileset: 6314274 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2020-10/03/18-48-59.765_mkngff/
Render Image 10648771
fs_contents ['882f80fa-f40f-455b-b923-09dce086675b.zarr', '882f80fa-f40f-455b-b923-09dce086675b.zarr.bfoptions']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2020-10/03/18-48-59.765_mkngff/882f80fa-f40f-455b-b923-09dce086675b.zarr to /idr0091/zarr/20161207_Pos1_GL26.ome.zarr
Symlink target not found: /idr0091/zarr/882f80fa-f40f-455b-b923-09dce086675b.zarr.bfoptions

will-moore · 2024-01-08T16:23:51Z

For idr0009 - need to add scanr.skip_missing_wells=false to a .bfoptions file for every Plate (964 plates)!
E.g. ?show=well-607352 plate has missing Wells. Trying to view any image after missing Wells shows wrong Image or error.

Fileset template prefixes - test with just a couple from that Plate and the next

http://localhost:1080/webclient/?show=plate-2703
http://localhost:1080/webclient/?show=plate-2705

bash-4.2$ cat idr0009_filesetprefix.csv
demo_2/2015-11/26/07-41-09.642
demo_2/2015-11/26/07-41-14.425

For each, create experiment_descriptor.xml.bfoptions

 for f in $(cat idr0009_filesetprefix.csv); do echo "scanr.skip_missing_wells=false" > "/data/OMERO/ManagedRepository/$f/experiment_descriptor.xml.bfoptions"; done

This has created .bfoptions file... for http://localhost:1080/webclient/?show=plate-2705

bash-4.2$ ls -alh /data/OMERO/ManagedRepository/demo_2/2015-11/26/07-41-14.425
total 112K
drwxrwxr-x.  3 omero-server omero-server  153 Jan  8 15:43 .
drwxrwxr-x. 72 omero-server omero-server 8.0K Nov 26  2015 ..
lrwxrwxrwx.  1 omero-server omero-server   97 Nov 26  2015 AcquisitionLog.dat -> /idr/filesets/idr0009-simpson-secretion/20150507-VSVG/VSVG/0305-10--2007-04-10/AcquisitionLog.dat
drwxrwxr-x.  2 omero-server omero-server  72K Nov 26  2015 data
lrwxrwxrwx.  1 omero-server omero-server  104 Nov 26  2015 experiment_descriptor.dat -> /idr/filesets/idr0009-simpson-secretion/20150507-VSVG/VSVG/0305-10--2007-04-10/experiment_descriptor.dat
lrwxrwxrwx.  1 omero-server omero-server  104 Nov 26  2015 experiment_descriptor.xml -> /idr/filesets/idr0009-simpson-secretion/20150507-VSVG/VSVG/0305-10--2007-04-10/experiment_descriptor.xml
-rw-r--r--.  1 omero-server omero-server   31 Jan  8 15:43 experiment_descriptor.xml.bfoptions

The bfmemo exists for some Plates, e.g. first Plate of idr0009 on idr-testing:

ls -alh /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2015-11/26/07-22-16.896
total 448K
drwxrwxr-x. 2 omero-server omero-server   47 Nov 30 17:36 .
drwxrwxr-x. 5 omero-server omero-server   66 Jan  8 15:58 ..
-rw-rw-r--. 1 omero-server omero-server 448K Nov 30 17:36 .experiment_descriptor.xml.bfmemo

and for Plate above http://localhost:1080/webclient/?show=plate-2705 with some missing Wells

bash-4.2$ ls -alh /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2015-11/26/07-41-14.425
total 444K
drwxrwxr-x. 2 omero-server omero-server   47 Jan  8 15:37 .
drwxrwxr-x. 5 omero-server omero-server   66 Jan  8 15:58 ..
-rw-rw-r--. 1 omero-server omero-server 443K Jan  8 15:37 .experiment_descriptor.xml.bfmemo

but not for other plates with missing Wells - e.g. first plate above

bash-4.2$ ls -alh /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2015-11/26/07-41-09.642
total 0
drwxrwxr-x. 2 omero-server omero-server  6 Jan  8 15:58 .
drwxrwxr-x. 5 omero-server omero-server 66 Jan  8 15:58 ..

lets try to delete .bfmemo and see if it fixes problem...

bash-4.2$ rm /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2015-11/26/07-41-14.425/.experiment_descriptor.xml.bfmemo

Now, we can no-longer view ANY images on http://localhost:1080/webclient/?show=plate-2705

    serverExceptionClass = ome.conditions.ResourceError
    message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2015-11/26/07-41-14.425/experiment_descriptor.xml

👎

After I delete the bfoptions file created above

$ rm /data/OMERO/ManagedRepository/demo_2/2015-11/26/07-41-14.425/experiment_descriptor.xml.bfoptions

This allows images to be viewable again and a new .bfmemo is created

$ ls -alh /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2015-11/26/07-41-14.425
total 444K
drwxrwxr-x. 2 omero-server omero-server   47 Jan  8 16:41 .
drwxrwxr-x. 5 omero-server omero-server   66 Jan  8 15:58 ..
-rw-rw-r--. 1 omero-server omero-server 443K Jan  8 16:41 .experiment_descriptor.xml.bfmemo

will-moore · 2024-01-15T11:17:41Z

idr0009 .bfoptions creation

See #684

On idr-next (prod120) as omero-server...

Created idr0009_filesets.csv

9509,demo_2/2015-09/23/10-40-23.554/
12601,demo_2/2015-09/30/09-20-50.608/
9512,demo_2/2015-09/23/10-48-58.762/
...
16663,demo_2/2015-11/26/09-12-37.788/
16664,demo_2/2015-11/26/09-15-28.047/
16665,demo_2/2015-11/26/09-15-36.785/

Also created addbfoptions.sql

CREATE OR REPLACE FUNCTION add_origfile_to_fileset(
    fileset_id bigint,
    name character varying,
    uuid character varying)
  RETURNS integer AS
$BODY$
DECLARE
   new_event integer;
   old_owner integer;
   old_group integer;
   old_perms integer;
   new_file integer;
   files_count integer;
   prefix character varying;

BEGIN

    select _current_or_new_event() into new_event;
    select
        owner_id, group_id, permissions, templateprefix
     into
        old_owner, old_group, old_perms, prefix
     from fileset where id = fileset_id;

    select count(id) from filesetentry where fileset = fileset_id into files_count;

    insert into originalfile (id, permissions, creation_id, group_id, owner_id, update_id, mimetype, repo, path, name)
        values (nextval('seq_originalfile'), old_perms, new_event, old_group, old_owner, new_event,
            'application/octet-stream', 'cdf35825-def1-4580-8d0b-9c349b8f78d6', prefix, uuid || name)
            
            returning id into new_file;

    insert into filesetentry
          (id, permissions, creation_id, group_id, owner_id, update_id, fileset, originalfile, fileset_index, clientpath)
          values (nextval('seq_filesetentry'), old_perms, new_event, old_group, old_owner, new_event,
                  fileset_id, new_file, files_count, name);

    RETURN new_file;
END;
$BODY$
  LANGUAGE plpgsql VOLATILE;

$ cat idr0009_filesets.csv | wc
   1034    1034   39125

$ psql -U omero -d idr -h $DBHOST -f addbfoptions.sql 
CREATE FUNCTION

$ for r in $(cat idr0009_filesets.csv); do
  fsid=$(echo $r | cut -d',' -f1)
  prefix=$(echo $r | cut -d',' -f2 | sed 's/.$//')
  echo "scanr.skip_missing_wells=false" > "/data/OMERO/ManagedRepository/$prefix/experiment_descriptor.xml.bfoptions";
  psql -U omero -d idr -h $DBHOST -c "select add_origfile_to_fileset($fsid, 'experiment_descriptor.xml.bfoptions', '$SECRET')"
  rm -f /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/$prefix/.experiment_descriptor.xml.bfmemo 
done

will-moore · 2024-01-17T15:26:16Z

Snapshot of idr-next just taken by Dom.

Upate idr-next to "revert" usage of our own s3 buckets at #675 (comment) and replace the NGFF Filesets created there with ones based on BioStudies-hosted data (but without adding an extra _mkngff suffix to the Fileset.templatePrefix....

Needs install of IDR/omero-mkngff#14 to create new Filesets without extra _mkngff suffix...
And use --fs_suffix=None below...

pip install 'omero-mkngff @ git+https://github.com/will-moore/omero-mkngff@fs_suffix'

idr0011.csv - with Fileset IDs from idr-next...

idr0011/Plate15-Yellow-A.ome.zarr,S-BIAD866/684211eb-77bc-4693-a904-851c79f19b28,6317438
idr0011/Plate13-Red-B.ome.zarr,S-BIAD866/a3781c5d-c4e2-4ad6-9d57-357069721845,6317439
idr0011/Plate3-Yellow-B.ome.zarr,S-BIAD866/311935f7-f3d5-4405-9456-d8c49f736f11,6317440
idr0011/Plate11-Blue-A.ome.zarr,S-BIAD866/e138116a-2215-4581-928e-73c32d667018,6317441
idr0011/Target-TS,S-BIAD866/ef7df8ac-0c98-47a6-944e-6e86f18f174a,6312743

idr0013.csv - with Fileset IDs from idr-next...

idr0013/LT0080_37.ome.zarr.zip,S-BIAD865/aea4aa32-60c2-4a38-8a91-9f303381e562,6317449
idr0013/LT0066_23.ome.zarr.zip,S-BIAD865/c1d9f06e-cfd0-43cd-be2f-3e5f39c3b62a,6317448
idr0013/LT0103_13.ome.zarr.zip,S-BIAD865/eae9bb4c-9504-4f88-9931-dbf234f86023,6317450

idr0016.csv - with Fileset IDs from idr-next...

idr0016/25585.ome.zarr,S-BIAD851/da7853ca-2f99-4b46-b159-00371ffbe4c3,6317451
idr0016/24634.ome.zarr,S-BIAD851/cd887d08-ea47-489f-ab2d-039a64af938b,6317452
idr0016/24667.ome.zarr,S-BIAD851/adbbfaf5-5ac4-48f1-acf7-3180b3a6445d,6317453
idr0016/25909.ome.zarr,S-BIAD851/076d5d53-e59f-40cd-bae9-c914c7a1ae26,6317454
idr0016/25592.ome.zarr,S-BIAD851/539f798f-99c9-4351-a4ca-35c6ad4f2d22,6317455
idr0016/25675.ome.zarr,S-BIAD851/312f9dbd-eaec-4993-b0c5-95c9db5a0248,6317456
idr0016/25740.ome.zarr,S-BIAD851/759823ab-291d-460b-bfca-9f8903e50631,6317457

For each of idr0011, idr0013, idr0016...

for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  omero mkngff sql $fsid --fs_suffix=None --clientpath="https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/$biapath/$uuid.zarr" "/bia-integrator-data/$biapath/$uuid.zarr" > "$IDRID/$fsid.sql"
done

for i in $(ls); do sed -i 's/SECRETUUID/e3e1ac30-7b69-473b-98b2-428780578b1c/g' $i; done

for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
  omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/bia-integrator-data/$biapath/$uuid.zarr" --fs_suffix=None --bfoptions
done

...
# idr0013
UPDATE 368
BEGIN
 mkngff_fileset 
----------------
        6317474
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-05/08/17-02-05.805_mkngff
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-05/08/17-02-05.805_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-05/08/17-02-05.805_mkngff/eae9bb4c-9504-4f88-9931-dbf234f86023.zarr -> /bia-integrator-data/S-BIAD865/eae9bb4c-9504-4f88-9931-dbf234f86023/eae9bb4c-9504-4f88-9931-dbf234f86023.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-05/08/17-02-05.805_mkngff
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2016-05/08/17-02-05.805_mkngff/eae9bb4c-9504-4f88-9931-dbf234f86023.zarr.bfoptions

#idr0016
...
UPDATE 2304
BEGIN
 mkngff_fileset 
----------------
        6317481
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/24/16-32-36.710_mkngff
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-06/24/16-32-36.710_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-06/24/16-32-36.710_mkngff/759823ab-291d-460b-bfca-9f8903e50631.zarr -> /bia-integrator-data/S-BIAD851/759823ab-291d-460b-bfca-9f8903e50631/759823ab-291d-460b-bfca-9f8903e50631.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/24/16-32-36.710_mkngff
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2016-06/24/16-32-36.710_mkngff/759823ab-291d-460b-bfca-9f8903e50631.zarr.bfoptions

idr0011:

http://localhost:1080/webclient/?show=plate-5639

will-moore · 2024-01-17T22:33:53Z

Same for idr0064...

idr0064.csv - IDs for idr-next...

idr0064/1.1.ome.zarr,S-BIAD992/ec0b496e-2d48-44ed-be4d-0339f8927eef,6317458
idr0064/1.2.ome.zarr,S-BIAD992/352f2cac-020d-494d-8e48-37a8f4e4f3f2,6317459
idr0064/1.3.ome.zarr,S-BIAD992/34948485-1bde-4da6-a4ca-c21e223092fc,6317460
idr0064/2.1.ome.zarr,S-BIAD992/641125ba-978c-488f-be0f-75b9bb09b6f6,6317461
idr0064/2.2.ome.zarr,S-BIAD992/4518e5a4-7804-4918-b42a-678ebd89ba0d,6317462
idr0064/2.3.ome.zarr,S-BIAD992/e40d09f6-152a-47c1-8986-a5f924d608f8,6317463
idr0064/3.1.ome.zarr,S-BIAD992/23d753d0-de0b-4f72-8b99-1fe39605cfc0,6317464
idr0064/3.2.ome.zarr,S-BIAD992/bf6dfdb8-d84c-47f6-a36a-11d1dddfe689,6317465
idr0064/3.3.ome.zarr,S-BIAD992/16d9a6cb-32d3-49a3-8654-9c869baa2394,6317466

...
UPDATE 50
BEGIN
 mkngff_fileset 
----------------
        6317490
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-12/2020-04/23/15-00-09.126_mkngff
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-12/2020-04/23/15-00-09.126_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-12/2020-04/23/15-00-09.126_mkngff/16d9a6cb-32d3-49a3-8654-9c869baa2394.zarr -> /bia-integrator-data/S-BIAD992/16d9a6cb-32d3-49a3-8654-9c869baa2394/16d9a6cb-32d3-49a3-8654-9c869baa2394.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-12/2020-04/23/15-00-09.126_mkngff
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-12/2020-04/23/15-00-09.126_mkngff/16d9a6cb-32d3-49a3-8654-9c869baa2394.zarr.bfoptions

Looks good...
http://localhost:1080/webclient/?show=screen-2351

will-moore · 2024-01-18T06:58:02Z

For testing, all these are now hosted by BioStudies...

idr0011/Plate15-Yellow-A
idr0011/Plate13-Red-B
idr0011/Plate3-Yellow-B
idr0011/Plate11-Blue-A
idr0011/Target-TS

idr0013/LT0080_37
idr0013/LT0066_23
idr0013/LT0103_13

idr0016/25585
idr0016/24634
idr0016/24667
idr0016/25909
idr0016/25592
idr0016/25675
idr0016/25740

idr0064 (all plates)

will-moore · 2024-01-24T10:02:47Z

Installed geesefs and switched goofys for geesefs mounts on each of the 5 nodes:
e.g.

[wmoore@prod120-omeroreadonly-4 ~]$ wget https://github.com/yandex-cloud/geesefs/releases/latest/download/geesefs-linux-amd64
2024-01-24 10:00:42 (297 MB/s) - ‘geesefs-linux-amd64’ saved [31464377/31464377]

[wmoore@prod120-omeroreadonly-4 ~]$ chmod +x geesefs-linux-amd64
[wmoore@prod120-omeroreadonly-4 ~]$ mv geesefs-linux-amd64 geesefs
[wmoore@prod120-omeroreadonly-4 ~]$ sudo cp geesefs /usr/local/bin
[wmoore@prod120-omeroreadonly-4 ~]$ sudo umount /bia-integrator-data
[wmoore@prod120-omeroreadonly-4 ~]$ sudo /usr/local/bin/geesefs --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other bia-integrator-data /bia-integrator-data
s3.INFO anonymous bucket detected
main.INFO File system has been successfully mounted.

will-moore · 2024-01-24T12:14:03Z

After testing by @pwalczysko, now switching back to use goofys on idr-next...

On each server, unmount (geesefs), re-mount (goofys), restart server...

[wmoore@prod120-proxy ~]$ for server in omeroreadwrite omeroreadonly-1 omeroreadonly-2 omeroreadonly-3 omeroreadonly-4; do ssh $server "sudo umount /bia-integrator-data"; done
[wmoore@prod120-proxy ~]$ for server in omeroreadwrite omeroreadonly-1 omeroreadonly-2 omeroreadonly-3 omeroreadonly-4; do ssh $server "sudo /usr/bin/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other bia-integrator-data /bia-integrator-data"; done
[wmoore@prod120-proxy ~]$ 
[wmoore@prod120-proxy ~]$ for server in omeroreadwrite omeroreadonly-1 omeroreadonly-2 omeroreadonly-3 omeroreadonly-4; do ssh $server "sudo service omero-server restart"; done
Redirecting to /bin/systemctl restart omero-server.service
Redirecting to /bin/systemctl restart omero-server.service
Redirecting to /bin/systemctl restart omero-server.service

Message from syslogd@localhost at Jan 24 12:12:21 ...
 haproxy[19461]: backend omero4064-0 has no server available!
Redirecting to /bin/systemctl restart omero-server.service
Redirecting to /bin/systemctl restart omero-server.service

Message from syslogd@localhost at Jan 24 12:13:20 ...
 haproxy[19461]: backend omero4064-1 has no server available!

will-moore · 2024-04-11T17:08:17Z

start from scratch on rocky9 pilot

Update ZarrReader to today's build, including Reintroduce S3FileSystemStore ome/ZarrReader#82

$ ssh pilot-idrngff -L 1080:localhost:80
$ sudo -u omero-server -s
$ cd
$ wget https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/33/label=testintegration/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.4.2-SNAPSHOT-jar-with-dependencies.jar
bash-5.1$ mv OMEZarrReader-0.4.2-SNAPSHOT-jar-with-dependencies.jar OMEZarrReader_b33.jar
bash-5.1$ rm OMERO.server/lib/server/OMEZarrReader.jar 
bash-5.1$ rm OMERO.server/lib/client/OMEZarrReader.jar 
bash-5.1$ cp OMEZarrReader_b33.jar OMERO.server/lib/server/
bash-5.1$ cp OMEZarrReader_b33.jar OMERO.server/lib/client/

[wmoore@pilot-idrngff-omeroreadwrite ~]$ sudo service omero-server restart

Install mkngff in venv, including recent PR branches (if not yet merged) - To include Alt store bfoptions omero-mkngff#17

$ sudo -u root -s
# source /opt/omero/server/venv3/bin/activate

pip install 'omero-mkngff @ git+https://github.com/will-moore/omero-mkngff@alt_store_bfoptions'

As omero-server, clone https://github.com/IDR/mkngff_upgrade_scripts

$ sudo -u omero-server -s
bash-5.1$ cd
bash-5.1$ git clone https://github.com/IDR/mkngff_upgrade_scripts.git

Delete duplicate plate for idr0015

screen -S mkngff
...
$ omero delete Plate:4801 --report > /tmp/delete_idr0015.log

Get $SECRET, update all sql commands with it and run them (see https://github.com/IDR/mkngff_upgrade_scripts)

for i in $(ls); do sed -i 's/SECRETUUID/e1e69d03-6642-4f0b-8c0c-c98c1f89fb4b/g' $i; done

for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
  omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/bia-integrator-data/$biapath/$uuid.zarr" --bfoptions --clientpath="https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/$biapath/$uuid.zarr"
done

idr0004 - remove zarrreader.quick_read=true from bfoptions for 2 plates...

Plate P115:

vi /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-46-42.965_mkngff/35cfc0db-7795-497c-aed5-1ae591b2d9f1.zarr.bfoptions

Plate P124:

vi /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-57-40.271_mkngff/ee8872c8-e4b1-41fa-aa4f-a9e3e200c540.zarr.bfoptions

Try to view image from 2nd Plate at 9:40...http://localhost:1080/webclient/?show=well-469196 - success ~ 10:00.
Also Plate P115 was viewable.

idr0010 - sql applied - viewing first plate 12:12 (15th Aug)...
idr0011 - sql applied... (start 12:16 15th Aug)...
idr0012 - sql applied... (start 13:15)
idr0013 - sql applied, start 13:42...
idr0015 - sql applied, start 15:13....
idr0016 - sql applied, start 16:21...
idr0025 - sql applied, start 19:25...
idr0026 - sql applied, start 20:00...
idr0035 - sql applied, start 20:35...
idr0036 - sql applied, start 22:29...
idr0051 - sql applied, start 22:42...
idr0054 - sql applied, start 22:45...
idr0064 - sql applied, start 22:53...
idr0090 - sql applied, start 22:57...
idr0091 - sql applied, start 23:04...

will-moore · 2024-04-18T16:07:38Z

Since idr0091 scripts (idr0091.csv) was not updated back to the original Fileset IDs, the wrong Filesets got replaced on pilot-idrngff above. Now, with the csv updated with correct IDs, we can't run this on pilot-idrngff because the OriginalFile path/names we want to create are already there and we aren't allowed duplicates! This could take a long time to clean-up, so an easier solution is to test idr0091 on a different server...

Use pilot-rocky9:

$ ssh pilot-rocky9

Added ZarrReader (build 41) as above (no OMEZarrReader.jar there before) but didn't restart server yet... (Dom using it).
installed omero-mkngff
Cloned mkngff_upgrade_scripts
Ran omero mkngff sql to regenerate idr0091 sql scripts with original IDs. Committed at IDR/mkngff_upgrade_scripts@500c997
Then...

cd idr0091
for i in $(ls); do sed -i 's/SECRETUUID/bc9a125d-fa7c-417c-a33b-506390bb3b7f/g' $i; done
cd ../
omero login
...
# applied sql scripts, symlinks, bfoptions etc.

BUT - this data isn't viewable since we don't have latest Bio-Formats on this server. Just the IDR version.

So, back to the drawing board...!

EDIT:

deleted all the incorrectly generated filesets above
ran idr0091 scripts from github
all OK 👍

will-moore · 2024-04-25T08:20:23Z

idr0009 update on pilot-idrngff as recorded at #684 (comment)

will-moore · 2024-04-25T15:59:17Z

Testing of svs update failed due to Bio-Formats version mismatches...

David:
"You should be able to replace the jars, the main jars are formats-api, formats-bsd, formats-gpl and turbojpeg. If that has the latest snapshot then you might also need to check the lower level components, so anything listed under component updates in https://www.openmicroscopy.org/2024/04/18/bio-formats-7-3-0.html has since been updated"

Downloaded various jars into bfjars_25thApril_b48:

bash-5.1$ ls -alh bfjars_25thApril_b48/
total 7.2M
drwxr-xr-x.  2 omero-server omero-server 4.0K Apr 25 12:43 .
drwxr-xr-x. 11 omero-server root         4.0K Apr 25 12:46 ..
-rw-r--r--.  1 omero-server omero-server 143K Apr 25 00:08 formats-api-8.0.0-SNAPSHOT.jar
-rw-r--r--.  1 omero-server omero-server 880K Apr 25 00:08 formats-bsd-8.0.0-SNAPSHOT.jar
-rw-r--r--.  1 omero-server omero-server 1.5M Apr 25 00:08 formats-gpl-8.0.0-SNAPSHOT.jar
-rw-r--r--.  1 omero-server omero-server  14K Apr 25 00:08 metakit-5.3.8-SNAPSHOT.jar
-rw-r--r--.  1 omero-server omero-server 7.8K Apr 25 00:08 mipav-stubs-6.0.3-SNAPSHOT.jar
-rw-r--r--.  1 omero-server omero-server  90K Apr 25 00:08 ome-codecs-1.0.4-SNAPSHOT.jar
-rw-r--r--.  1 omero-server omero-server 139K Apr 25 00:07 ome-common-6.0.23-SNAPSHOT.jar
-rw-r--r--.  1 omero-server omero-server 1.0M Apr 25 00:08 ome-jai-0.1.5-SNAPSHOT.jar
-rw-r--r--.  1 omero-server omero-server  90K Apr 25 00:08 ome-mdbtools-5.3.4-SNAPSHOT.jar
-rw-r--r--.  1 omero-server omero-server 928K Apr 25 00:07 ome-poi-5.3.10-SNAPSHOT.jar
-rw-r--r--.  1 omero-server omero-server 639K Apr 25 00:07 ome-xml-6.3.7-SNAPSHOT.jar
-rw-r--r--.  1 omero-server omero-server 615K Apr 25 00:07 specification-6.3.7-SNAPSHOT.jar
-rw-r--r--.  1 omero-server omero-server 1.3M Apr 25 00:08 turbojpeg-8.0.0-SNAPSHOT.jar

For each of these, moved the existing equivalent jar from OMERO.server/lib/server into bfjars_orig,
and deleted jar from OMERO.server/lib/client.
NB: couldn't find existing jar equivalent to mipav-stubs-6.0.3-SNAPSHOT.jar.

bash-5.1$ ls -alh bfjars_orig
total 7.2M
drwxr-xr-x.  2 omero-server omero-server 4.0K Apr 25 12:56 .
drwxr-xr-x. 11 omero-server root         4.0K Apr 25 12:46 ..
-rw-r--r--.  1 omero-server omero-server 140K Apr 11 14:02 formats-api.jar
-rw-r--r--.  1 omero-server omero-server 877K Apr 11 14:02 formats-bsd.jar
-rw-r--r--.  1 omero-server omero-server 1.5M Apr 22 12:15 formats-gpl-8.0.0-SNAPSHOT_b45.jar
-rw-r--r--.  1 omero-server omero-server  14K Apr 11 14:02 metakit.jar
-rw-r--r--.  1 omero-server omero-server  89K Apr 11 14:02 ome-codecs.jar
-rw-r--r--.  1 omero-server omero-server 140K Apr 11 14:02 ome-common.jar
-rw-r--r--.  1 omero-server omero-server 1.1M Apr 11 14:02 ome-jai.jar
-rw-r--r--.  1 omero-server omero-server  91K Apr 11 14:02 ome-mdbtools.jar
-rw-r--r--.  1 omero-server omero-server 930K Apr 11 14:02 ome-poi.jar
-rw-r--r--.  1 omero-server omero-server 638K Apr 11 14:02 ome-xml.jar
-rw-r--r--.  1 omero-server omero-server 615K Apr 11 14:02 specification.jar
-rw-r--r--.  1 omero-server omero-server 1.3M Apr 11 14:02 turbojpeg.jar

Then copied all new jars into OMERO.server/lib/server/ and OMERO.server/lib/client/.

cp bfjars_25thApril_b48/* OMERO.server/lib/server/
cp bfjars_25thApril_b48/* OMERO.server/lib/client/

Restarted server...

will-moore added this to NGFF conversion Oct 10, 2023

will-moore moved this to create new Filesets in idr-next in NGFF conversion Oct 10, 2023

This was referenced Oct 17, 2023

idr0125 NGFF import IDR/idr0125-way-cellpainting#4

Open

ZarrReader performance improvements ome/ZarrReader#64

Merged

will-moore mentioned this issue Nov 29, 2023

Add check_pixels.py script IDR/idr-utils#55

Open

will-moore mentioned this issue Jan 10, 2024

idr0009 ScanR missing Wells #684

Open

will-moore mentioned this issue Feb 12, 2024

Reduce number of fields in memo file ome/ZarrReader#78

Merged

will-moore mentioned this issue Apr 16, 2024

idr0091-julou-lacinduction S-BIAD852 #650

Open

will-moore mentioned this issue Apr 18, 2024

Alt store bfoptions IDR/omero-mkngff#17

Merged

will-moore mentioned this issue Apr 25, 2024

Add stats for prod121 IDR/idr.openmicroscopy.org#195

Merged

will-moore mentioned this issue Apr 25, 2024

broken svs on Bio-Formats update #691

Open

will-moore moved this from check_pixels to Other issues (not studies) in NGFF conversion May 21, 2024

will-moore mentioned this issue Jun 17, 2024

Bump server to 5.6.11 and incl ZarrReader IDR/deployment#420

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

idr-next steps for NGFF upgrade #675

idr-next steps for NGFF upgrade #675

will-moore commented Oct 10, 2023 •

edited

Loading

will-moore commented Oct 10, 2023 •

edited

Loading

will-moore commented Oct 10, 2023

will-moore commented Oct 10, 2023

will-moore commented Oct 10, 2023

will-moore commented Oct 10, 2023

will-moore commented Oct 11, 2023 •

edited

Loading

will-moore commented Oct 15, 2023 •

edited

Loading

will-moore commented Nov 1, 2023

will-moore commented Nov 3, 2023 •

edited

Loading

will-moore commented Nov 6, 2023 •

edited

Loading

will-moore commented Nov 6, 2023

will-moore commented Nov 6, 2023

will-moore commented Nov 6, 2023

will-moore commented Nov 6, 2023

will-moore commented Dec 6, 2023 •

edited

Loading

will-moore commented Dec 11, 2023 •

edited

Loading

will-moore commented Dec 14, 2023

will-moore commented Jan 5, 2024 •

edited

Loading

will-moore commented Jan 8, 2024 •

edited

Loading

will-moore commented Jan 15, 2024

will-moore commented Jan 17, 2024 •

edited

Loading

will-moore commented Jan 17, 2024

will-moore commented Jan 18, 2024

will-moore commented Jan 24, 2024

will-moore commented Jan 24, 2024

will-moore commented Apr 11, 2024 •

edited

Loading

will-moore commented Apr 18, 2024 •

edited

Loading

will-moore commented Apr 25, 2024

will-moore commented Apr 25, 2024

idr-next steps for NGFF upgrade #675

idr-next steps for NGFF upgrade #675

Comments

will-moore commented Oct 10, 2023 • edited Loading

will-moore commented Oct 10, 2023 • edited Loading

will-moore commented Oct 10, 2023

will-moore commented Oct 10, 2023

will-moore commented Oct 10, 2023

will-moore commented Oct 10, 2023

will-moore commented Oct 11, 2023 • edited Loading

will-moore commented Oct 15, 2023 • edited Loading

will-moore commented Nov 1, 2023

will-moore commented Nov 3, 2023 • edited Loading

will-moore commented Nov 6, 2023 • edited Loading

will-moore commented Nov 6, 2023

will-moore commented Nov 6, 2023

will-moore commented Nov 6, 2023

will-moore commented Nov 6, 2023

will-moore commented Dec 6, 2023 • edited Loading

idr-next (prod120) steps...

will-moore commented Dec 11, 2023 • edited Loading

will-moore commented Dec 14, 2023

will-moore commented Jan 5, 2024 • edited Loading

idr-next uk1s3 steps

idr0011, idr0013, idr0016

idr0064

idr0091

will-moore commented Jan 8, 2024 • edited Loading

will-moore commented Jan 15, 2024

idr0009 .bfoptions creation

will-moore commented Jan 17, 2024 • edited Loading

will-moore commented Jan 17, 2024

will-moore commented Jan 18, 2024

will-moore commented Jan 24, 2024

will-moore commented Jan 24, 2024

will-moore commented Apr 11, 2024 • edited Loading

start from scratch on rocky9 pilot

will-moore commented Apr 18, 2024 • edited Loading

will-moore commented Apr 25, 2024

will-moore commented Apr 25, 2024

will-moore commented Oct 10, 2023 •

edited

Loading

will-moore commented Oct 10, 2023 •

edited

Loading

will-moore commented Oct 11, 2023 •

edited

Loading

will-moore commented Oct 15, 2023 •

edited

Loading

will-moore commented Nov 3, 2023 •

edited

Loading

will-moore commented Nov 6, 2023 •

edited

Loading

will-moore commented Dec 6, 2023 •

edited

Loading

will-moore commented Dec 11, 2023 •

edited

Loading

will-moore commented Jan 5, 2024 •

edited

Loading

will-moore commented Jan 8, 2024 •

edited

Loading

will-moore commented Jan 17, 2024 •

edited

Loading

will-moore commented Apr 11, 2024 •

edited

Loading

will-moore commented Apr 18, 2024 •

edited

Loading