Document mkngff workflow #2

will-moore · 2023-07-07T13:04:10Z

NB This workflow has been moved to https://github.com/IDR/mkngff_upgrade_scripts (main README)

Once a submission has been processed by BioStudies, it will become available at a URL like:
https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/pages/S-BIAD815.html

We need to harvest the uuids from the links in the "Viewable Images" table. We can do this with the following JavaScript code, pasted into the Console tab of the browser dev tools:

let csv = "";
$("#viewable tbody tr").each(function() {
    let $this = $(this);
    if ($("a", $this).length == 0) return
    let uid = $( "a:first", $this).attr("href").replace(".html", "");
    let zarrname = $( "td:nth-child(3)", $this).text().replace(".zip", "");
    csv += `${zarrname},${uid}\n`
});
console.log(csv);

Which will print something like this:

idr0051/180712_H2B_22ss_Courtney_p00_c00_reg_preview.klb.ome.zarr,S-BIAD815/51afff7c-eed4-44b4-95c7-1437d8807b97
idr0051/embryo_dmso_2_new_17-00-44_p00_c00_reg_preview.klb.ome.zarr,S-BIAD815/b2633930-86b0-489e-a845-d2a7afe6ff15
idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr,S-BIAD815/c49efcfd-e767-4ae5-adbf-299cafd92120
idr0051/2018-06-28_21ss_DMSO_TF_20180628-185945_p00_c00_reg_preview.ome.zarr,S-BIAD815/e12a8e2a-4fce-4579-a78b-b0c4597c3ada

That CSV is a table of filesetName.ome.zarr, UUID. We need to add the Fileset IDs from IDR to that table, using idr-util scripts from IDR/idr-utils#56
That PR contains a file idr_filesets.csv which contains Fileset ID, filesetName.ome.zarr from IDR.
It also contains a script to take the csv from above and add the appropriate Fileset IDs (from idr_filesets.csv).

Checkout the idr-utils branch of that PR. This can be done on a local machine.
Copy the csv generated by JavaScript above and save it into a file like idr-utils/scripts/ngff_filesets/idr0051.csv. You will see some examples included in that PR.
Then run the script, passing in the IDR ID...

$ cd idr-utils/scripts/ngff_filesets
$ python parse_bia_uuids.py idr0051

This will update the csv file you just created, adding in the Fileset IDs to a new 3rd column.

Now we want to use that data with omero-mkngff.
We need to do everything as the omero-server user since we'll want to be able to create symlinks from the ManagedRep.

E.g. working on idr0138-pilot...

$ sudo -u omero-server -s

Created conda environment created as omero-server user, e.g. mkngff and installed omero-py and omero-mkngff

conda create -n mkngff -c conda-forge -c ome omero-py bioformats2raw
conda activate mkngff
pip install 'omero-mkngff @ git+https://github.com/IDR/omero-mkngff@main'

Get Database password (and host) needed for psql, and set these to env variables. Also set variable for $IDRID so you can copy and paste other commands from below...

export IDRID=idr0012
export OMERODIR=/opt/omero/server/OMERO.server
omero config get | grep omero.db.host
$ export DBHOST=192.168.10.231
omero config get --show-password | grep omero.db.pass
export PGPASSWORD=[********]

Use psql to get SECRET (last session ID). NB: for pilot servers we only have 1 process (as in this example). For other servers, update the 1 to 3 in this psql command:

psql -U omero -d idr -h $DBHOST -c "select uuid from (select * from session where node = 0 and owner = 0 and defaulteventtype = 'Sessions' order by id desc limit 1) x order by x.id asc limit 1;"
                 uuid                 
--------------------------------------
 8add790d-7855-46f6-8239-c6a72937d572
(1 row)

export SECRET=8add790d-7855-46f6-8239-c6a72937d572

Copy the contents of idr0051.csv table from above (contains Fileset ID and UUID) and create a copy of the csv in the omero-server user's home dir...

$ cd
$ vi $IDRID.csv   # paste in the csv contents from above

Now we can read that csv and create an sql file for each Fileset (named FILESET_ID.sql).
In the loop below, biapath is like S-BIAD815/51afff7c-eed4-44b4-95c7-1437d8807b97 and uuid is like 51afff7c-eed4-44b4-95c7-1437d8807b97.

The BIA s3 repository should be mounted under /bia-integrator-data:

sudo mkdir /bia-integrator-data && sudo /opt/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other bia-integrator-data /bia-integrator-data

Check that e.g. $ ls /bia-integrator-data/S-BIAD815/51afff7c-eed4-44b4-95c7-1437d8807b97/51afff7c-eed4-44b4-95c7-1437d8807b97.zarr will give you 0 OME
Theomero mkngff command below also creates the symlinks we need, from the ManagedRepository to the s3-mounted data (if they don't already exist).

# first output sql functions and login...
omero mkngff setup > setup.sql
omero login

$ mkdir -p $IDRID
$ for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  omero mkngff sql $fsid --clientpath="https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/$biapath/$uuid.zarr" "/bia-integrator-data/$biapath/$uuid.zarr" > "$IDRID/$fsid.sql"
done

# IF YOU WANT TO EXECUTE SQL IMMEDIATELY... include $SECRET and create symlinks...
$ for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  omero mkngff sql --symlink_repo /data/OMERO/ManagedRepository --secret=$SECRET $fsid --clientpath="https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/$biapath/$uuid.zarr" "/bia-integrator-data/$biapath/$uuid.zarr" >> "$IDRID/$fsid.sql" --bfoptions
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
done

Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-14/2018-11/26 // 10-39-49.639 for fileset 604306
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-14/2018-11/26/10-39-49.639
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-14/2018-11/26/10-39-49.639_converted/bia-integrator-data/S-BIAD815/51afff7c-eed4-44b4-95c7-1437d8807b97
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-14/2018-11/26/10-39-49.639_converted/bia-integrator-data/S-BIAD815/51afff7c-eed4-44b4-95c7-1437d8807b97/51afff7c-eed4-44b4-95c7-1437d8807b97.zarr -> /bia-integrator-data/S-BIAD815/51afff7c-eed4-44b4-95c7-1437d8807b97/51afff7c-eed4-44b4-95c7-1437d8807b97.zarr
BEGIN
 mkngff_fileset 
----------------
        5811532
(1 row)
COMMIT
...

Running sql on a different server (using saved sql)

Zip and copy sql to a different server.
Unzip and update the SECRET in all sql files, getting current $SECRET as above
The replace didn't work using $SECRET etc in the regex, so just use actual values...
SECRETUUID is the default placeholder if you didn't use --secret option to create sql.

$ for i in $(ls); do sed -i 's/SECRETUUID/fc5d3566-eea0-412c-849e-daa6d3c6bfcc/g' $i; done

We want to execute all sql, using the csv, and also to use omero mkngff to do just the symlink creation...

$ for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
  omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/bia-integrator-data/$biapath/$uuid.zarr" --bfoptions
done

Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721
Creating dir at /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2017-03/07/16-50-40.721_mkngff/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr -> /bia-integrator-data/S-BIAD848/e45c988b-945e-49d6-8c6a-7284a2b0525e/e45c988b-945e-49d6-8c6a-7284a2b0525e.zarr

Now we can try viewing the images in webclient.
NB: sometimes this can take a while for the memo file to be regenerated. To check on the timings you can use unique string from the fileset name

grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "46.368_mkngff"

2023-08-29 12:21:51,993 DEBUG [                   loci.formats.Memoizer] (l.Server-4) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-2/2023-05/11/22-57-46.368_mkngff/HT20.ome.zarr/OME/.METADATA.ome.xml.bfmemo (3838714 bytes)
2023-08-29 12:21:51,993 DEBUG [                   loci.formats.Memoizer] (l.Server-4) start[1693309192879] time[2519114] tag[loci.formats.Memoizer.setId]
2023-08-29 12:21:51,995 INFO  [                ome.io.nio.PixelsService] (l.Server-4) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-2/2023-05/11/22-57-46.368_mkngff/HT20.ome.zarr/OME/METADATA.ome.xml Series: 0

E.g. 2519114 ms is 42 minutes.

The text was updated successfully, but these errors were encountered:

joshmoore · 2023-07-10T05:42:08Z

Where do we get the paths for making the mkdir command?

The output sql shows the path.

I see that all the examples have _converted in them. Is this part of the API?

It's hard-coded in the script at the moment but otherwise pretty arbitrary.

The // in the output is also arbitrary but you are right we could print the target directory there. 👍🏼

will-moore · 2023-07-10T08:24:48Z

On idr0125-pilot...

Created conda environment mkngff and installed omero-py and omero-mkngff as above...

Get Database password...

export OMERODIR=/opt/omero/server/OMERO.server
omero config get --show-password

Use psql to get SECRET (last session ID):

PGPASSWORD=****** psql -U omero -d idr -h 192.168.10.102
idr=> select uuid from (select * from session where node = 0 and owner = 0 and defaulteventtype = 'Sessions' order by id desc limit 3) x order by x.id asc limit 1;
                 uuid                 
--------------------------------------
 688fcb80-91f1-46ec-92ff-0547a1f6553x
(1 row)

export SECRET=688fcb80-91f1-46ec-92ff-0547a1f6553x
omero mkngff setup > setup.sql
omero login
omero mkngff sql --secret=$SECRET 5287119 "/idr0054/zarr/Tonsil 1.ome.zarr/" > idr0054.sql
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18 // 09-50-49.857 for fileset 5287119

SQL has paths like demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//idr0054/zarr/Tonsil 1.ome.zarr/OME/

Create symlinks...

$ sudo -u omero-server mkdir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted
$ sudo -u omero-server ln -s "/idr0054/zarr/Tonsil 1.ome.zarr" "/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/Tonsil 1.ome.zarr"

$ ls "/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/Tonsil 1.ome.zarr"
0  OME

will-moore · 2023-07-10T09:29:36Z

$ PGPASSWORD=****** psql -U omero -d idr -h 192.168.10.102 -f setup.sql
CREATE FUNCTION

$ PGPASSWORD=****** psql -U omero -d idr -h 192.168.10.102 -f idr0054.sql 
BEGIN
psql:idr0054.sql:952: ERROR:  cannot set original repo property without secret key
CONTEXT:  PL/pgSQL function _protect_originalfile_repo_insert() line 28 at RAISE
SQL statement "insert into originalfile
          (id, permissions, creation_id, group_id, owner_id, update_id, mimetype, repo, path, name)
          values (nextval('seq_originalfile'), old_perms, new_event, old_group, old_owner, new_event,
            info[i][3], repo, info[i][1], uuid || info[i][2])
          returning id"
PL/pgSQL function mkngff_fileset(bigint,character varying,character varying,character varying,text[]) line 42 at SQL statement
ROLLBACK

I don't know if that is a permissions issue?
I'm running the sql commands there a bit differently from instructions above.

Trying those instructions...

$ sudo -u postgres psql idr < setup.sql
could not change directory to "/home/wmoore": Permission denied
CREATE FUNCTION

Maybe that's also a permissions issue, but don't know what's needed there?

will-moore · 2023-07-10T09:41:43Z

Looking at the sql it has:

begin;
    select mkngff_fileset(
      5287119,
      '3a4d3a3f-ac98-4b1a-84ab-XXXXXXXXXXX',
      'cdf35825-def1-4580-8d0b-XXXXXXXXXXX',
      'demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/',
      array[
          ['demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted//idr0054/zarr/Tonsil 1.ome.zarr/', '.zattrs', 'application/octet-stream'],
...

the 3a4d3a3f... value is the $SECRET that still matches:

select uuid from (select * from session where node = 0 and owner = 0 and defaulteventtype = 'Sessions' order by id desc limit 3) x order by x.id asc limit 1;

but I don't know what the cdf35825-def1-4580-8d0b-XXXXXXXXXXX value is?

joshmoore · 2023-07-12T11:22:44Z

but I don't know what the cdf35825-def1-4580-8d0b-XXXXXXXXXXX value is?

That's the repositoryID.

joshmoore · 2023-07-12T11:23:22Z

$ sudo -u postgres psql idr < setup.sql
could not change directory to "/home/wmoore": Permission denied
CREATE FUNCTION

This is just because sudo is trying to change the directory and can't. If you have correct SQL, it wasn't fatal.

joshmoore · 2023-07-12T11:23:59Z

psql:idr0054.sql:952: ERROR:  cannot set original repo property without secret key

This will mean it's the wrong secret key.

will-moore · 2023-07-13T07:20:58Z

With the instructions you provided for getting the secret key (at the top) it prints lots of info - including 3 session keys.
I have tried to create a less-ambiguous way of getting the secret key.
Is this the correct way to get the secret key?

select uuid from (select * from session where node = 0 and owner = 0 and defaulteventtype = 'Sessions' order by id desc limit 3) x order by x.id asc limit 1;

joshmoore · 2023-07-13T11:42:17Z

Part of the problem is that it's different on the pilots than it is from main servers. (It's based on the number of java processes that are started!)

will-moore · 2023-08-08T01:58:27Z

OK, so it's different on pilots from other servers. So what should I do?

Trying this on idr0125-pilot...

idr=> select uuid from (select * from session where node = 0 and owner = 0 and defaulteventtype = 'Sessions' order by id desc limit 3) x order by x.id asc limit 1;
                 uuid                 
--------------------------------------
 3a4d3a3f-ac98-4b1a-84ab-5534e364f181
(1 row)

$ export SECRET=3a4d3a3f-ac98-4b1a-84ab-5534e364f181

Followed same steps as above and got the same error...

PGPASSWORD=****** psql -U omero -d idr -h 192.168.10.102 -f idr0054.sql 
BEGIN
psql:idr0054.sql:952: ERROR:  cannot set original repo property without secret key

So this seems to be not the right way to get the secret on the pilots.
How do I need to adapt this to work on the pilots?

joshmoore · 2023-08-08T08:51:12Z

Sorry. The "limit 3" in the above is for a full server (with Indexer and PixelData process). Changing that to limit 1 should do the trick:

select uuid from (select * from session where node = 0 and owner = 0 and defaulteventtype = 'Sessions' order by id desc limit 1) x order by x.id asc limit 1;

will-moore · 2023-08-08T12:55:13Z

idr=> select uuid from (select * from session where node = 0 and owner = 0 and defaulteventtype = 'Sessions' order by id desc limit 1) x order by x.id asc limit 1;
                 uuid                 
--------------------------------------
 8add790d-7855-46f6-8239-c6a72937d572
(1 row)

export SECRET=8add790d-7855-46f6-8239-c6a72937d572
omero login
omero mkngff sql --secret=$SECRET 5287119 "/idr0054/zarr/Tonsil 1.ome.zarr/" > idr0054.sql
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18 // 09-50-49.857 for fileset 5287119

PGPASSWORD=****** psql -U omero -d idr -h 192.168.10.102 -f idr0054.sql
BEGIN
 mkngff_fileset 
----------------
        5287366
(1 row)

COMMIT

will-moore · 2023-08-08T14:26:11Z

Getting ResourceError when viewing the image.
Need to update ZarrReader to latest merge build. Following steps at IDR/idr-metadata#643 (comment)

Get link from https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/499/default/artifact/bio-formats-build/ZarrReader/target/

sudo -u omero-server -s
ssh -A -o 'ProxyCommand ssh idr-pilot.openmicroscopy.org -W %h:%p' idr0125-omeroreadwrite -L 1080:localhost:80

wget https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/499/default/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar

$ mv OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar OMEZarrReader.jar
$ rm /opt/omero/server/OMERO.server/lib/client/OMEZarrReader.jar
$ cp OMEZarrReader.jar /opt/omero/server/OMERO.server/lib/client/
$ rm /opt/omero/server/OMERO.server/lib/server/OMEZarrReader.jar
$ cp OMEZarrReader.jar /opt/omero/server/OMERO.server/lib/server/

# as regular user
$ sudo service omero-server restart

But I still see ResourceError when viewing the image...

ResourceError

Traceback (most recent call last):

  File "/opt/omero/web/venv3/lib64/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)

  File "/opt/omero/web/venv3/lib64/python3.6/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)

  File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omeroweb/decorators.py", line 538, in wrapped
    retval = f(request, *args, **kwargs)

  File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omeroweb/decorators.py", line 597, in wrapper
    context = f(request, *args, **kwargs)

  File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omeroweb/webclient/views.py", line 1777, in load_metadata_preview
    rdefId = manager.image.getRenderingDefId()

  File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omero/gateway/__init__.py", line 7944, in wrapped
    if not self._prepareRenderingEngine() \

  File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omero/gateway/__init__.py", line 8220, in _prepareRenderingEngine
    self._re = self._prepareRE(rdid=rdid)

  File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omero/gateway/__init__.py", line 8199, in _prepareRE
    re.load(ctx)

  File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omero/gateway/__init__.py", line 4859, in __call__
    return self.handle_exception(e, *args, **kwargs)

  File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omeroweb/webclient/webclient_gateway.py", line 2109, in handle_exception
    super(OmeroWebSafeCallWrapper, self).handle_exception(e, *args, **kwargs)

  File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omero/gateway/__init__.py", line 4856, in __call__
    return self.f(*args, **kwargs)

  File "/opt/omero/web/venv3/lib64/python3.6/site-packages/omero_api_RenderingEngine_ice.py", line 1192, in load
    return _M_omero.api.RenderingEngine._op_load.invoke(self, ((), _ctx))

omero.ResourceError: exception ::omero::ResourceError
{
    serverStackTrace = ome.conditions.ResourceError: Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/idr0054/zarr/Tonsil 1.ome.zarr/OME/METADATA.ome.xml
	at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:907)
	at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:653)
	at ome.io.nio.PixelsService.getPixelBuffer(PixelsService.java:571)
	at ome.services.RenderingBean$12.doWork(RenderingBean.java:2205)
	at jdk.internal.reflect.GeneratedMethodAccessor350.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:568)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at ome.security.basic.EventHandler.invoke(EventHandler.java:154)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:119)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
	at com.sun.proxy.$Proxy92.doWork(Unknown Source)
	at ome.services.util.Executor$Impl.execute(Executor.java:447)
	at ome.services.util.Executor$Impl.execute(Executor.java:392)
	at ome.services.RenderingBean.getPixelBuffer(RenderingBean.java:2202)
	at ome.services.RenderingBean.load(RenderingBean.java:417)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
	at com.sun.proxy.$Proxy121.load(Unknown Source)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:93)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:43)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
	at com.sun.proxy.$Proxy121.load(Unknown Source)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at ome.services.blitz.util.IceMethodInvoker.invoke(IceMethodInvoker.java:172)
	at ome.services.throttling.Callback.run(Callback.java:56)
	at ome.services.throttling.InThreadThrottlingStrategy.callInvokerOnRawArgs(InThreadThrottlingStrategy.java:56)
	at ome.services.blitz.impl.AbstractAmdServant.callInvokerOnRawArgs(AbstractAmdServant.java:140)
	at ome.services.blitz.impl.RenderingEngineI.load_async(RenderingEngineI.java:316)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at omero.cmd.CallContext.invoke(CallContext.java:85)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
	at com.sun.proxy.$Proxy123.load_async(Unknown Source)
	at omero.api._RenderingEngineTie.load_async(_RenderingEngineTie.java:248)
	at omero.api._RenderingEngineDisp.___load(_RenderingEngineDisp.java:1223)
	at omero.api._RenderingEngineDisp.__dispatch(_RenderingEngineDisp.java:2405)
	at IceInternal.Incoming.invoke(Incoming.java:221)
	at Ice.ConnectionI.invokeAll(ConnectionI.java:2536)
	at Ice.ConnectionI.dispatch(ConnectionI.java:1145)
	at Ice.ConnectionI.message(ConnectionI.java:1056)
	at IceInternal.ThreadPool.run(ThreadPool.java:395)
	at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
	at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
	at java.base/java.lang.Thread.run(Thread.java:829)

    serverExceptionClass = ome.conditions.ResourceError
    message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/idr0054/zarr/Tonsil 1.ome.zarr/OME/METADATA.ome.xml
}

<WSGIRequest: GET '/webclient/metadata_preview/image/5025551/?_=1691504442078'>

joshmoore · 2023-08-14T09:33:42Z

The contents of your _converted directory are:

[jamoore@pilot-idr0125-omeroreadwrite log]$ ls /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/
Tonsil 1.ome.zarr

but the error messages says:

2023-08-08 13:38:15,237 INFO  [        ome.services.util.ServiceHandler] (l.Server-2)  Excp:    ome.conditions.ResourceError: Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/idr0054/zarr/Tonsil 1.ome.zarr/OME/METADATA.ome.xml

i.e. there are two extra directory paths: "idr0054/zarr/"

will-moore · 2023-08-15T15:55:09Z

# remove symlink and re-create...
rm "/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/Tonsil 1.ome.zarr"
sudo -u omero-server mkdir -p /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/idr0054/zarr/
sudo -u omero-server ln -s "/idr0054/zarr/Tonsil 1.ome.zarr" "/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/idr0054/zarr/Tonsil 1.ome.zarr"

Looks better...

$ ls /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/idr0054/zarr/Tonsil\ 1.ome.zarr/
0  OME

But I still see:

serverStackTrace = ome.conditions.ResourceError: Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/idr0054/zarr/Tonsil 1.ome.zarr/OME/METADATA.ome.xml
	at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:907)
	at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:653)

joshmoore · 2023-08-16T07:32:54Z

Hmmm..... did you possibly run the script twice? In the server log I see:

Caused by: ome.conditions.SecurityViolation:
file demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857_converted/idr0054/zarr/Tonsil 1.ome.zarr/0/.zattrs
does not have a unique readable fileset

will-moore · 2023-08-16T10:00:46Z

Trying with a different Fileset on idr0125-pilot...

$ omero mkngff sql --secret=$SECRET 5287120 "/idr0054/zarr/Tonsil 2.ome.zarr/" > idr0054_2.sql
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18 // 09-51-02.628 for fileset 5287120

Extrapolate the dirs we need for symlinks... demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628_converted/

sudo -u omero-server mkdir -p /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628_converted/idr0054/zarr/
sudo -u omero-server ln -s "/idr0054/zarr/Tonsil 2.ome.zarr" "/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628_converted/idr0054/zarr/Tonsil 2.ome.zarr"

Run sql...

$ psql -U omero -d idr -h 192.168.10.102 -f setup.sql
CREATE FUNCTION
$ psql -U omero -d idr -h 192.168.10.102 -f idr0054_2.sql
BEGIN
 mkngff_fileset 
----------------
        5287367
(1 row)

COMMIT

Rendering image...

serverExceptionClass = ome.conditions.ResourceError
    message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628_converted/idr0054/zarr/Tonsil 2.ome.zarr/OME/METADATA.ome.xml

will-moore · 2023-08-16T10:05:37Z

Blitz log has same error as before:

2023-08-16 10:04:34,084 ERROR [                ome.io.nio.PixelsService] (l.Server-6) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628_converted/idr0054/zarr/Tonsil 2.ome.zarr/OME/METADATA.ome.xml
java.lang.RuntimeException: ome.conditions.SecurityViolation: file demo_2/Blitz-0-Ice.ThreadPool.Server-1/2023-05/18/09-51-02.628_converted/idr0054/zarr/Tonsil 2.ome.zarr/.zattrs does not have a unique readable fileset
	at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79)

will-moore · 2023-08-16T11:01:18Z

Trying different server: idr0138-pilot... install miniconda etc...

$ wget https://repo.anaconda.com/miniconda/Miniconda3-py39_4.10.3-Linux-x86_64.sh
$ bash Miniconda3-py39_4.10.3-Linux-x86_64.sh
$ source /home/wmoore/.bashrc
$ conda create -n mkngff -c conda-forge -c ome omero-py bioformats2raw
$ conda activate mkngff
$ pip install 'omero-mkngff @ git+https://github.com/joshmoore/omero-mkngff@main'

$ psql -U omero -d idr -h 192.168.10.231
psql (11.16)
Type "help" for help.

idr=> select uuid from (select * from session where node = 0 and owner = 0 and defaulteventtype = 'Sessions' order by id desc limit 1) x order by x.id asc limit 1;
                 uuid                 
--------------------------------------
 e1571d5f-ff16-4e79-b0d3-0352d4c4df44
(1 row)

Need to mount BIA s3 on idr0138-pilot....

sudo mkdir /bia-integrator-data && sudo /opt/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other bia-integrator-data /bia-integrator-data

$ omero mkngff sql --secret=$SECRET 1591301 "/idr0054/zarr/Tonsil 1.ome.zarr/" > idr0054.sql
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15 // 15-27-40.023 for fileset 1591301

demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023

sudo -u omero-server mkdir -p /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_converted/idr0054/zarr/
$ sudo -u omero-server ln -s "/idr0054/zarr/Tonsil 1.ome.zarr" "/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_converted/idr0054/zarr/Tonsil 1.ome.zarr"

$ ls /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_converted/idr0054/zarr/Tonsil\ 1.ome.zarr/

$ psql -U omero -d idr -h 192.168.10.231 -f setup.sql
$ psql -U omero -d idr -h 192.168.10.231 -f idr0054.sql
BEGIN

 mkngff_fileset 
----------------
        5811532
(1 row)

COMMIT

Rendering fails - expected since we don't have latest ZarrReader...

2023-08-16 10:54:40,374 ERROR [                ome.io.nio.PixelsService] (.Server-10) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_converted/idr0054/zarr/Tonsil 1.ome.zarr/OME/METADATA.ome.xml
java.lang.RuntimeException: java.lang.IllegalArgumentException: Invalid index: -1

Update ZarrReader on idr0138-pilot:

wget https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/default/507/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar
mv OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar OMEZarrReader.jar

$ rm /opt/omero/server/OMERO.server/lib/client/OMEZarrReader.jar
rm: cannot remove ‘/opt/omero/server/OMERO.server/lib/client/OMEZarrReader.jar’: No such file or directory

Hmm - seems that idr0138 doesn't have ZarrReader - can we just add it?

sudo cp OMEZarrReader.jar /opt/omero/server/OMERO.server/lib/client/
sudo cp OMEZarrReader.jar /opt/omero/server/OMERO.server/lib/server/
sudo service omero-server restart
Redirecting to /bin/systemctl restart omero-server.service

Try rendering again - now we get same error as on idr0125-pilot:

2023-08-16 11:00:20,702 ERROR [                ome.io.nio.PixelsService] (l.Server-8) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_converted/idr0054/zarr/Tonsil 1.ome.zarr/OME/METADATA.ome.xml
java.lang.RuntimeException: ome.conditions.SecurityViolation: file demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_converted/idr0054/zarr/Tonsil 1.ome.zarr/.zattrs does not have a unique readable fileset
	at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79)

will-moore · 2023-09-21T13:35:23Z

Testing execute sql from one server on another with idr0033.
Sql generated on idr-testing, copied to idr0125-pilot, updated SECRET etc (also symlinks created).

$ psql -U omero -d idr -h $DBHOST -f 23119.sql 
UPDATE 3456
BEGIN
 mkngff_fileset 
----------------
        5287787
(1 row)

COMMIT

Checking image... http://localhost:1040/webclient/?show=image-3249534

(base) bash-4.2$ grep -A 2 "458_mkngff/9c1a5073" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "saved memo"
2023-09-21 14:54:41,906 DEBUG [                   loci.formats.Memoizer] (l.Server-5) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/07/23-50-06.458_mkngff/9c1a5073-3ebd-4e04-894c-900022e554e1.zarr/OME/.METADATA.ome.xml.bfmemo (10558639 bytes)
2023-09-21 14:54:41,906 DEBUG [                   loci.formats.Memoizer] (l.Server-5) start[1695303204087] time[4877818] tag[loci.formats.Memoizer.setId]
2023-09-21 14:54:41,906 INFO  [                ome.io.nio.PixelsService] (l.Server-5) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/07/23-50-06.458_mkngff/9c1a5073-3ebd-4e04-894c-900022e554e1.zarr/OME/METADATA.ome.xml Series: 9

4877818 ms is 81 minutes

will-moore · 2023-09-21T16:09:14Z

Test whether we need to add Directory rows to the NGFF Filesets.

On idr0125-pilot, using the sql from idr-testing, remove rows with Directory from sql...

$ sed -i '/Directory/d' 23120.sql

$ psql -U omero -d idr -h $DBHOST -f 23120.sql
UPDATE 3456
BEGIN
 mkngff_fileset 
----------------
        5287789
(1 row)
COMMIT

http://localhost:1040/webclient/?show=well-1316985 ...

This is viewable with

File count: 21525 compared with
File count: 39206 for the Filesets above (containing Directories).

(base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ grep -A 2 "252_mkngff/7e7bc" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "saved memo"
2023-09-21 17:40:22,272 DEBUG [                   loci.formats.Memoizer] (.Server-10) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/08/03-28-36.252_mkngff/7e7bc669-6cc3-460f-bac5-9f84f9237b0e.zarr/OME/.METADATA.ome.xml.bfmemo (10558531 bytes)
2023-09-21 17:40:22,272 DEBUG [                   loci.formats.Memoizer] (.Server-10) start[1695312725659] time[5296612] tag[loci.formats.Memoizer.setId]
2023-09-21 17:40:22,273 INFO  [                ome.io.nio.PixelsService] (.Server-10) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/08/03-28-36.252_mkngff/7e7bc669-6cc3-460f-bac5-9f84f9237b0e.zarr/OME/METADATA.ome.xml Series: 117
--
2023-09-21 18:57:56,345 DEBUG [                   loci.formats.Memoizer] (l.Server-3) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/08/03-28-36.252_mkngff/7e7bc669-6cc3-460f-bac5-9f84f9237b0e.zarr/OME/.METADATA.ome.xml.bfmemo (10558487 bytes)
2023-09-21 18:57:56,345 DEBUG [                   loci.formats.Memoizer] (l.Server-3) start[1695317413272] time[5263072] tag[loci.formats.Memoizer.setId]
2023-09-21 18:57:56,345 INFO  [                ome.io.nio.PixelsService] (l.Server-3) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/08/03-28-36.252_mkngff/7e7bc669-6cc3-460f-bac5-9f84f9237b0e.zarr/OME/METADATA.ome.xml Series: 117

5296612 ms is 88 minutes.

will-moore · 2023-10-06T11:52:42Z

Testing https://github.com/yandex-cloud/geesefs as an alternative to goofys...

On idr0125-pilot...

$ wget https://github.com/yandex-cloud/geesefs/releases/latest/download/geesefs-linux-amd64
$ chmod +x geesefs-linux-amd64
$ sudo mv geesefs-linux-amd64 /opt/geesefs

# similar command and options as goofys...

$ sudo mkdir /bia-integrator-geesefs
$ sudo /opt/geesefs --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other bia-integrator-data /bia-integrator-geesefs
s3.INFO anonymous bucket detected
main.INFO File system has been successfully mounted.

$ ls /bia-integrator-geesefs/S-BIAD861
0063750d-cd10-4759-8eca-c706a07b6693
...

will-moore · 2023-10-06T13:07:45Z

Installed geesefs on idr-testing:omeroreadwrite as above.

Try running mkngff there using geesefs compared with using goofys.
As wmoore user...

screen -S mkngff
cd
mkdir idr0013_geese
wget https://raw.githubusercontent.com/IDR/idr-utils/cac35aa0d1731afb5db0ab6b60e10bdf03c591fd/scripts/ngff_filesets/idr0013.csv


for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  omero mkngff sql $fsid "/bia-integrator-geesefs/$biapath/$uuid.zarr" > "idr0013_geese/$fsid.sql"
done

will-moore · 2023-10-09T07:07:38Z

mkngff is still running on idr0013.csv on idr-testing above...
Hasn't failed after 337 sql generated...

[wmoore@test120-omeroreadwrite ~]$ ls idr0013_geese/ | wc
    337     337    3370

Since 1pm on 6th Oct approx 67 hours -> 12 minutes per sql on average, although the timings are quite variable...
E.g. range of 27 minutes to 3 minutes in sample below...

[wmoore@test120-omeroreadwrite ~]$ for r in $(cat idr0013.csv); do   fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]');   ls -alh idr0013_geese/$fsid.sql; done
...
-rw-rw-r--. 1 wmoore wmoore 694K Oct  9 03:37 idr0013_geese/18781.sql
-rw-rw-r--. 1 wmoore wmoore 694K Oct  9 03:41 idr0013_geese/18572.sql
-rw-rw-r--. 1 wmoore wmoore 215K Oct  9 03:52 idr0013_geese/18942.sql
-rw-rw-r--. 1 wmoore wmoore 701K Oct  9 04:31 idr0013_geese/18475.sql
-rw-rw-r--. 1 wmoore wmoore 701K Oct  9 04:58 idr0013_geese/18509.sql
-rw-rw-r--. 1 wmoore wmoore 694K Oct  9 05:07 idr0013_geese/18785.sql
-rw-rw-r--. 1 wmoore wmoore 694K Oct  9 05:21 idr0013_geese/18583.sql
-rw-rw-r--. 1 wmoore wmoore 694K Oct  9 05:44 idr0013_geese/18680.sql
-rw-rw-r--. 1 wmoore wmoore 701K Oct  9 05:50 idr0013_geese/18378.sql
-rw-rw-r--. 1 wmoore wmoore 701K Oct  9 06:25 idr0013_geese/18357.sql
-rw-rw-r--. 1 wmoore wmoore 701K Oct  9 06:47 idr0013_geese/18536.sql
-rw-rw-r--. 1 wmoore wmoore 576K Oct  9 06:50 idr0013_geese/18832.sql
-rw-rw-r--. 1 wmoore wmoore 0 Oct  9 06:50 idr0013_geese/18653.sql

at IDR#2

will-moore mentioned this issue Jul 10, 2023

Add new option to list pixels in getUsedFiles ome/ZarrReader#58

Closed

will-moore changed the title ~~Usage instructions....~~ Document mkngff workflow Aug 21, 2023

will-moore added this to NGFF conversion Aug 21, 2023

will-moore moved this to create new Fileset to replace original Fileset in NGFF conversion Aug 21, 2023

This was referenced Aug 22, 2023

Ignore chunks #5

Merged

idr0035-caie-drugresponse S-BIAD847 IDR/idr-metadata#639

Open

idr0012-fuchs-cellmorph S-BIAD845 IDR/idr-metadata#643

Open

will-moore mentioned this issue Sep 6, 2023

Testing mkngff on idr-testing IDR/idr-metadata#672

Open

16 tasks

will-moore mentioned this issue Sep 27, 2023

Filesets to swap IDR/idr-utils#56

Closed

will-moore mentioned this issue Oct 9, 2023

idr0090-ashdown-malaria S-BIAD882 IDR/idr-metadata#651

Open

will-moore mentioned this issue Oct 10, 2023

idr-next steps for NGFF upgrade IDR/idr-metadata#675

Open

14 tasks

will-moore added a commit to will-moore/omero-mkngff that referenced this issue Mar 28, 2024

Initial docs from existing issue

3289c53

at IDR#2

will-moore mentioned this issue Mar 28, 2024

Add link to usage docs #16

Open

will-moore moved this from check_pixels to Other issues (not studies) in NGFF conversion May 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Document mkngff workflow #2

Document mkngff workflow #2

will-moore commented Jul 7, 2023 •

edited

Loading

joshmoore commented Jul 10, 2023

will-moore commented Jul 10, 2023 •

edited

Loading

will-moore commented Jul 10, 2023

will-moore commented Jul 10, 2023

joshmoore commented Jul 12, 2023

joshmoore commented Jul 12, 2023

joshmoore commented Jul 12, 2023

will-moore commented Jul 13, 2023 •

edited

Loading

joshmoore commented Jul 13, 2023

will-moore commented Aug 8, 2023

joshmoore commented Aug 8, 2023

will-moore commented Aug 8, 2023

will-moore commented Aug 8, 2023

joshmoore commented Aug 14, 2023

will-moore commented Aug 15, 2023

joshmoore commented Aug 16, 2023

will-moore commented Aug 16, 2023

will-moore commented Aug 16, 2023

will-moore commented Aug 16, 2023

will-moore commented Sep 21, 2023 •

edited

Loading

will-moore commented Sep 21, 2023 •

edited

Loading

will-moore commented Oct 6, 2023

will-moore commented Oct 6, 2023

will-moore commented Oct 9, 2023 •

edited

Loading

Document mkngff workflow #2

Document mkngff workflow #2

Comments

will-moore commented Jul 7, 2023 • edited Loading

joshmoore commented Jul 10, 2023

will-moore commented Jul 10, 2023 • edited Loading

will-moore commented Jul 10, 2023

will-moore commented Jul 10, 2023

joshmoore commented Jul 12, 2023

joshmoore commented Jul 12, 2023

joshmoore commented Jul 12, 2023

will-moore commented Jul 13, 2023 • edited Loading

joshmoore commented Jul 13, 2023

will-moore commented Aug 8, 2023

joshmoore commented Aug 8, 2023

will-moore commented Aug 8, 2023

will-moore commented Aug 8, 2023

joshmoore commented Aug 14, 2023

will-moore commented Aug 15, 2023

joshmoore commented Aug 16, 2023

will-moore commented Aug 16, 2023

will-moore commented Aug 16, 2023

will-moore commented Aug 16, 2023

will-moore commented Sep 21, 2023 • edited Loading

will-moore commented Sep 21, 2023 • edited Loading

will-moore commented Oct 6, 2023

will-moore commented Oct 6, 2023

will-moore commented Oct 9, 2023 • edited Loading

will-moore commented Jul 7, 2023 •

edited

Loading

will-moore commented Jul 10, 2023 •

edited

Loading

will-moore commented Jul 13, 2023 •

edited

Loading

will-moore commented Sep 21, 2023 •

edited

Loading

will-moore commented Sep 21, 2023 •

edited

Loading

will-moore commented Oct 9, 2023 •

edited

Loading