Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

idr0011-ledesmafernandez-dad4 S-BIAD866 #642

Open
will-moore opened this issue Feb 22, 2023 · 28 comments
Open

idr0011-ledesmafernandez-dad4 S-BIAD866 #642

will-moore opened this issue Feb 22, 2023 · 28 comments
Labels
resubmitted to EBI

Comments

@will-moore
Copy link
Member

idr0011-ledesmafernandez-dad4
@will-moore will-moore moved this to test convert in NGFF conversion Feb 22, 2023
@dominikl dominikl moved this from test convert to re-import test image in NGFF conversion Feb 27, 2023
@pwalczysko
Copy link

pwalczysko commented Feb 28, 2023 •
edited
Loading

Import failed, see details below

2023-02-27 17:25:29,331 11258844   [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 1121916
2023-02-27 17:25:47,761 11277274   [l.Client-1] ERROR     ome.formats.importer.cli.ErrorHandler - INTERNAL_EXCEPTION: /data/ngff/idr0011/Plate2-TS-Blue-B.ome.zarr/A/1/0/0/0/0/0/0/0
java.lang.RuntimeException: Failure response on import!
Category: ::omero::grid::ImportRequest
Name: import-request-failure
Parameters: {stacktrace=java.lang.RuntimeException: omero.ValidationException
    serverStackTrace = "ome.conditions.ValidationException: could not insert: [ome.model.screen.Well]; SQL [insert into well (alpha, blue, "column", creation_id, external_id, group_id, owner_id, permissions, update_id, externalDescription, externalIdentifier, green, plate, red, "row", status, type, version, id) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]; constraint [well_col_row]; nested exception is org.hibernate.exception.ConstraintViolationException: could not insert: [ome.model.screen.Well]
                        	at org.springframework.orm.hibernate3.SessionFactoryUtils.convertHibernateAccessException(SessionFactoryUtils.java:660)
                        	at org.springframework.orm.hibernate3.HibernateAccessor.convertHibernateAccessException(HibernateAccessor.java:416)
                        	at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:125)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
                        	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
                        	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
                        	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
                        	at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
                        	at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
                        	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
                        	at com.sun.proxy.$Proxy100.doWork(Unknown Source)
                        	at ome.services.util.Executor$Impl.execute(Executor.java:447)
                        	at ome.services.util.Executor$Impl.execute(Executor.java:392)
                        	at ome.services.throttling.Adapter.run(Adapter.java:50)
                        	at ome.services.throttling.InThreadThrottlingStrategy.runnableCall(InThreadThrottlingStrategy.java:89)
                        	at ome.services.blitz.impl.AbstractAmdServant.runnableCall(AbstractAmdServant.java:154)
                        	at ome.services.blitz.impl.MetadataStoreI.saveToDB_async(MetadataStoreI.java:208)
                        	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                        	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                        	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                        	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
                        	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
                        	at omero.cmd.CallContext.invoke(CallContext.java:85)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
                        	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
                        	at com.sun.proxy.$Proxy139.saveToDB_async(Unknown Source)
                        	at omero.api._MetadataStoreTie.saveToDB_async(_MetadataStoreTie.java:84)
                        	at omero.api._MetadataStoreDisp.___saveToDB(_MetadataStoreDisp.java:258)
                        	at omero.api._MetadataStoreDisp.__dispatch(_MetadataStoreDisp.java:416)
                        	at IceInternal.Incoming.invoke(Incoming.java:221)
                        	at Ice.ConnectionI.invokeAll(ConnectionI.java:2536)
                        	at Ice.ConnectionI.dispatch(ConnectionI.java:1145)
                        	at Ice.ConnectionI.message(ConnectionI.java:1056)
                        	at IceInternal.ThreadPool.run(ThreadPool.java:395)
                        	at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
                        	at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
                        	at java.base/java.lang.Thread.run(Thread.java:829)
                        "
    serverExceptionClass = "ome.conditions.ValidationException"
    message = "could not insert: [ome.model.screen.Well]; SQL [insert into well (alpha, blue, "column", creation_id, external_id, group_id, owner_id, permissions, update_id, externalDescription, externalIdentifier, green, plate, red, "row", status, type, version, id) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]; constraint [well_col_row]; nested exception is org.hibernate.exception.ConstraintViolationException: could not insert: [ome.model.screen.Well]"
	at ome.formats.OMEROMetadataStoreClient.saveToDB(OMEROMetadataStoreClient.java:1958)
	at ome.services.blitz.repo.ManagedImportRequestI.importMetadata(ManagedImportRequestI.java:653)
	at ome.services.blitz.repo.ManagedImportRequestI.step(ManagedImportRequestI.java:523)
	at omero.cmd.HandleI.steps(HandleI.java:448)
	at omero.cmd.HandleI$RunSteps.innerWork(HandleI.java:509)
	at omero.cmd.HandleI$2.doWork(HandleI.java:383)
	at omero.cmd.HandleI$2.doWork(HandleI.java:380)
	at jdk.internal.reflect.GeneratedMethodAccessor302.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:568)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at ome.security.basic.EventHandler.invoke(EventHandler.java:154)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:119)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
	at com.sun.proxy.$Proxy100.doWork(Unknown Source)
	at ome.services.util.Executor$Impl.execute(Executor.java:447)
	at omero.cmd.HandleI.run(HandleI.java:379)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at ome.services.util.Executor$Impl$1.call(Executor.java:488)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: omero.ValidationException
    serverStackTrace = "ome.conditions.ValidationException: could not insert: [ome.model.screen.Well]; SQL [insert into well (alpha, blue, "column", creation_id, external_id, group_id, owner_id, permissions, update_id, externalDescription, externalIdentifier, green, plate, red, "row", status, type, version, id) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]; constraint [well_col_row]; nested exception is org.hibernate.exception.ConstraintViolationException: could not insert: [ome.model.screen.Well]
                        	at org.springframework.orm.hibernate3.SessionFactoryUtils.convertHibernateAccessException(SessionFactoryUtils.java:660)
                        	at org.springframework.orm.hibernate3.HibernateAccessor.convertHibernateAccessException(HibernateAccessor.java:416)
                        	at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:125)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
                        	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
                        	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
                        	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
                        	at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
                        	at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
                        	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
                        	at com.sun.proxy.$Proxy100.doWork(Unknown Source)
                        	at ome.services.util.Executor$Impl.execute(Executor.java:447)
                        	at ome.services.util.Executor$Impl.execute(Executor.java:392)
                        	at ome.services.throttling.Adapter.run(Adapter.java:50)
                        	at ome.services.throttling.InThreadThrottlingStrategy.runnableCall(InThreadThrottlingStrategy.java:89)
                        	at ome.services.blitz.impl.AbstractAmdServant.runnableCall(AbstractAmdServant.java:154)
                        	at ome.services.blitz.impl.MetadataStoreI.saveToDB_async(MetadataStoreI.java:208)
                        	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                        	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                        	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                        	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
                        	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
                        	at omero.cmd.CallContext.invoke(CallContext.java:85)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
                        	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
                        	at com.sun.proxy.$Proxy139.saveToDB_async(Unknown Source)
                        	at omero.api._MetadataStoreTie.saveToDB_async(_MetadataStoreTie.java:84)
                        	at omero.api._MetadataStoreDisp.___saveToDB(_MetadataStoreDisp.java:258)
                        	at omero.api._MetadataStoreDisp.__dispatch(_MetadataStoreDisp.java:416)
                        	at IceInternal.Incoming.invoke(Incoming.java:221)
                        	at Ice.ConnectionI.invokeAll(ConnectionI.java:2536)
                        	at Ice.ConnectionI.dispatch(ConnectionI.java:1145)
                        	at Ice.ConnectionI.message(ConnectionI.java:1056)
                        	at IceInternal.ThreadPool.run(ThreadPool.java:395)
                        	at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
                        	at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
                        	at java.base/java.lang.Thread.run(Thread.java:829)
                        "
    serverExceptionClass = "ome.conditions.ValidationException"
    message = "could not insert: [ome.model.screen.Well]; SQL [insert into well (alpha, blue, "column", creation_id, external_id, group_id, owner_id, permissions, update_id, externalDescription, externalIdentifier, green, plate, red, "row", status, type, version, id) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]; constraint [well_col_row]; nested exception is org.hibernate.exception.ConstraintViolationException: could not insert: [ome.model.screen.Well]"
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
	at java.base/java.lang.Class.newInstance(Class.java:584)
	at IceInternal.BasicStream.createUserException(BasicStream.java:2785)
	at IceInternal.BasicStream.access$300(BasicStream.java:14)
	at IceInternal.BasicStream$EncapsDecoder11.throwException(BasicStream.java:3620)
	at IceInternal.BasicStream.throwException(BasicStream.java:2291)
	at IceInternal.OutgoingAsync.throwUserException(OutgoingAsync.java:399)
	at omero.api.MetadataStorePrxHelper.end_saveToDB(MetadataStorePrxHelper.java:715)
	at omero.api.MetadataStorePrxHelper.saveToDB(MetadataStorePrxHelper.java:593)
	at omero.api.MetadataStorePrxHelper.saveToDB(MetadataStorePrxHelper.java:580)
	at ome.formats.OMEROMetadataStoreClient.saveToDB(OMEROMetadataStoreClient.java:1942)
	... 36 more
, message=omero.ValidationException
    serverStackTrace = "ome.conditions.ValidationException: could not insert: [ome.model.screen.Well]; SQL [insert into well (alpha, blue, "column", creation_id, external_id, group_id, owner_id, permissions, update_id, externalDescription, externalIdentifier, green, plate, red, "row", status, type, version, id) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]; constraint [well_col_row]; nested exception is org.hibernate.exception.ConstraintViolationException: could not insert: [ome.model.screen.Well]
                        	at org.springframework.orm.hibernate3.SessionFactoryUtils.convertHibernateAccessException(SessionFactoryUtils.java:660)
                        	at org.springframework.orm.hibernate3.HibernateAccessor.convertHibernateAccessException(HibernateAccessor.java:416)
                        	at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:125)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
                        	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
                        	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
                        	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
                        	at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
                        	at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
                        	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
                        	at com.sun.proxy.$Proxy100.doWork(Unknown Source)
                        	at ome.services.util.Executor$Impl.execute(Executor.java:447)
                        	at ome.services.util.Executor$Impl.execute(Executor.java:392)
                        	at ome.services.throttling.Adapter.run(Adapter.java:50)
                        	at ome.services.throttling.InThreadThrottlingStrategy.runnableCall(InThreadThrottlingStrategy.java:89)
                        	at ome.services.blitz.impl.AbstractAmdServant.runnableCall(AbstractAmdServant.java:154)
                        	at ome.services.blitz.impl.MetadataStoreI.saveToDB_async(MetadataStoreI.java:208)
                        	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                        	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                        	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                        	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
                        	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
                        	at omero.cmd.CallContext.invoke(CallContext.java:85)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
                        	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
                        	at com.sun.proxy.$Proxy139.saveToDB_async(Unknown Source)
                        	at omero.api._MetadataStoreTie.saveToDB_async(_MetadataStoreTie.java:84)
                        	at omero.api._MetadataStoreDisp.___saveToDB(_MetadataStoreDisp.java:258)
                        	at omero.api._MetadataStoreDisp.__dispatch(_MetadataStoreDisp.java:416)
                        	at IceInternal.Incoming.invoke(Incoming.java:221)
                        	at Ice.ConnectionI.invokeAll(ConnectionI.java:2536)
                        	at Ice.ConnectionI.dispatch(ConnectionI.java:1145)
                        	at Ice.ConnectionI.message(ConnectionI.java:1056)
                        	at IceInternal.ThreadPool.run(ThreadPool.java:395)
                        	at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
                        	at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
                        	at java.base/java.lang.Thread.run(Thread.java:829)
                        "
    serverExceptionClass = "ome.conditions.ValidationException"
    message = "could not insert: [ome.model.screen.Well]; SQL [insert into well (alpha, blue, "column", creation_id, external_id, group_id, owner_id, permissions, update_id, externalDescription, externalIdentifier, green, plate, red, "row", status, type, version, id) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]; constraint [well_col_row]; nested exception is org.hibernate.exception.ConstraintViolationException: could not insert: [ome.model.screen.Well]"}

	at ome.formats.importer.ImportLibrary$ImportCallback.onFinished(ImportLibrary.java:807)
	at omero.cmd.CmdCallbackI.finished(CmdCallbackI.java:334)
	at omero.cmd._CmdCallbackDisp.___finished(_CmdCallbackDisp.java:118)
	at omero.cmd._CmdCallbackDisp.__dispatch(_CmdCallbackDisp.java:145)
	at IceInternal.Incoming.invoke(Incoming.java:221)
	at Ice.ConnectionI.invokeAll(ConnectionI.java:2536)
	at Ice.ConnectionI.dispatch(ConnectionI.java:1145)
	at Ice.ConnectionI.message(ConnectionI.java:1056)
	at IceInternal.ThreadPool.run(ThreadPool.java:395)
	at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
	at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
	at java.base/java.lang.Thread.run(Thread.java:829)

java.lang.RuntimeException: Failure response on import!
Category: ::omero::grid::ImportRequest
Name: import-request-failure
Parameters: {stacktrace=java.lang.RuntimeException: omero.ValidationException
    serverStackTrace = "ome.conditions.ValidationException: could not insert: [ome.model.screen.Well]; SQL [insert into well (alpha, blue, "column", creation_id, external_id, group_id, owner_id, permissions, update_id, externalDescription, externalIdentifier, green, plate, red, "row", status, type, version, id) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]; constraint [well_col_row]; nested exception is org.hibernate.exception.ConstraintViolationException: could not insert: [ome.model.screen.Well]
                        	at org.springframework.orm.hibernate3.SessionFactoryUtils.convertHibernateAccessException(SessionFactoryUtils.java:660)
                        	at org.springframework.orm.hibernate3.HibernateAccessor.convertHibernateAccessException(HibernateAccessor.java:416)
                        	at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:125)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
                        	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
                        	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
                        	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
                        	at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
                        	at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
                        	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
                        	at com.sun.proxy.$Proxy100.doWork(Unknown Source)
                        	at ome.services.util.Executor$Impl.execute(Executor.java:447)
                        	at ome.services.util.Executor$Impl.execute(Executor.java:392)
                        	at ome.services.throttling.Adapter.run(Adapter.java:50)
                        	at ome.services.throttling.InThreadThrottlingStrategy.runnableCall(InThreadThrottlingStrategy.java:89)
                        	at ome.services.blitz.impl.AbstractAmdServant.runnableCall(AbstractAmdServant.java:154)
                        	at ome.services.blitz.impl.MetadataStoreI.saveToDB_async(MetadataStoreI.java:208)
                        	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                        	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                        	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                        	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
                        	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
                        	at omero.cmd.CallContext.invoke(CallContext.java:85)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
                        	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
                        	at com.sun.proxy.$Proxy139.saveToDB_async(Unknown Source)
                        	at omero.api._MetadataStoreTie.saveToDB_async(_MetadataStoreTie.java:84)
                        	at omero.api._MetadataStoreDisp.___saveToDB(_MetadataStoreDisp.java:258)
                        	at omero.api._MetadataStoreDisp.__dispatch(_MetadataStoreDisp.java:416)
                        	at IceInternal.Incoming.invoke(Incoming.java:221)
                        	at Ice.ConnectionI.invokeAll(ConnectionI.java:2536)
                        	at Ice.ConnectionI.dispatch(ConnectionI.java:1145)
                        	at Ice.ConnectionI.message(ConnectionI.java:1056)
                        	at IceInternal.ThreadPool.run(ThreadPool.java:395)
                        	at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
                        	at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
                        	at java.base/java.lang.Thread.run(Thread.java:829)
                        "
    serverExceptionClass = "ome.conditions.ValidationException"
    message = "could not insert: [ome.model.screen.Well]; SQL [insert into well (alpha, blue, "column", creation_id, external_id, group_id, owner_id, permissions, update_id, externalDescription, externalIdentifier, green, plate, red, "row", status, type, version, id) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]; constraint [well_col_row]; nested exception is org.hibernate.exception.ConstraintViolationException: could not insert: [ome.model.screen.Well]"
	at ome.formats.OMEROMetadataStoreClient.saveToDB(OMEROMetadataStoreClient.java:1958)
	at ome.services.blitz.repo.ManagedImportRequestI.importMetadata(ManagedImportRequestI.java:653)
	at ome.services.blitz.repo.ManagedImportRequestI.step(ManagedImportRequestI.java:523)
	at omero.cmd.HandleI.steps(HandleI.java:448)
	at omero.cmd.HandleI$RunSteps.innerWork(HandleI.java:509)
	at omero.cmd.HandleI$2.doWork(HandleI.java:383)
	at omero.cmd.HandleI$2.doWork(HandleI.java:380)
	at jdk.internal.reflect.GeneratedMethodAccessor302.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:568)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at ome.security.basic.EventHandler.invoke(EventHandler.java:154)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:119)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
	at com.sun.proxy.$Proxy100.doWork(Unknown Source)
	at ome.services.util.Executor$Impl.execute(Executor.java:447)
	at omero.cmd.HandleI.run(HandleI.java:379)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at ome.services.util.Executor$Impl$1.call(Executor.java:488)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: omero.ValidationException
    serverStackTrace = "ome.conditions.ValidationException: could not insert: [ome.model.screen.Well]; SQL [insert into well (alpha, blue, "column", creation_id, external_id, group_id, owner_id, permissions, update_id, externalDescription, externalIdentifier, green, plate, red, "row", status, type, version, id) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]; constraint [well_col_row]; nested exception is org.hibernate.exception.ConstraintViolationException: could not insert: [ome.model.screen.Well]
                        	at org.springframework.orm.hibernate3.SessionFactoryUtils.convertHibernateAccessException(SessionFactoryUtils.java:660)
                        	at org.springframework.orm.hibernate3.HibernateAccessor.convertHibernateAccessException(HibernateAccessor.java:416)
                        	at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:125)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
                        	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
                        	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
                        	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
                        	at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
                        	at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
                        	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
                        	at com.sun.proxy.$Proxy100.doWork(Unknown Source)
                        	at ome.services.util.Executor$Impl.execute(Executor.java:447)
                        	at ome.services.util.Executor$Impl.execute(Executor.java:392)
                        	at ome.services.throttling.Adapter.run(Adapter.java:50)
                        	at ome.services.throttling.InThreadThrottlingStrategy.runnableCall(InThreadThrottlingStrategy.java:89)
                        	at ome.services.blitz.impl.AbstractAmdServant.runnableCall(AbstractAmdServant.java:154)
                        	at ome.services.blitz.impl.MetadataStoreI.saveToDB_async(MetadataStoreI.java:208)
                        	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                        	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                        	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                        	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
                        	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
                        	at omero.cmd.CallContext.invoke(CallContext.java:85)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
                        	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
                        	at com.sun.proxy.$Proxy139.saveToDB_async(Unknown Source)
                        	at omero.api._MetadataStoreTie.saveToDB_async(_MetadataStoreTie.java:84)
                        	at omero.api._MetadataStoreDisp.___saveToDB(_MetadataStoreDisp.java:258)
                        	at omero.api._MetadataStoreDisp.__dispatch(_MetadataStoreDisp.java:416)
                        	at IceInternal.Incoming.invoke(Incoming.java:221)
                        	at Ice.ConnectionI.invokeAll(ConnectionI.java:2536)
                        	at Ice.ConnectionI.dispatch(ConnectionI.java:1145)
                        	at Ice.ConnectionI.message(ConnectionI.java:1056)
                        	at IceInternal.ThreadPool.run(ThreadPool.java:395)
                        	at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
                        	at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
                        	at java.base/java.lang.Thread.run(Thread.java:829)
                        "
    serverExceptionClass = "ome.conditions.ValidationException"
    message = "could not insert: [ome.model.screen.Well]; SQL [insert into well (alpha, blue, "column", creation_id, external_id, group_id, owner_id, permissions, update_id, externalDescription, externalIdentifier, green, plate, red, "row", status, type, version, id) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]; constraint [well_col_row]; nested exception is org.hibernate.exception.ConstraintViolationException: could not insert: [ome.model.screen.Well]"
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
	at java.base/java.lang.Class.newInstance(Class.java:584)
	at IceInternal.BasicStream.createUserException(BasicStream.java:2785)
	at IceInternal.BasicStream.access$300(BasicStream.java:14)
	at IceInternal.BasicStream$EncapsDecoder11.throwException(BasicStream.java:3620)
	at IceInternal.BasicStream.throwException(BasicStream.java:2291)
	at IceInternal.OutgoingAsync.throwUserException(OutgoingAsync.java:399)
	at omero.api.MetadataStorePrxHelper.end_saveToDB(MetadataStorePrxHelper.java:715)
	at omero.api.MetadataStorePrxHelper.saveToDB(MetadataStorePrxHelper.java:593)
	at omero.api.MetadataStorePrxHelper.saveToDB(MetadataStorePrxHelper.java:580)
	at ome.formats.OMEROMetadataStoreClient.saveToDB(OMEROMetadataStoreClient.java:1942)
	... 36 more
, message=omero.ValidationException
    serverStackTrace = "ome.conditions.ValidationException: could not insert: [ome.model.screen.Well]; SQL [insert into well (alpha, blue, "column", creation_id, external_id, group_id, owner_id, permissions, update_id, externalDescription, externalIdentifier, green, plate, red, "row", status, type, version, id) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]; constraint [well_col_row]; nested exception is org.hibernate.exception.ConstraintViolationException: could not insert: [ome.model.screen.Well]
                        	at org.springframework.orm.hibernate3.SessionFactoryUtils.convertHibernateAccessException(SessionFactoryUtils.java:660)
                        	at org.springframework.orm.hibernate3.HibernateAccessor.convertHibernateAccessException(HibernateAccessor.java:416)
                        	at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:125)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
                        	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
                        	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
                        	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
                        	at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
                        	at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
                        	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
                        	at com.sun.proxy.$Proxy100.doWork(Unknown Source)
                        	at ome.services.util.Executor$Impl.execute(Executor.java:447)
                        	at ome.services.util.Executor$Impl.execute(Executor.java:392)
                        	at ome.services.throttling.Adapter.run(Adapter.java:50)
                        	at ome.services.throttling.InThreadThrottlingStrategy.runnableCall(InThreadThrottlingStrategy.java:89)
                        	at ome.services.blitz.impl.AbstractAmdServant.runnableCall(AbstractAmdServant.java:154)
                        	at ome.services.blitz.impl.MetadataStoreI.saveToDB_async(MetadataStoreI.java:208)
                        	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                        	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                        	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                        	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
                        	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
                        	at omero.cmd.CallContext.invoke(CallContext.java:85)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
                        	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
                        	at com.sun.proxy.$Proxy139.saveToDB_async(Unknown Source)
                        	at omero.api._MetadataStoreTie.saveToDB_async(_MetadataStoreTie.java:84)
                        	at omero.api._MetadataStoreDisp.___saveToDB(_MetadataStoreDisp.java:258)
                        	at omero.api._MetadataStoreDisp.__dispatch(_MetadataStoreDisp.java:416)
                        	at IceInternal.Incoming.invoke(Incoming.java:221)
                        	at Ice.ConnectionI.invokeAll(ConnectionI.java:2536)
                        	at Ice.ConnectionI.dispatch(ConnectionI.java:1145)
                        	at Ice.ConnectionI.message(ConnectionI.java:1056)
                        	at IceInternal.ThreadPool.run(ThreadPool.java:395)
                        	at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
                        	at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
                        	at java.base/java.lang.Thread.run(Thread.java:829)
                        "
    serverExceptionClass = "ome.conditions.ValidationException"
    message = "could not insert: [ome.model.screen.Well]; SQL [insert into well (alpha, blue, "column", creation_id, external_id, group_id, owner_id, permissions, update_id, externalDescription, externalIdentifier, green, plate, red, "row", status, type, version, id) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]; constraint [well_col_row]; nested exception is org.hibernate.exception.ConstraintViolationException: could not insert: [ome.model.screen.Well]"}

	at ome.formats.importer.ImportLibrary$ImportCallback.onFinished(ImportLibrary.java:807)
	at omero.cmd.CmdCallbackI.finished(CmdCallbackI.java:334)
	at omero.cmd._CmdCallbackDisp.___finished(_CmdCallbackDisp.java:118)
	at omero.cmd._CmdCallbackDisp.__dispatch(_CmdCallbackDisp.java:145)
	at IceInternal.Incoming.invoke(Incoming.java:221)
	at Ice.ConnectionI.invokeAll(ConnectionI.java:2536)
	at Ice.ConnectionI.dispatch(ConnectionI.java:1145)
	at Ice.ConnectionI.message(ConnectionI.java:1056)
	at IceInternal.ThreadPool.run(ThreadPool.java:395)
	at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
	at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
	at java.base/java.lang.Thread.run(Thread.java:829)
2023-02-27 17:25:47,768 11277281   [2-thread-1] ERROR        ome.formats.importer.ImportLibrary - Error on import
java.lang.Exception: Import failure
	at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:701)
	at ome.formats.importer.ImportLibrary$1.call(ImportLibrary.java:354)
	at ome.formats.importer.ImportLibrary$1.call(ImportLibrary.java:328)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
2023-02-27 17:25:47,768 11277281   [2-thread-1] INFO         ome.formats.importer.ImportLibrary - Exiting on error

==> Summary
9597 files uploaded, 0 filesets created, 0 images imported, 1 error in 3:07:36.884

@pwalczysko
Copy link

Updated the error with full stacktrace #642 (comment)

@pwalczysko
Copy link

ToDo

  • take ome-xml alone and import
  • will bfconvert show an error on the file ?

@dominikl dominikl added the bug label Mar 6, 2023
@pwalczysko
Copy link

cc @dgault

@pwalczysko pwalczysko mentioned this issue Mar 6, 2023
Open
@pwalczysko
Copy link

The import of METADATA.ome.xml file which was copied to local machine from /data/ngff/idr0011/Plate2-TS-Blue-B.ome.zarr/OME/METADATA.ome.xml was successful, see user-3 https://merge-ci.openmicroscopy.org/web/webclient/?show=image-258515

@dgault dgault mentioned this issue Mar 17, 2023
Merged
@pwalczysko
Copy link

pwalczysko commented Mar 28, 2023 •
edited
Loading

Imported the idr0011 plate on pilot-idrtesting - all looks fine, thumbs and full images are generated, no errors. The OMEZarrReader 0.3.1 was used, see IDR/deployment#380

Time to reimport this plate was 2h 15mins

@pwalczysko pwalczysko removed the bug label Mar 28, 2023
@pwalczysko pwalczysko moved this from re-import test image to convert all data to NGFF in NGFF conversion Mar 28, 2023
@dominikl dominikl assigned dominikl and unassigned dgault Jun 12, 2023
@dominikl
Copy link
Member

I'll start conversion to NGFF of the full dataset.

@dominikl
Copy link
Member

Converted files will be in pilot-zarr1-dev:/data/idr0011

@will-moore
Copy link
Member Author

Looks like this is done:

(base) [wmoore@pilot-zarr1-dev ~]$ ls /data/idr0011 | wc
    182     182    4555

Matches the 182 filesets expected (see IDR/idr-utils#56)

@will-moore will-moore moved this from convert all data to NGFF to upload data to s3 in NGFF conversion Jun 15, 2023
@dominikl
Copy link
Member

👍 It is, thanks Will.

@will-moore will-moore assigned will-moore and unassigned dominikl Jun 15, 2023
@will-moore
Copy link
Member Author

will-moore commented Jun 15, 2023 •
edited
Loading

Created bucket:

$ aws --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3 mb s3://idr0011
make_bucket: idr0011
$ aws --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3api put-bucket-policy --bucket idr0011 --policy file://policy.json
$ aws --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3api put-bucket-cors --bucket idr0011 --cors-configuration file://cors.json

$ /home/wmoore/mc cp -r idr0011/ uk1s3/idr0011/zarr
...
/data/idr0011/Target-2-B.ome.zarr/OME/METADATA.ome.xml:     469.25 GiB / 469.25 GiB ━━━━━━━━━━

# took several hours to upload, but looks like all complete
$ /home/wmoore/mc ls uk1s3/idr0011/zarr | wc
    182     910   10743
$ ls idr0011 | wc
    182     182    4555

e.g.
https://hms-dbmi.github.io/vizarr/?source=https://uk1s3.embassy.ebi.ac.uk/idr0011/zarr/Target-2-B.ome.zarr

Screenshot 2023-06-15 at 13 25 14

Validator: https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/idr0011/zarr/Target-2-B.ome.zarr

@will-moore will-moore moved this from upload data to s3 to create new Fileset to replace original Fileset in NGFF conversion Jun 16, 2023
@will-moore will-moore moved this from create new Fileset to replace original Fileset to upload some data to s3 and test in NGFF conversion Jun 26, 2023
@will-moore
Copy link
Member Author

Going to try import of a plate or two to validate on idr0125-pilot...

ssh -A -o 'ProxyCommand ssh idr-pilot.openmicroscopy.org -W %h:%p' idr0125-omeroreadwrite -L 1080:localhost:80

# list data on s3...
aws s3 ls --no-sign-request --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3://idr0011/zarr/ 

mkdir idr0011
cd idr0011
mkdir Target-1-A.ome.zarr && cd Target-1-A.ome.zarr
aws s3 sync --no-sign-request --exclude '*' --include "*/.z*" --include "*.xml" --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3://idr0011/zarr/Target-1-A.ome.zarr .

cd ../
mkdir Plate1-Blue-A.ome.zarr && cd Plate1-Blue-A.ome.zarr
aws s3 sync --no-sign-request --exclude '*' --include "*/.z*" --include "*.xml" --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3://idr0011/zarr/Plate1-Blue-A.ome.zarr .

cd ../../
sudo mv -f idr0011 /ngff/
sudo chown -R omero-server idr001

Import as usual...

cd /ngff/idr0011
for dir in *; do
  omero import --transfer=ln_s --depth=100 --name=${dir/.ome.zarr/} --skip=all $dir --file /tmp/$dir.log  --errs /tmp/$dir.err;
done

logs...

2023-06-26 12:41:15,068 236225     [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_PROCESSED Step: 4 of 5  Logfile: 50486066
2023-06-26 12:41:15,101 236258     [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5  Logfile: 50486066
2023-06-26 12:41:15,801 236958     [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /ngff/idr0011/Plate1-Blue-A.ome.zarr/OME/METADATA.ome.xml
Other imported objects:
Fileset:5287127

==> Summary
346 files uploaded, 1 fileset created, 48 images imported, 0 errors in 0:03:42.206

2023-06-26 12:42:32,512 74446      [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5  Logfile: 50486523
2023-06-26 12:42:32,830 74764      [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /ngff/idr0011/Target-1-A.ome.zarr/OME/METADATA.ome.xml
Other imported objects:
Fileset:5287128

==> Summary
150 files uploaded, 1 fileset created, 20 images imported, 0 errors in 0:01:06.269

Mount s3 buckets and update symlinks...

sudo mkdir /idr0011 && sudo /opt/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other idr0011 /idr0011

Can't see the Plates in webclient...
No Plates etc.
See what is linked to the Fileset(s) created above...

$ omero delete Fileset:5287128 --dry-run --report
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
omero.cmd.Delete2 Fileset:5287128 Dry run performed
ok
Steps: 4
Elapsed time: 2.03 secs.
Flags: []
Deleted objects
  Detector:155556-155558
  DetectorSettings:51647-51706
  Instrument:146154
  Objective:147902
  ObjectiveSettings:149099-149118
  CommentAnnotation:33516736
  FilesetAnnotationLink:5286926
  Channel:44137844-44137903
  Image:14834772-14834791
  LogicalChannel:16075189-16075248
  OriginalFile:50486523-50486673
  Pixels:14834772-14834791
  PlaneInfo:45114279-45115538
  Fileset:5287128
  FilesetEntry:35937074-35937223
  FilesetJobLink:21495711-21495714
  IndexingJob:21505878
  JobOriginalFileLink:5304542
  MetadataImportJob:21505876
  PixelDataJob:21505877
  UploadJob:21505875



$ omero delete Fileset:5287127 --dry-run --report
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
omero.cmd.Delete2 Fileset:5287127 Dry run performed
ok
Steps: 4
Elapsed time: 5.836 secs.
Flags: []
Deleted objects
  Detector:155553-155555
  DetectorSettings:51503-51646
  Instrument:146153
  Objective:147901
  ObjectiveSettings:149051-149098
  CommentAnnotation:33516735
  FilesetAnnotationLink:5286925
  Channel:44137700-44137843
  Image:14834724-14834771
  LogicalChannel:16075045-16075188
  OriginalFile:50486066-50486412
  Pixels:14834724-14834771
  PlaneInfo:45111255-45114278
  Fileset:5287127
  FilesetEntry:35936662-35936700,35936767-35937073
  FilesetJobLink:21495707-21495710
  IndexingJob:21505874
  JobOriginalFileLink:5304541
  MetadataImportJob:21505872
  PixelDataJob:21505873
  UploadJob:21505871

No Plates or Wells etc?! Looks like they got imported as Images (orphaned).

@will-moore
Copy link
Member Author

Realised that I'd failed to copy the top level .zattrs for the plates above. Updated them and tried again...

2023-06-26 15:01:10,877 186163     [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /ngff/idr0011/Plate1-Blue-A.ome.zarr/OME/METADATA.ome.xml
Other imported objects:
Fileset:5287129

==> Summary
347 files uploaded, 1 fileset, 1 plate created, 48 images imported, 0 errors in 0:02:51.928

2023-06-26 15:02:32,005 78658      [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5  Logfile: 50487383
2023-06-26 15:02:32,511 79164      [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /ngff/idr0011/Target-1-A.ome.zarr/OME/METADATA.ome.xml
Other imported objects:
Fileset:5287130

==> Summary
151 files uploaded, 1 fileset, 1 plate created, 20 images imported, 0 errors in 0:01:10.869

Wanted to find Plate linked to Fileset but...

$ omero delete Fileset:5287129 --dry-run --report
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
omero.cmd.Delete2 Fileset:5287129 failed: 'graph-fail'
failed: may not delete Image[14834826] while WellSample[9235200] remains
Steps: 4
Elapsed time: 3.318 secs.
Flags: [FAILURE, CANCELLED]

$ omero delete Fileset:5287130 --dry-run --report
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
omero.cmd.Delete2 Fileset:5287130 failed: 'graph-fail'
failed: may not delete Image[14834858] while WellSample[9235232] remains
Steps: 4
Elapsed time: 1.728 secs.
Flags: [FAILURE, CANCELLED]

Put the 2 Plates into a screen in webclient, then...

$ python idr-utils/scripts/managed_repo_symlinks.py Screen:3253 /idr0011/zarr/ --report

Fileset: 5287129 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-7/2023-06/26/14-58-19.152/
Render Image 14834806
fileset_dirs {}
fs_contents ['Plate1-Blue-A.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-7/2023-06/26/14-58-19.152/Plate1-Blue-A.ome.zarr to /idr0011/zarr/Plate1-Blue-A.ome.zarr

Fileset: 5287130 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-2/2023-06/26/15-01-21.806/
Render Image 14834851
fileset_dirs {}
fs_contents ['Target-1-A.ome.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-2/2023-06/26/15-01-21.806/Target-1-A.ome.zarr to /idr0011/zarr/Target-1-A.ome.zarr

Both plates look identical to IDR:

Image

@will-moore will-moore moved this from upload some data to s3 and test to Zip and upload to BioStudies in NGFF conversion Jun 26, 2023
@will-moore
Copy link
Member Author

Started zip creation...

ssh pilot-zarr1-dev
cd /data/idr0011
screen -S idr0011_zip
for i in */; do sudo zip -r "${i%/}.zip" "$i"; done

@will-moore
Copy link
Member Author

Last zip created this morning at Jun 28 03:34.
Took > 30 hours to zip 182 zarrs.

@will-moore
Copy link
Member Author

will-moore commented Jun 28, 2023 •
edited
Loading 

wget https://ak-delivery04-mul.dhe.ibm.com/sar/CMA/OSA/08q6g/0/ibm-aspera-cli-3.9.6.1467.159c5b1-linux-64-release.sh
chmod +x ibm-aspera-cli-3.9.6.1467.159c5b1-linux-64-release.sh 
bash ibm-aspera-cli-3.9.6.1467.159c5b1-linux-64-release.sh 
cd /.aspera/cli/bin
$ ./ascp -P33001 -i ../etc/asperaweb_id_dsa.openssh -d /data/idr0011/idr0011/ [email protected]:5f/136e8d-**********

...
Completed: 315655334K bytes transferred in 10586 seconds
 (244269K bits/sec), in 182 files, 1 directory.

@will-moore will-moore moved this from Zip and upload to BioStudies to BioStudies Submission in NGFF conversion Jun 28, 2023
@will-moore will-moore assigned francesw and unassigned will-moore Jun 28, 2023
@francesw francesw changed the title idr0011-ledesmafernandez-dad4 to NGFF idr0011-ledesmafernandez-dad4 S-BIAD866 Aug 31, 2023
@francesw francesw removed their assignment Aug 31, 2023
@francesw francesw moved this from BioStudies Submission to Data on Embassy s3 in NGFF conversion Aug 31, 2023
@will-moore will-moore moved this from Data on Embassy s3 to create new Filesets in idr-next in NGFF conversion Sep 1, 2023
@will-moore
Copy link
Member Author

will-moore commented Sep 1, 2023 •
edited
Loading

All 182 Filesets are viewable at https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/pages/S-BIAD866.html

Testing on idr0138-pilot with the first 10...

idr0011/Plate11-Green-B.ome.zarr,S-BIAD866/013015ae-64f8-494c-85ee-b4e364960b75,21832
idr0011/Plate2-Yellow-A.ome.zarr,S-BIAD866/029bd12c-9ee5-4278-a177-a14f52466abc,21757
idr0011/Plate16-Blue-B.ome.zarr,S-BIAD866/0980f7a9-2ab6-4116-b9a9-59fcbc5ffc63,21823
idr0011/Plate15-Yellow-A.ome.zarr,S-BIAD866/0e5e1147-449f-4a89-96c6-3a3c85f8c1a7,21851
idr0011/Plate9-Red-A.ome.zarr,S-BIAD866/0ebb098d-7b05-4d54-b9f0-128833580fba,21764
idr0011/Plate5-TS-Yellow-B-repeated.ome.zarr,S-BIAD866/10cd251a-f2ff-4623-99dc-a8aec9555c4f,21910
idr0011/Target-2-A.ome.zarr,S-BIAD866/1217032d-8a87-4bc7-8d22-665791732ec6,21935
idr0011/Plate6-Yellow-A.ome.zarr,S-BIAD866/121878f9-674a-45e1-a051-d883d19fbef3,21858
idr0011/Plate5-TS-Yellow-A.ome.zarr,S-BIAD866/129cf0d8-6f16-4817-bc6c-904c9731dcd7,21908
idr0011/Plate3-Yellow-A.ome.zarr,S-BIAD866/15c1d12d-2d14-4c7a-a90b-1864e59e741b,21790

Found prefix demo_2/2016-07/28 // 18-03-56.973 for fileset 21832
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-07/28/18-03-56.973
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-07/28/18-03-56.973_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-07/28/18-03-56.973_mkngff/013015ae-64f8-494c-85ee-b4e364960b75.zarr -> /bia-integrator-data/S-BIAD866/013015ae-64f8-494c-85ee-b4e364960b75/013015ae-64f8-494c-85ee-b4e364960b75.zarr
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/2016-07/28 // 15-41-13.359 for fileset 21757
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-07/28/15-41-13.359
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-07/28/15-41-13.359_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-07/28/15-41-13.359_mkngff/029bd12c-9ee5-4278-a177-a14f52466abc.zarr -> /bia-integrator-data/S-BIAD866/029bd12c-9ee5-4278-a177-a14f52466abc/029bd12c-9ee5-4278-a177-a14f52466abc.zarr
...

@will-moore
Copy link
Member Author 

...
 mkngff_fileset 
----------------
        5811635
(1 row)
COMMIT
UPDATE 22
BEGIN
 mkngff_fileset 
----------------
        5811636
(1 row)
COMMIT
UPDATE 19
BEGIN
 mkngff_fileset 
----------------
        5811637
(1 row)
COMMIT
UPDATE 48
BEGIN
 mkngff_fileset 
----------------
        5811638
(1 row)
COMMIT
UPDATE 27
BEGIN
 mkngff_fileset 
----------------
        5811639
(1 row)
COMMIT
UPDATE 48
BEGIN
 mkngff_fileset 
----------------
        5811640
(1 row)
COMMIT

Find last Fileset created...

=> select id from image where fileset = 5811640;
   id    
---------
2852632
...

http://localhost:1040/webclient/?show=image-2852632

Didn't take long to view images...
Screenshot 2023-09-01 at 17 40 17

@will-moore
Copy link
Member Author

will-moore commented Sep 11, 2023 •
edited
Loading

On idr-testing:omeroreadwrite...

Had to manually tweak Fileset names where the Plate names in IDR don't match the fileset names on disk:

idr0011/Plate1-Blue-A_TS-Stinger.ome.zarr,S-BIAD866/16014bb5-1926-49f6-80e0-b18cd6e3f3cd
idr0011/Plate1-Red-A-TS-Stinger.ome.zarr,S-BIAD866/1f03d4ec-2783-4d8e-94a1-ee4862ea3d22
idr0011/Plate1-Yellow-B-TS-Stinger.ome.zarr,S-BIAD866/4215f37e-483d-4471-ad57-c5c052a5bda3
idr0011/Plate1-Green-A-TS-Stinger.ome.zarr,S-BIAD866/42db5162-4855-4167-921d-4f99e7c12fea 
idr0011/Plate1-Blue-B-TS-Stinger.ome.zarr,S-BIAD866/82231f2b-aa52-4069-9ced-a8d1f1608422
idr0011/Plate1-Red-B-TS-Stinger.ome.zarr,S-BIAD866/88ca4912-eaaf-4f35-8777-8b649155007e
idr0011/Plate1-Yellow-A-TS-Stinger.ome.zarr,S-BIAD866/ced2d9c5-8eb9-4c48-938d-d768df0611c1
idr0011/Plate1-Green-B-TS-Stinger.ome.zarr,S-BIAD866/dc7c9a6c-db59-4652-a372-78a309594036
idr0011/TS-Stinger-Target-1&2.ome.zarr,S-BIAD866/f58816a3-311c-4b60-aa92-fbc85a619a79

Commited idr0011.csv to https://github.com/IDR/idr-utils/pull/56/files

Started mkngff... - took about 3.5 hours for 182 Filesets - about 1 minute each.

@will-moore
Copy link
Member Author

will-moore commented Sep 12, 2023 •
edited
Loading

Ran sql 6:30 am.

Viewing image from first plate to setId: http://localhost:1080/webclient/?show=image-2849940

$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "13-56-03.328_mkngff"
2023-09-12 05:34:53,887 DEBUG [                   loci.formats.Memoizer] (l.Server-4) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2016-07/28/13-56-03.328_mkngff/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f.zarr/OME/.METADATA.ome.xml.bfmemo (6989006 bytes)
2023-09-12 05:34:53,887 DEBUG [                   loci.formats.Memoizer] (l.Server-4) start[1694496832274] time[61613] tag[loci.formats.Memoizer.setId]
2023-09-12 05:34:53,887 INFO  [                ome.io.nio.PixelsService] (l.Server-4) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2016-07/28/13-56-03.328_mkngff/7f95aba3-cfbf-4ae8-a106-edaa36f5b07f.zarr/OME/METADATA.ome.xml Series: 0

61613ms is 1 minute

@will-moore will-moore mentioned this issue Sep 12, 2023
Open
16 tasks
@will-moore
Copy link
Member Author

will-moore commented Nov 14, 2023 •
edited
Loading

Four plates found to have missing files, although they all look OK where uploaded above. eg. https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/idr0011/zarr/Plate3-Yellow-B.ome.zarr

  • Plate3-Yellow-B
  • Plate11-Blue-A
  • Plate13-Red-B
  • Plate15-Yellow-A

We could try to download them from there, but probably easier to simply recreate them...

on zarr1-dev-pilot /data/idr0011

screen -S idr0011_bf2raw
conda activate bioformats2raw2
cd /data/idr0011
for i in Plate3-Yellow-B Plate11-Blue-A Plate13-Red-B Plate15-Yellow-A; do
~/bioformats2raw-0.6.0-24/bin/bioformats2raw --memo-directory ../memo  /uod/idr/metadata/idr0011-ledesmafernandez-dad4/screens/$i.screen $i.ome.zarr; done

EDIT: uploading...

./ascp -P33001 -i ../etc/asperaweb_id_dsa.openssh -d /data/idr0011/idr0011 [email protected]:/5f/xxxxxx
Plate11-Blue-A.ome.zarr.zip                                                          100% 1703MB  152Mb/s    02:29
Plate13-Red-B.ome.zarr.zip                                                          100% 1668MB  154Mb/s    04:40
Plate15-Yellow-A.ome.zarr.zip                                                       100% 1662MB  229Mb/s    06:45
Plate3-Yellow-B.ome.zarr.zip                                                        100% 1646MB  280Mb/s    09:07
Completed: 6841932K bytes transferred in 547 seconds
 (102354K bits/sec), in 4 files, 1 directory.

@will-moore
Copy link
Member Author

With check_pixels.py run on ALL images across 4 Screens: no Errors found except the 4 known Plates above:
IDR/idr-utils#55 (comment)

@will-moore
Copy link
Member Author

checking for updates...
https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/pages/S-BIAD866.html

No change...
idr0011/Plate11-Blue-A
https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD866/bd01548b-da85-4a66-8422-7b16995fabba/bd01548b-da85-4a66-8422-7b16995fabba.zarr/

@will-moore
Copy link
Member Author

will-moore commented Jan 4, 2024 •
edited
Loading

We have the 4 plates available on uk1s3 and they are all valid eg. https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/idr0011/zarr/Plate3-Yellow-B.ome.zarr

Let's use those for mkngff so as not to be held up by EBI re-processing them.
Plan is to run mkngff on those plates hosted on uk1s3 and apply the sql on idr-next.
When the data becomes available on BioStudies, we can update symlinks and clientpaths accordingly.

First, let's create sql and test on idr0125-pilot...

Use login to idr.openmicroscopy.org for mkngff sql (using original Fileset IDs), but we'll need to use different IDs for testing?

idr0011.csv based on IDR/idr-utils@125c4e5

idr0011/Plate15-Yellow-A.ome.zarr,Plate15-Yellow-A.ome.zarr,21851
idr0011/Plate13-Red-B.ome.zarr,Plate13-Red-B.ome.zarr,21842
idr0011/Plate3-Yellow-B.ome.zarr,Plate3-Yellow-B.ome.zarr,21789
idr0011/Plate11-Blue-A.ome.zarr,Plate11-Blue-A.ome.zarr,21834

for r in $(cat $IDRID.csv); do
  platename=$(echo $r | cut -d',' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  omero mkngff sql $fsid --clientpath="https://uk1s3.embassy.ebi.ac.uk/idr0011/zarr/$platename" "/idr0011/zarr/$platename" > "$IDRID/$fsid.sql"
done

$ for i in $(ls idr0011); do grep -c ".zattrs" idr0011/$i; done
98
98
98
98

Previously, sql were missing some files:

(base) Williams-MacBook-Pro:mkngff_upgrade_scripts wmoore$ for i in 21789 21834 21842 21851; do grep -c ".zattrs" ngff_filesets/idr0011/$i.sql; done
96
81
81
81

Fixed in IDR/mkngff_upgrade_scripts@1f2949c

@will-moore
Copy link
Member Author

Since idr0125-pilot has not had mkngff run on those 4 Plates, we can run sql with original Fileset IDs
as omero-server...

for i in $(ls); do sed -i 's/SECRETUUID/9630ba1e-ed3a-42e3-9296-.../g' $i; done

for r in $(cat $IDRID.csv); do
  platename=$(echo $r | cut -d',' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
  omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/idr0011/zarr/$platename" --bfoptions
done

(venv3) (base) bash-4.2$ echo $IDRID
idr0015
(venv3) (base) bash-4.2$ export IDRID=idr0011
(venv3) (base) bash-4.2$
(venv3) (base) bash-4.2$
(venv3) (base) bash-4.2$
(venv3) (base) bash-4.2$ for r in $(cat $IDRID.csv); do   platename=$(echo $r | cut -d',' -f2);   fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]');   psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql";   omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/idr0011/zarr/$platename" --bfoptions; done
UPDATE 48
BEGIN
 mkngff_fileset
----------------
        5289232
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-07/28/18-37-49.702
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-07/28/18-37-49.702_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-07/28/18-37-49.702_mkngff/Plate15-Yellow-A.ome.zarr -> /idr0011/zarr/Plate15-Yellow-A.ome.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-07/28/18-37-49.702
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2016-07/28/18-37-49.702_mkngff/Plate15-Yellow-A.ome.zarr.bfoptions
UPDATE 48
BEGIN
 mkngff_fileset 
----------------
        5289233
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-07/28/18-22-12.907
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-07/28/18-22-12.907_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-07/28/18-22-12.907_mkngff/Plate13-Red-B.ome.zarr -> /idr0011/zarr/Plate13-Red-B.ome.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-07/28/18-22-12.907
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2016-07/28/18-22-12.907_mkngff/Plate13-Red-B.ome.zarr.bfoptions
UPDATE 48
BEGIN
 mkngff_fileset 
----------------
        5289234
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-07/28/16-42-19.788
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-07/28/16-42-19.788_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-07/28/16-42-19.788_mkngff/Plate3-Yellow-B.ome.zarr -> /idr0011/zarr/Plate3-Yellow-B.ome.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-07/28/16-42-19.788
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2016-07/28/16-42-19.788_mkngff/Plate3-Yellow-B.ome.zarr.bfoptions
UPDATE 48
BEGIN
 mkngff_fileset 
----------------
        5289235
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-07/28/18-07-38.354
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-07/28/18-07-38.354_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-07/28/18-07-38.354_mkngff/Plate11-Blue-A.ome.zarr -> /idr0011/zarr/Plate11-Blue-A.ome.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-07/28/18-07-38.354
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2016-07/28/18-07-38.354_mkngff/Plate11-Blue-A.ome.zarr.bfoptions

@will-moore
Copy link
Member Author

will-moore commented Jan 4, 2024 •
edited
Loading

Check all images from 4 plates:

for i in 5534 5551 5542 5489; do
  python check_pixels.py Plate:$i --max-planes=sizeC >> /tmp/check_pix_20240104_idr0011.log;
done

(base) bash-4.2$ grep "Check Image" /tmp/check_pix_20240104_idr0011.log | wc
    192    1536   11576
(base) bash-4.2$ grep "Error" /tmp/check_pix_20240104_idr0011.log | wc
      0       0       0

@will-moore will-moore mentioned this issue Jan 5, 2024
Open
14 tasks
@will-moore
Copy link
Member Author

On idr-testing, test newly available on BIA...

idr0011.csv
Fileset IDs from idr-testing...

idr0011/Plate15-Yellow-A.ome.zarr,S-BIAD866/684211eb-77bc-4693-a904-851c79f19b28,6313184
idr0011/Plate13-Red-B.ome.zarr,S-BIAD866/a3781c5d-c4e2-4ad6-9d57-357069721845,6313214
idr0011/Plate3-Yellow-B.ome.zarr,S-BIAD866/311935f7-f3d5-4405-9456-d8c49f736f11,6313274
idr0011/Plate11-Blue-A.ome.zarr,S-BIAD866/e138116a-2215-4581-928e-73c32d667018,6313304
idr0011/Target-TS,S-BIAD866/ef7df8ac-0c98-47a6-944e-6e86f18f174a,6313354

export IDRID=idr0011
for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  omero mkngff sql $fsid --fs_suffix=None --clientpath="https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/$biapath/$uuid.zarr" "/bia-integrator-data/$biapath/$uuid.zarr" > "$IDRID/$fsid.sql"
done

for i in $(ls); do sed -i 's/SECRETUUID/f464e059-16b5-4013-b9a2-417e5976371c/g' $i; done

for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
  omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/bia-integrator-data/$biapath/$uuid.zarr" --fs_suffix=None --bfoptions
done

@will-moore
Copy link
Member Author

Manually updated idr0011 sql scripts for 5 plates to work with original Fileset IDs: IDR/mkngff_upgrade_scripts@e48a80b

@will-moore will-moore moved this from check_pixels to Round 2 - psql fileset IDs checked in NGFF conversion Mar 18, 2024
@will-moore will-moore moved this from Other issues (not studies) to NGFF studies in NGFF conversion May 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
resubmitted to EBI
Projects
NGFF conversion
Status: NGFF studies
Milestone
No milestone
Development

No branches or pull requests
5 participants
@will-moore @dgault @pwalczysko @dominikl @francesw