PublishDir step fails on push to S3 even though pipeline succeeds

[markpanganiban]

Bug report

Reported by a user.

I recently observed a run on Nextflow Tower that was reported to be completed successfully on platform, but the publish directory on S3 was empty.
The corresponding directory in the working directory was fully populated with all expected outputs.

Log file showed the following:

Nov-16 07:14:35.678 [PublishDir-1] WARN  nextflow.processor.PublishDir - Failed to publish file: s3://s3-path/scratch/2NQLCVMB9up84P/dd/be6eea71f1d896340271b30c751e39/results; to: s3://s3-path/users/kkosheleva/tower/EXP23002963/MAX078/results [copy] -- See log file for details
java.nio.file.NoSuchFileException: the path: /s3-path/users/kkosheleva/tower/EXP23002963/MAX078/results/variants not exists
    at com.upplication.s3fs.S3FileSystemProvider.delete(S3FileSystemProvider.java:561)
    at java.base/java.nio.file.Files.delete(Files.java:1152)
    at nextflow.file.FileHelper$3.postVisitDirectory(FileHelper.groovy:1007)
    at nextflow.file.FileHelper$3.postVisitDirectory(FileHelper.groovy)
    at java.base/java.nio.file.Files.walkFileTree(Files.java:2828)
    at java.base/java.nio.file.Files.walkFileTree(Files.java:2882)
    at nextflow.file.FileHelper.deleteDir0(FileHelper.groovy:999)
    at nextflow.file.FileHelper.deletePath(FileHelper.groovy:991)
    at nextflow.processor.PublishDir.processFile(PublishDir.groovy:390)
    at nextflow.processor.PublishDir.safeProcessFile(PublishDir.groovy:359)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1258)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
    at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:1036)
    at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:1019)
    at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:97)
    at nextflow.processor.PublishDir$_apply1_closure1.doCall(PublishDir.groovy:332)
    at nextflow.processor.PublishDir$_apply1_closure1.call(PublishDir.groovy)
    at groovy.lang.Closure.run(Closure.java:493)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:833)
Nov-16 07:14:35.679 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'PublishDir' shutdown completed (hard=false)

[pditommaso]

What's the nextflow version ?

[markpanganiban]

What's the nextflow version ?

23.04.2.

[bentsherman]

I think we can avoid this error by ignoring the NoSuchFileException on the directory delete, since we're deleting the whole thing anyway.

[bentsherman]

Actually it should be solved by #3933 , which will only publish individual files and will never need to delete a directory.