Offload publishDir data copy to job script

[pditommaso]

Process data publishing defined by using the publishDir directive is handled by the main nextflow application. This approach does not scale for large workloads because all taks outputs will need to be copied by the same NF instance.

A possible improvement consists off-loading task outputs publishing to the job script wrapper.

This would only be possible for publishDir directive not specifying the saveAs attribute because in this case the closure needs to be evaluated by the NF runtime, therefore it cannot be resolved in the Bash wrapper script.

Also it could only be applied when the publishing target directory is on the same file system as the workflow work directory or the bucket directory.

Suggested implementation:

1. determine if the publishDir is eligible to be managed at task level;
2. if yes, add the relevant information here;
3. generate the appropriate copy command, along the same line of outputFiles;
4. otherwise fallback on the default behaviour.

Related: #856.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.