Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Workflow output definition (second preview) #5185

Merged
merged 21 commits into from
Oct 24, 2024
Merged
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
5978db5
Finalize workflow output definition
bentsherman Jul 25, 2024
b3d5f0d
Restore preview flag
bentsherman Sep 23, 2024
9eb960a
Add warning for unused targets in output block
bentsherman Sep 23, 2024
7a7a2db
Fix bug with enabled and index file
bentsherman Sep 23, 2024
1be3ca7
Add support for json index file
bentsherman Sep 23, 2024
2909469
Update dynamic path syntax
bentsherman Sep 23, 2024
89beee4
Improve syntax checking of output block
bentsherman Sep 23, 2024
8e255b6
Update docs
bentsherman Sep 23, 2024
9418992
Update tests
bentsherman Sep 23, 2024
99341a7
update docs [ci skip]
bentsherman Sep 23, 2024
04a1e98
Add onWorkflowPublish event
bentsherman Sep 23, 2024
cc4927f
Merge branch 'master' into workflow-output-definition-final
bentsherman Sep 25, 2024
05fc184
Merge branch 'master' into workflow-output-definition-final
bentsherman Oct 2, 2024
5569eec
Normalize publish paths
bentsherman Oct 7, 2024
3a70ba8
Merge branch 'master' into workflow-output-definition-final [e2e stage]
pditommaso Oct 17, 2024
a2a690b
[e2e prod] Merge branch 'master' into workflow-output-definition-final
pditommaso Oct 18, 2024
a27c6f9
Update docs/reference/config.md [ci skip]
pditommaso Oct 21, 2024
b31e1c6
Make e2e test comply with strict syntax
bentsherman Oct 21, 2024
56f6afa
Add deprecation warning for `nextflow.publish` config scope
bentsherman Oct 22, 2024
7912bd4
Minor change for sake of readability [ci fast]
pditommaso Oct 22, 2024
7d5d69b
Merge branch 'master' into workflow-output-definition-final
bentsherman Oct 23, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/reference/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -1012,6 +1012,11 @@ The `run` command is used to execute a local pipeline script or remote pipeline
`-offline`
: Do not check for remote project updates.

`-o, -output-dir` (`results`)
: :::{versionadded} 24.10.0
:::
: Directory where workflow outputs are stored.

`-params-file`
: Load script parameters from a JSON/YAML file.

Expand Down
81 changes: 81 additions & 0 deletions docs/reference/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,11 @@ This page lists all of the available settings in the {ref}`Nextflow configuratio
`dumpHashes`
: If `true`, dump task hash keys in the log file, for debugging purposes. Equivalent to the `-dump-hashes` option of the `run` command.

`outputDir`
: :::{versionadded} 24.10.0
:::
: Defines the pipeline output directory. Equivalent to the `-output-dir` option of the `run` command.

`resume`
: If `true`, enable the use of previously cached task executions. Equivalent to the `-resume` option of the `run` command.

Expand Down Expand Up @@ -1600,6 +1605,9 @@ The following settings are available:

## `workflow`

:::{versionadded} 24.10.0
:::

The `workflow` scope provides workflow execution options.

`workflow.failOnIgnore`
Expand All @@ -1612,3 +1620,76 @@ The `workflow` scope provides workflow execution options.

`workflow.onError`
: Specify a closure that will be invoked if a workflow run is terminated. See {ref}`workflow-handlers` for more information.

`workflow.output.contentType`
: *Currently only supported for S3.*
: Specify the media type a.k.a. [MIME type](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_Types) of published files (default: `false`). Can be a string (e.g. `'text/html'`), or `true` to infer the content type from the file extension.
pditommaso marked this conversation as resolved.
Show resolved Hide resolved

`workflow.output.enabled`
: Enable or disable publishing (default: `true`).

`workflow.output.ignoreErrors`
: When `true`, the workflow will not fail if a file can't be published for some reason (default: `false`).

`workflow.output.mode`
: The file publishing method (default: `'symlink'`). The following options are available:

`'copy'`
: Copy each file into the output directory.

`'copyNoFollow'`
: Copy each file into the output directory without following symlinks, i.e. only the link is copied.

`'link'`
: Create a hard link in the output directory for each file.

`'move'`
: Move each file into the output directory.
: Should only be used for files which are not used by downstream processes in the workflow.

`'rellink'`
: Create a relative symbolic link in the output directory for each file.

`'symlink'`
: Create an absolute symbolic link in the output directory for each output file.

`workflow.output.overwrite`
: When `true` any existing file in the specified folder will be overwritten (default: `'standard'`). The following options are available:

`false`
: Never overwrite existing files.

`true`
: Always overwrite existing files.

`'deep'`
: Overwrite existing files when the file content is different.

`'lenient'`
: Overwrite existing files when the file size is different.

`'standard'`
: Overwrite existing files when the file size or last modified timestamp is different.

`workflow.output.retryPolicy.delay`
: Delay when retrying a failed publish operation (default: `350ms`).

`workflow.output.retryPolicy.jitter`
: Jitter value when retrying a failed publish operation (default: `0.25`).

`workflow.output.retryPolicy.maxAttempt`
: Max attempts when retrying a failed publish operation (default: `5`).

`workflow.output.retryPolicy.maxDelay`
: Max delay when retrying a failed publish operation (default: `90s`).

`workflow.output.storageClass`
: *Currently only supported for S3.*
: Specify the storage class for published files.

`workflow.output.tags`
: *Currently only supported for S3.*
: Specify arbitrary tags for published files. For example:
```groovy
tags FOO: 'hello', BAR: 'world'
```
5 changes: 5 additions & 0 deletions docs/reference/stdlib.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,11 @@ The following constants are globally available in a Nextflow script:
`workflow.manifest`
: Entries of the workflow manifest.

`workflow.outputDir`
: :::{versionadded} 24.10.0
:::
: Workflow output directory.

`workflow.preview`
: :::{versionadded} 24.04.0
:::
Expand Down
Loading
Loading