diff --git a/docs/config.md b/docs/config.md index 8ad31e7e58..dc8dd37c67 100644 --- a/docs/config.md +++ b/docs/config.md @@ -4,20 +4,21 @@ ## Configuration file -When a pipeline script is launched, Nextflow looks for configuration files in multiple locations. Since each configuration file can contain conflicting settings, the sources are ranked to determine which settings are applied. Possible configuration sources, in order of priority: +When a pipeline script is launched, Nextflow looks for configuration files in multiple locations. Since each configuration file may contain conflicting settings, they are applied in the following order (from lowest to highest priority): -1. Parameters specified on the command line (`--something value`) -2. Parameters provided using the `-params-file` option -3. Config file specified using the `-c my_config` option -4. The config file named `nextflow.config` in the current directory -5. The config file named `nextflow.config` in the workflow project directory -6. The config file `$HOME/.nextflow/config` -7. Values defined within the pipeline script itself (e.g. `main.nf`) +1. Parameters defined in pipeline scripts (e.g. `main.nf`) +2. Module config files defined alongside module scripts (see {ref}`module-condig` for details) +3. The config file `$HOME/.nextflow/config` +4. The config file `nextflow.config` in the project directory +5. The config file `nextflow.config` in the launch directory +6. Config file specified using the `-c ` option +7. Parameters specified in a params file (`-params-file` option) +8. Parameters specified on the command line (`--something value`) When more than one of these options for specifying configurations are used, they are merged, so that the settings in the first override the same settings appearing in the second, and so on. :::{tip} -If you want to ignore any default configuration files and use only a custom one, use `-C `. +You can use the `-C ` option to use a single configuration file and ignore all other files. ::: ### Config syntax @@ -1306,9 +1307,9 @@ process { } ``` -:::{note} -The `withName` selector applies to a process even when it is included from a module under an alias. For example, `withName: hello` will apply to any process originally defined as `hello`, regardless of whether it is included under an alias. Similarly, it will not apply to any process not originally defined as `hello`, even if it is included under the alias `hello`. -::: +The `withName` selector applies both to processes defined with the same name and processes included with the same alias. For example, `withName: hello` will apply to any process originally defined as `hello`, as well as any process included with the alias `hello`. + +Furthermore, selectors for the alias of an included process take priority over selectors for the original name of the process. For example, given a process defined as `foo` and included as `bar`, the selectors `withName: foo` and `withName: bar` will both be applied to the process, with the second selector taking priority over the first. :::{tip} Label and process names do not need to be enclosed with quotes, provided the name does not include special characters (`-`, `!`, etc) and is not a keyword or a built-in type identifier. When in doubt, you can enclose the label name or process name with single or double quotes. @@ -1347,12 +1348,14 @@ The above configuration snippet sets 2 cpus for the processes annotated with the #### Selector priority -When mixing generic process configuration and selectors the following priority rules are applied (from lower to higher): +Process configuration settings are applied to a process in the following order (from lowest to highest priority): -1. Process generic configuration. -2. Process specific directive defined in the workflow script. -3. `withLabel` selector definition. -4. `withName` selector definition. +1. Process configuration settings (without a selector) +2. Process directives in the process definition +3. `withLabel` selectors matching any of the process labels +4. `withName` selectors matching the process name +5. `withName` selectors matching the process included alias +5. `withName` selectors matching the process fully qualified name For example: @@ -1360,11 +1363,16 @@ For example: process { cpus = 4 withLabel: foo { cpus = 8 } - withName: bar { cpus = 32 } + withName: bar { cpus = 16 } + withName: 'baz:bar' { cpus = 32 } } ``` -Using the above configuration snippet, all workflow processes use 4 cpus if not otherwise specified in the workflow script. Moreover processes annotated with the `foo` label use 8 cpus. Finally the process named `bar` uses 32 cpus. +With the above configuration: +- All processes will use 4 cpus (unless otherwise specified in their process definition). +- Processes annotated with the `foo` label will use 8 cpus. +- Any process named `bar` (or imported as `bar`) will use 16 cpus. +- Any process named `bar` (or imported as `bar`) invoked by a workflow named `baz` with use 32 cpus. (config-report)= diff --git a/docs/module.md b/docs/module.md index 59fc886887..03577919fa 100644 --- a/docs/module.md +++ b/docs/module.md @@ -174,6 +174,56 @@ The above snippet prints: Ciao world! ``` +(module-config)= + +## Module configuration + +:::{versionadded} 24.01.0-edge +::: + +Modules can define a `nextflow.config` file in the module directory to apply process configuration to processes that are invoked in the module. Unlike a regular Nextflow configuration file, the module config only supports the {ref}`process scope `. + +Here is an example module with a module config: + +``` + +|── main.nf +└── nextflow.config +``` + +```groovy +// main.nf +process BAR { +} + +workflow FOO { + BAR() +} +``` + +```groovy +// nextflow.config +process { + publishDir = params.outdir + + withName:'FOO:BAR' { + ext.args = '--n-iters 1000' + publishDir = "${params.outdir}/foo_bar" + } +} +``` + +In the above example, the selector `FOO:BAR` matches the process `BAR` invoked by workflow `FOO`. The selector `BAR` would have also worked in this case, even if `BAR` is used elsewhere in the pipeline, because the module config is only applied to this module. This assurance is the advantage of defining process config in module config files instead of the global pipeline config. + +Process configuration is applied to a process in the following order (from lowest to highest priority): + +1. Process {ref}`directives ` +2. Module config files +3. Pipeline config files +4. Command line options (e.g. `-process.*`) + +Similarly, if a "caller" module invokes a process in a "callee" module, the "caller" module config will take priority over the "callee" module config. In this way, a module can define a "default" configuration that can be overridden at higher and higher levels, where a process might be called in different contexts that require different config settings. + (module-templates)= ## Module templates diff --git a/modules/nextflow/src/main/groovy/nextflow/script/ExecutionStack.groovy b/modules/nextflow/src/main/groovy/nextflow/script/ExecutionStack.groovy index 5c2e62f8b4..df2815a94f 100644 --- a/modules/nextflow/src/main/groovy/nextflow/script/ExecutionStack.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/script/ExecutionStack.groovy @@ -70,6 +70,14 @@ class ExecutionStack { ctx instanceof WorkflowDef ? ctx : null } + static List workflows() { + final result = [] as List + for( def entry : stack ) + if( entry instanceof WorkflowDef ) + result << entry + return result + } + static void push(ExecutionContext script) { stack.push(script) } diff --git a/modules/nextflow/src/main/groovy/nextflow/script/IncludeDef.groovy b/modules/nextflow/src/main/groovy/nextflow/script/IncludeDef.groovy index 08e6e5566e..f1db686d1a 100644 --- a/modules/nextflow/src/main/groovy/nextflow/script/IncludeDef.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/script/IncludeDef.groovy @@ -16,10 +16,6 @@ package nextflow.script -import nextflow.exception.ScriptCompilationException -import nextflow.plugin.extension.PluginExtensionProvider -import nextflow.plugin.Plugins - import java.nio.file.NoSuchFileException import java.nio.file.Path @@ -31,7 +27,11 @@ import groovy.transform.PackageScope import groovy.util.logging.Slf4j import nextflow.NF import nextflow.Session +import nextflow.config.ConfigParser import nextflow.exception.IllegalModulePath +import nextflow.exception.ScriptCompilationException +import nextflow.plugin.extension.PluginExtensionProvider +import nextflow.plugin.Plugins /** * Implements a script inclusion * @@ -136,12 +136,41 @@ class IncludeDef { static BaseScript loadModule0(Path path, Map params, Session session) { final binding = new ScriptBinding() .setParams(params) - // the execution of a library file has as side effect the registration of declared processes - new ScriptParser(session) + // executing a module script also registers any component definitions + final script = new ScriptParser(session) .setModule(true) .setBinding(binding) .runScript(path) .getScript() + + // load module config if it exists + final configPath = path.parent.resolve('nextflow.config') + if( configPath.exists() ) { + final config = new ConfigParser() + .setParams(params) + .parse(configPath) + .toMap() + + // check for unsupported config scopes + def unsupportedScopes = new ArrayList(config.keySet()) + unsupportedScopes.remove('params') + unsupportedScopes.remove('process') + if( unsupportedScopes.size() > 0 ) + log.warn "Module config only supports the process scope, other scopes will be ignored: ${unsupportedScopes.join(',')} (in module config for: $path)" + + // remove any params inserted by the config parser + if( !config.process ) + config.process = [:] + + final processConfig = config.process as Map + for( def value : processConfig.values() ) + if( value instanceof Map ) + value.remove('params') + + ScriptMeta.get(script).setConfig(config) + } + + return script } @PackageScope diff --git a/modules/nextflow/src/main/groovy/nextflow/script/ProcessDef.groovy b/modules/nextflow/src/main/groovy/nextflow/script/ProcessDef.groovy index 3c54f0e426..ad184c5d78 100644 --- a/modules/nextflow/src/main/groovy/nextflow/script/ProcessDef.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/script/ProcessDef.groovy @@ -113,7 +113,23 @@ class ProcessDef extends BindableDef implements IterableDef, ChainableDef { throw new ScriptRuntimeException("Missing script in the specified process block -- make sure it terminates with the script string to be executed") // apply config settings to the process - processConfig.applyConfig((Map)session.config.process, baseName, simpleName, processName) + List configs = [] + + // -- process module config + configs << ScriptMeta.get(owner).getConfig() + + // -- workflow module configs + for( def workflow : ExecutionStack.workflows() ) + configs << ScriptMeta.get(workflow.getOwner()).getConfig() + + // -- session config + configs << (Map)session.config + + for( def config : configs ) { + if( !config || !config.process ) + continue + processConfig.applyConfig((Map)config.process, baseName, simpleName, processName) + } } @Override diff --git a/modules/nextflow/src/main/groovy/nextflow/script/ScriptMeta.groovy b/modules/nextflow/src/main/groovy/nextflow/script/ScriptMeta.groovy index 17a42f6f91..83ecb11414 100644 --- a/modules/nextflow/src/main/groovy/nextflow/script/ScriptMeta.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/script/ScriptMeta.groovy @@ -97,6 +97,9 @@ class ScriptMeta { /** The module components included in the script */ private Map imports = new HashMap<>(10) + /** The module config associated with this script */ + private Map config = new HashMap<>() + /** Whenever it's a module script or the main script */ private boolean module @@ -110,6 +113,8 @@ class ScriptMeta { boolean isModule() { module } + Map getConfig() { config } + ScriptMeta(BaseScript script) { this.clazz = script.class for( def entry : definedFunctions0(script) ) { @@ -130,6 +135,11 @@ class ScriptMeta { this.module = val } + @PackageScope + void setConfig(Map config) { + this.config = config + } + private void incFunctionCount(String name) { final count = functionsCount.getOrDefault(name, 0) functionsCount.put(name, count+1) @@ -296,11 +306,11 @@ class ScriptMeta { void addModule(ScriptMeta script, String name, String alias) { assert script assert name - // include a specific - def item = script.getComponent(name) - if( !item ) + // include a specific component + def component = script.getComponent(name) + if( !component ) throw new MissingModuleComponentException(script, name) - addModule0(item, alias) + addModule0(component, alias) } protected void addModule0(ComponentDef component, String alias=null) { diff --git a/modules/nextflow/src/test/groovy/nextflow/script/ProcessDefTest.groovy b/modules/nextflow/src/test/groovy/nextflow/script/ProcessDefTest.groovy index 6b8fc4805a..867696ec98 100644 --- a/modules/nextflow/src/test/groovy/nextflow/script/ProcessDefTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/script/ProcessDefTest.groovy @@ -1,5 +1,6 @@ package nextflow.script +import nextflow.NF import nextflow.Session import spock.lang.Specification /** @@ -8,6 +9,10 @@ import spock.lang.Specification */ class ProcessDefTest extends Specification { + def setupSpec() { + NF.init() + } + def 'should clone a process with a new name '() { given: @@ -39,6 +44,7 @@ class ProcessDefTest extends Specification { def 'should apply process config' () { given: def OWNER = Mock(BaseScript) + def META = ScriptMeta.register(OWNER) def CONFIG = [ process:[ cpus:2, memory: '1GB',