Allow nested sub-workflows #4477
Conversation
|
Here's a working example on this branch. Main source tree: Main workflow definition: $ cat flow.cylc
[scheduling]
cycling mode = integer
initial cycle point = 1
[[graph]]
P1 = "foo => sub => bar"
[runtime]
[[foo, bar]]
script = true
[[sub]]
script = """
SUBWF_SRCE="${CYLC_WORKFLOW_RUN_DIR}/sub"
SUBWF_NAME="${CYLC_WORKFLOW_ID}-sub"
SUBWF_RUNN=$CYLC_TASK_CYCLE_POINT
cylc install -C $SUBWF_SRCE --flow-name=$SUBWF_NAME --run-name=$SUBWF_RUNN
cylc play --no-detach $SUBWF_NAME/$CYLC_TASK_CYCLE_POINT
"""Sub-workflow definition: $ cat sub/flow.cylc
[scheduling]
[[graph]]
R1 = "sub1 => sub2 => sub3"
[runtime]
[[sub1, sub2, sub3]]
script = sleep 3This example runs correctly, and the resulting run directory tree looks like this: Note the sub-workflow run-directory should be assocatiated with a particular installed instance of the main workflow (
If this PR flies, we should document that this is how to run sub-workflows at Cylc 8, and what the limitations are. |
hjoliver
force-pushed
the
allow-sub-workflows
branch
from
58e723c
to
101d388
Compare
@hjoliver unfortunately this breaks the There are two ways around this, the first is to go with this structure (outlined on the linked issue): Which is nice and tidy, however, you have to remember to add the The second option is to install the sub workflows under different names which is fairly simple: Here's an example of the latter I tested on this branch: script = """
cylc install \
-C "$CYLC_WORKFLOW_RUN_DIR/sub" \
--flow-name "$CYLC_WORKFLOW_NAME-sub" \
--run-name "$CYLC_TASK_CYCLE_POINT"
cylc play \
"$CYLC_WORKFLOW_NAME-sub/$CYLC_TASK_CYCLE_POINT" \
--no-detach
""" |
Can you be specific about what breaks exactly?
(That's correct, because the source for |
|
Your suggested alternatives don't work as-is, because the sub-workflows of multiple installed main workflows clash. Although they can be tweaked to fix that, of course. (And they still require the change on this branch - i.e. allow source files in the cylc-run dir.) |
|
If we can settle on a good run-dir structure for managing sub-workflows we could provide basic built-in support for it (for install and play). Or at least document it. Even though I haven't found anything that's actually broken in my original suggested structure, which tries to hide the sub-workflow run dir inside the main one, I concede it might be simpler to keep the two separate with no nesting. Another idea: mirror the main workflow run-dir path exactly, but under a new sub-directory of cylc-run. This avoids nested installs, and the result is completely unambiguous. Result, for two main-workflow runs: |
|
Ah, I hadn't spotted you had craftily nested a $ tree -L 2 ~/cylc-run/hillary-sub
.
|-- _cylc-install
| `-- source -> ~/cylc-src/hillary-sub
|-- run1
| |-- flow.cylc
| |-- log
| |-- share
| |-- sub
| `-- work
|-- run1-sub # looks like a run dir managed by its sibling `_cylc-install` but isn't
| |-- 1
| |-- 2
| |-- 3
| `-- _cylc-install # can confirm reinstall for run1-sub will use this one (correctly)!
`-- runN -> run1It's a little confusing and I'm surprised it works, but it does work and work rather nicely, both flows reinstall from the correct source! |
Agreed, so some thoughts... Ideally:
Ideas:1) Put the sub workflows in the run dir, link rather than copy files:The IDs become:
Example management cmds: $ cylc stop idea-one/run1 # stop master workflow
$ cylc stop 'idea-one/run1/*' # stop all sub-workflows (and master?)
$ cylc stop idea-one/run1/sub-a/1 # similar to `cylc kill idea-one/run1 sub-a/1`
$ cylc clean idea-one/run12) Put the sub workflows in the work dir, link rather than copy files:$ tree ~/cylc-run/idea-two
.
|-- _cylc-install
| `-- source -> ~/cylc-src/whatever
`-- run1
|-- flow.cylc
|-- sub-a
| `-- flow.cylc
|-- sub-b
| `-- flow.cylc
`-- work
|-- 20000101T0000Z
| |-- sub-a
| | `-- flow.cylc -> ../../sub-a/flow.cylc
| `-- sub-b
| `-- flow.cylc -> ../../sub-b/flow.cylc
`-- 20010101T0000Z
|-- sub-a
| `-- flow.cylc -> ../../sub-a/flow.cylc
`-- sub-b
`-- flow.cylc -> ../../sub-b/flow.cylc
Housekeeping is now the same for sub-workflows as it is for native tasks which will make configuring The IDs become:
Example management commands: |
Agreed. Install and reinstall work correctly in my example (given that a sub-workflow instance needs to be based on the installed main workflow source) but there's not much point in explicitly exposing that because we don't want to encourage users to edit sub-workflow sources (or anything else) in the main run directory. Note from last meeting we agreed not to let this issue hold up 8.0rc1. If it's much work to implement the above we can start by recommending users take the straightforward manual (but ugly) approach of managing subworkflows as entirely separate workflows, which is how it works at Cylc 7 anyway. |
hjoliver
changed the title
|
Superseded by #4811 |
Allow sub-workflow definitions under the
~/cylc-rundirectory.This simple change address part of the discussion under #4453. It just changes (and rewords) an exception into a warning, so that
cylc installdoes not balk at installing the sub-workflow at runtime, when itsflow.cylcis found under~/cylc-run.Context: a sub-workflow definition is just another source flie for the main workflow. As such, it should be installed with other main workflow source files into the main workflow run directory. Then at run-time, the sub-workflow gets "instantiated" anew for each main workflow cycle point (via another
cylc install) from the installed sub-workflow source.Requirements check-list
CONTRIBUTING.mdand added my name as a Code Contributor.setup.pyandconda-environment.yml.