Support no-command meta stages

[mdekstrand]

I would find it very useful to be able to define a DVC stage that has inputs but no command or outputs. The use case is to create meta-stages that serve only to depend on other stages, either to provide convenient ways to run part of the pipeline, or in the top-level dvc.yaml to make a bare dvc repro reproduce outputs defined in subdirectories.

I see two options to enable this that shouldn't cause conflict with existing pipelines:

• Support a stage without a cmd entry; such a stage checks its inputs and updates its md5 but does not run any command or produce outputs.

• If implicit behavior without cmd is undesirable, require cmd, but allow it to be YAML null:

  cmd: null
  

  This will make it explicit in the dvc.yaml file that this stage does not have a command.

I currently have two workarounds for this lack:

• If my pipeline will only be used on *nix systems, use cmd: 'true'.
• Use another no-op command, like cmd: python -V; it works, but it's a bit weird and doesn't clearly communicate the intent of the stage.

[karajan1001]

either to provide convenient ways to run part of the pipeline, or in the top-level dvc.yaml to make a bare dvc repro reproduce outputs defined in subdirectories.
Excuse me, it looks like this kind of no-command meta stage is used for grouping stages? If so, could the--glob flag in dvc repro meet your requirements, for example, dvc repro --glob train-*?

[mdekstrand]

There are two use cases I have:

• Grouping stages. Globs in repro can help with this, but are less convenient, particularly if the stages to group are split across multiple subdirectories (each with its own dvc.yaml), and/or may change from time to time. Defined meta-stages allows you to document things like "run dvc repro dataprep to ensure data preparation is up to date", and just keep the meta-stage dataprep definition up to date to ensure the instructions keep working.
• Top-level stages. When running dvc repro without arguments in a project root, it reproduces all stages in the root dvc.yaml; stages defined in dvc.yaml files in subdirectories don't get reproduced unless something in the root dvc.yaml depends on one of their outputs. Putting a meta-stage in the top-level dvc.yaml would provide an easy way to include those outputs in a no-argument dvc repro run.

The second of these is currently my more common use case, but I have felt the need for the first from time to time.

[karajan1001]

@dberenbaum What do you think of this? Although In my previous work I just use for each stages on to level.

[dberenbaum]

Interesting idea. Your use cases make sense, but I think it's worth considering other ways to achieve managing complex stage groupings.

* Grouping stages. Globs in `repro` can help with this, but are less convenient, particularly if the stages to group are split across multiple subdirectories (each with its own `dvc.yaml`), and/or may change from time to time. Defined meta-stages allows you to document things like "run `dvc repro dataprep` to ensure data preparation is up to date", and just keep the meta-stage `dataprep` definition up to date to ensure the instructions keep working.

I could see it being useful to have a way to configure a group of stages to run, but:

• As you mention, you can already do this with no-op stages, so it doesn't seem high priority.
• As you also mention, no-op stages don't clearly communicate the intent. I don't think the proposed solution seems much clearer in communicating this intent. In either case, you could add a comment to the meta-stage to describe the intent.

* Top-level stages. When running `dvc repro` without arguments in a project root, it reproduces all stages in the root `dvc.yaml`; stages defined in `dvc.yaml` files in subdirectories don't get reproduced unless something in the root `dvc.yaml` depends on one of their outputs. Putting a meta-stage in the top-level `dvc.yaml` would provide an easy way to include those outputs in a no-argument `dvc repro` run.

Can this be achieved with dvc repro --all-pipelines?

[mdekstrand]

I could see it being useful to have a way to configure a group of stages to run, but:

• As you mention, you can already do this with no-op stages, so it doesn't seem high priority.
• As you also mention, no-op stages don't clearly communicate the intent. I don't think the proposed solution seems much clearer in communicating this intent. In either case, you could add a comment to the meta-stage to describe the intent.

That's fair. And the most self-documenting would be to create some kind of 'finalize' script that reports something meaningful but brief, and use that as a command.

Can this be achieved with dvc repro --all-pipelines?

Yes, it can (provided that all stages are wanted, and not just some of them). Somehow I missed that as a prospective solution.

If you want to close this, it's fine - can make things work with these solutions, and it's only affecting the potential ergonomics of my pipelines, not my ability to specify the pipelines I need. Just thought I'd suggest it to raise discussion about the use cases and in case others might find it useful as well :).

[dberenbaum]

Thanks for the explanation and follow-up, @mdekstrand! I'm going to convert to a discussion and see if others follow up with additional needs or ideas here.