Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document multiline string cmd #2721

Closed
wants to merge 2 commits into from
Closed

Document multiline string cmd #2721

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
22f3f0c
Update pipelines-files.md
skshetry Aug 16, 2021
ce8bf0f
Update repro.md
skshetry Aug 16, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions content/docs/command-reference/repro.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,10 @@ graph (a [DAG](https://en.wikipedia.org/wiki/Directed_acyclic_graph)) implicitly
defined by the stages listed in `dvc.yaml`. The commands defined in these stages
are then executed in the correct order.

For stages with multiple commands (having a list in the `cmd` field), commands
are run one after the other in the order they are defined. The failure of any
command will halt the remaining stage execution, and raises an error.
For stages with multiple commands (having a list or a multiline string in the
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a standard for "multiline strings" in YAML (starts with | or > or any string with newlines)?

Copy link
Member Author

@skshetry skshetry Aug 17, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's the one with |. The > is quite opposite, changes a newline into a space, i.e. multiline to a single line string.

stages:
  stage1:
    cmd: |
       wget https://code.dvc.org/get-started/code.zip
       unzip code.zip
       rm -f code.zip

See http://yaml-multiline.info. I haven't found a good place to accomodate this as an example, we don't have examples on dvc.yaml guide.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @skshetry ! That's exactly what I was looking for - that link. We can make the multiline string a link? (and probably the list also). But even in that case I would include explicitly that we mean | style blocks, right here in the text.

Copy link
Contributor

@iesahin iesahin Aug 17, 2021 •
edited by jorgeorpinel
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can also document this in https://dvc.org/doc/user-guide/project-structure/pipelines-files#stages

Copy link
Contributor

@jorgeorpinel jorgeorpinel Aug 18, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

haven't found a good place to accomodate this as an example, we don't have examples on dvc.yaml guide

There are many snippet examples all over. https://dvc.org/doc/user-guide/project-structure/pipelines-files#templating has a > one for example. We should change that one if we recommend | better:

image

p.s. Agree on linking and adding the info to the pipelines file guide.

Copy link
Member Author

@skshetry skshetry Aug 18, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do have examples, but not for the possibilities of every keyword, just for the high-level features. Also, we support list of commands, which does not have examples. Would have been great if it were collapsible, I think.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would have been great if it were collapsible

Good idea! It can still be great, feel free to contribute

`cmd` field), commands are run one after the other in the order they are
defined. The failure of any command will halt the remaining stage execution, and
raises an error.

> Pipeline stages are defined in `dvc.yaml` (either manually or by using
> `dvc run`) while initial data dependencies can be registered with `dvc add`.
Expand Down
2 changes: 1 addition & 1 deletion content/docs/user-guide/project-structure/pipelines-files.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -350,7 +350,7 @@ These are the fields that are accepted in each stage:

| Field | Description |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `cmd` | (Required) One or more commands executed by the stage (may contain either a single value or a list). Commands are executed sequentially until all are finished or until one of them fails (see `dvc repro`). |
| `cmd` | (Required) One or more commands executed by the stage. Can be a single-line string, a multi-line string or a list. Commands are executed sequentially until all are finished or until one of them fails (see `dvc repro`). |
| `wdir` | Working directory for the stage command to run in (relative to the file's location). Any paths in other fields are also based on this. It defaults to `.` (the file's location). |
| `deps` | List of <abbr>dependency</abbr> paths of this stage (relative to `wdir`). |
| `outs` | List of <abbr>output</abbr> paths of this stage (relative to `wdir`). These can contain certain optional [subfields](#output-subfields). |
Expand Down