docs: add desc field #1951

efiop · 2020-11-17T01:01:06Z

You may disregard these recommendations if you used the Edit on GitHub button from dvc.org to improve a doc in place.

❗ Please read the guidelines in the Contributing to the Documentation list if you make any substantial changes to the documentation or JS engine.

🐛 Please make sure to mention Fix #issue (if applicable) in the description of the PR. This causes GitHub to close it automatically when the PR is merged.

Please choose to allow us to edit your branch when creating the PR.

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

shcheklein · 2020-11-17T01:04:58Z

content/docs/command-reference/add.md

@@ -148,6 +148,8 @@ directory symlinks.
  pattern as specified by `target`. Shell-style wildcards are supported: `*`,
  `?`, `[seq]`, `[!seq]`, and `**`.

+- `--desc` - arbitrary string description for generated `.dvc` file.


for generated .dvc file -> for data artifact ... or something similar, it does not describe .dvc file.

I wonder if we should provide a short-cut (-d)? also, would -m be a better option? to remind git commit -m ?

it should be also --desc <description> I think?

for generated .dvc file -> for data artifact ... or something similar, it does not describe .dvc file.

Depends on how to look at it. It is not really for specific out either, from the internal perspective. In 2.0 we'll likely simplify the internal structure for dvc add-generated file, and then your suggestion will be 100% correct.

I wonder if we should provide a short-cut (-d)? also, would -m be a better option? to remind git commit -m ?

Both -d and -m are taken. Don't want to take up short options for it just yet. Also analogy with git commit -m is not that intuitive for this, IMO.

it should be also --desc I think?

Added. Thanks!

Depends on how to look at it.

it's important to understand what is the purpose of this from the end user perspective and describe it in a way that would resonates with that in the first place. To my mind it's about providing descriptions to data unless I'm missing something. Now I read as I describe the file - why would I even want to do that?

Don't want to take up short options for it just yet.

yep, fine 👍

To my mind it's about providing descriptions to data

Agree. There's already the meta field for general user info about the .dvc file itself.

would -m be a better option? to remind git commit -m

Good idea but I also think it's not intuitive: It could be confusing since this is add, not commit.

Both -d and -m are taken.

~~Not for add.~~ Ah but they are fun run which also gets this new option, I see. Never mind.

My suggestion (for now):

Suggested change

- `--desc` - arbitrary string description for generated `.dvc` file.

- `--desc <text>` - arbitrary description for the data `targets`. This doesn't

affect any DVC operations.

Please confirm metavar per iterative/dvc#4901 (review))

BTW does the same desc get applied to multiple targets (if several are provided to dvc add)? Thanks

Thanks, updated the doc and the help in the original PR.

BTW does the same desc get applied to multiple targets (if several are provided to dvc add)? Thanks

Yes, it does.

shcheklein

A few comments.

How about desc for imports/import-url?

Also, for outputs in the pipeline?

efiop · 2020-11-17T01:23:07Z

How about desc for imports/import-url?

Will add for symmetry. 👍

Also, for outputs in the pipeline?

I don't think it was part of the plan. But let me clarify...

content/docs/command-reference/run.md

jorgeorpinel · 2020-11-17T21:36:43Z

content/docs/user-guide/dvc-files-and-directories.md

 - `meta` (optional): Arbitrary metadata can be added manually with this field.
  Any YAML contents is supported. `meta` contents are ignored by DVC, but they
  can be meaningful for user processes that read `.dvc` files.
+- `desc` (optional): Arbitrary string description for this `.dvc` file.


Wait, so this field is for the whole .dvc file? So what's the difference with meta? I thought this would be per output (so, in the next bullet list).

Yes, both. meta is not used by us at all, it is a completely arbitrary metadata. desc is required to be a string/text and will be used by Viewer and maybe even dvc at some point.

I see, thanks for the explanation @efiop. Would it be better to keep it hidden (undocumented) for now? I see no significative difference between meta and desc from the DVC user perspective. Possible source of confusion

p.s. I'd even suggest the Viewed uses a special format in meta values instead cc @fabiosantoscode maybe? For consideration

Hey, I didn't know about this but it sounds like a good idea. Regarding screen real estate we can only really display it when you hover or click a file version, where we are going to show some more long form information like the full hash, delta and change dates.

@shcheklein what do you think?

We've talked about meta before. meta is for arbitrary data that is not for us at all. Not even for viewer. desc is something that we recognize and something that viewer will be using very soon. We will be using it in some dvc commands in the future as well.

desc is something that we recognize

But https://dvc.org/doc/user-guide/dvc-files-and-directories reads "This doesn't affect any DVC operations." so it's the Viewer that recognizes it (and to my mind it could do the same with a special format in meta but up to you guys). I suspect at some point the Viewer may also display meta for the user, so the distinction becomes blurry again.

My main objection is to have both these options documented since they look like the same from the user perspective (and desc is a little inconsistent as mentioned in #1951 (review) below).

@jorgeorpinel It doesn't affect any dvc operations right now, but we still recognize it as a stage/output description string. Meta is also recognized by us but as an arbitrary object. Viewer will likely show it, but dvc - likely not. Still, those are separate things. We've considered using meta for it and decided not to.

@shcheklein Not at all! I didn't know about this and it wasn't present in the designs

but we still recognize it as a stage/output description string

What does it mean to "recognize it"? Does DVC actually read/validate/process it in some way it doesn't do with meta? I do realize add/run have the --desc flag that writes it (except for stage outputs) so if that's what we mean then OK.

We've considered using meta for it and decided not to.

Sure, I'm not insisting. I'm just detecting possible inconsistencies and confusions. Maybe I'll just extract this to an issue so we can officially burry it as a nice-to-have 😋

Thanks for all the answers!

What does it mean to "recognize it"? Does DVC actually read/validate/process it in some way it doesn't do with meta? I do realize add/run have the --desc flag that writes it (except for stage outputs) so if that's what we mean then OK.

Yes, it does read/validate it as str. Meta is arbitrary, we don't validate it ever.

Per iterative/dvc#4901

efiop · 2020-11-18T19:50:27Z

Combined your suggestions in both PRs @jorgeorpinel . Thanks!

jorgeorpinel · 2020-11-30T08:14:48Z

content/docs/user-guide/dvc-files-and-directories.md

 - `cache`: Whether or not this file or directory is <abbr>cached</abbr> (`true`
  by default, if not present). See the `--no-commit` option of `dvc add`.
+- `desc`: User description for this output. This doesn't affect any DVC
+  operations.


Also, is it correct that for .dvc files desc is a subfield inside outs? Seems odd since it's a top-level field in dvc.yaml files.

It is both.

Sorry, didn't get that. I guess you're saying that yes, this is correct: desc is at the outs level in .dvc files, but at the top-level for dvc.yaml. This "inconsistency" seems unintuitive to me.

desc can be for stage or for specific output

@jorgeorpinel Let's not focus on this too much, this is minor for docs.

desc may be minor but 1. that doesn't mean its OK to have an error in docs if we see it (not the case here but I wasn't sure until now) and 2. this is making me realize the way we present outputs vs dvc.yaml in https://dvc.org/doc/user-guide/dvc-files-and-directories is confusing because outputs are only expanded on in the .dvc file section... which is kind of major. I'll address that elsewhere.

jorgeorpinel · 2020-12-04T04:07:23Z

One last comment from me here: The dvc.yaml schema doesn't seem to have the field yet cc @skshetry thanks

See https://github.com/iterative/dvcyaml-schema/blob/master/schema.json

skshetry · 2020-12-04T07:13:55Z

Thanks @jorgeorpinel for reminding. I'm planning to add them during 2.0 releases. I'm hoping if I can generate it automatically.

shcheklein temporarily deployed to dvc-landing-fix-dvc-490-bznorm November 17, 2020 01:01 Inactive

efiop mentioned this pull request Nov 17, 2020

dvcfile: add desc field for outs/stages iterative/dvc#4901

Merged

2 tasks

shcheklein reviewed Nov 17, 2020

View reviewed changes

shcheklein requested changes Nov 17, 2020

View reviewed changes

efiop force-pushed the fix-dvc-4901 branch from c45eabf to a2d6127 Compare November 17, 2020 01:13

shcheklein temporarily deployed to dvc-landing-fix-dvc-490-bznorm November 17, 2020 01:14 Inactive

efiop changed the title ~~docs: add desc field~~ [WIP] docs: add desc field Nov 17, 2020

jorgeorpinel reviewed Nov 17, 2020

View reviewed changes

content/docs/command-reference/run.md Outdated Show resolved Hide resolved

jorgeorpinel reviewed Nov 17, 2020

View reviewed changes

efiop force-pushed the fix-dvc-4901 branch from a2d6127 to e49faad Compare November 18, 2020 19:03

shcheklein temporarily deployed to dvc-landing-fix-dvc-490-bznorm November 18, 2020 19:03 Inactive

efiop force-pushed the fix-dvc-4901 branch from e49faad to 1699ddd Compare November 18, 2020 19:07

shcheklein temporarily deployed to dvc-landing-fix-dvc-490-bznorm November 18, 2020 19:08 Inactive

docs: add desc field for stages and outs

7d7bc2e

Per iterative/dvc#4901

efiop force-pushed the fix-dvc-4901 branch from 1699ddd to 7d7bc2e Compare November 18, 2020 19:17

shcheklein temporarily deployed to dvc-landing-fix-dvc-490-bznorm November 18, 2020 19:17 Inactive

efiop changed the title ~~[WIP] docs: add desc field~~ docs: add desc field Nov 18, 2020

efiop merged commit 4038814 into master Nov 26, 2020

efiop deleted the fix-dvc-4901 branch November 26, 2020 07:03

jorgeorpinel reviewed Nov 30, 2020

View reviewed changes

jorgeorpinel mentioned this pull request Dec 4, 2020

Consistency of desc fields in dvc.yaml and .dvc files iterative/dvc#5025

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: add desc field #1951

docs: add desc field #1951

efiop commented Nov 17, 2020 •

edited

Loading

shcheklein Nov 17, 2020

shcheklein Nov 17, 2020

efiop Nov 17, 2020 •

edited

Loading

shcheklein Nov 17, 2020

jorgeorpinel Nov 17, 2020 •

edited

Loading

jorgeorpinel Nov 17, 2020 •

edited

Loading

efiop Nov 18, 2020 •

edited

Loading

shcheklein left a comment

efiop commented Nov 17, 2020

jorgeorpinel Nov 17, 2020

efiop Nov 18, 2020

jorgeorpinel Nov 30, 2020 •

edited

Loading

fabiosantoscode Nov 30, 2020 •

edited

Loading

efiop Nov 30, 2020

jorgeorpinel Dec 1, 2020 •

edited

Loading

efiop Dec 1, 2020

fabiosantoscode Dec 2, 2020 •

edited

Loading

jorgeorpinel Dec 4, 2020

efiop Dec 10, 2020

efiop commented Nov 18, 2020

jorgeorpinel Nov 30, 2020 •

edited

Loading

efiop Nov 30, 2020

jorgeorpinel Dec 1, 2020

efiop Dec 1, 2020

efiop Dec 1, 2020

jorgeorpinel Dec 3, 2020 •

edited

Loading

jorgeorpinel commented Dec 4, 2020

skshetry commented Dec 4, 2020 •

edited

Loading

	- `--desc` - arbitrary string description for generated `.dvc` file.
	- `--desc <text>` - arbitrary description for the data `targets`. This doesn't
	affect any DVC operations.

docs: add desc field #1951

docs: add desc field #1951

Conversation

efiop commented Nov 17, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

efiop Nov 17, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorgeorpinel Nov 17, 2020 • edited Loading

Choose a reason for hiding this comment

jorgeorpinel Nov 17, 2020 • edited Loading

Choose a reason for hiding this comment

efiop Nov 18, 2020 • edited Loading

Choose a reason for hiding this comment

shcheklein left a comment

Choose a reason for hiding this comment

efiop commented Nov 17, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorgeorpinel Nov 30, 2020 • edited Loading

Choose a reason for hiding this comment

fabiosantoscode Nov 30, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorgeorpinel Dec 1, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fabiosantoscode Dec 2, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

efiop commented Nov 18, 2020

jorgeorpinel Nov 30, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorgeorpinel Dec 3, 2020 • edited Loading

Choose a reason for hiding this comment

jorgeorpinel commented Dec 4, 2020

skshetry commented Dec 4, 2020 • edited Loading

efiop commented Nov 17, 2020 •

edited

Loading

efiop Nov 17, 2020 •

edited

Loading

jorgeorpinel Nov 17, 2020 •

edited

Loading

jorgeorpinel Nov 17, 2020 •

edited

Loading

efiop Nov 18, 2020 •

edited

Loading

jorgeorpinel Nov 30, 2020 •

edited

Loading

fabiosantoscode Nov 30, 2020 •

edited

Loading

jorgeorpinel Dec 1, 2020 •

edited

Loading

fabiosantoscode Dec 2, 2020 •

edited

Loading

jorgeorpinel Nov 30, 2020 •

edited

Loading

jorgeorpinel Dec 3, 2020 •

edited

Loading

skshetry commented Dec 4, 2020 •

edited

Loading