ref: `exp init` improvements #3071

jorgeorpinel · 2021-12-07T22:10:38Z

Follow-up to #3015

Move before exp run in nav
Copy edits/ Improve explanations
Address ~~[WIP] cmd-ref: document exp init #3015 (review) if still needed~~ ref: exp init improvements #3071 (review).
~~Full Configuration section~~ Linked to https://dvc.org/doc/command-reference/config#exp + config: reorder and summarize sections (in ref.) #3098

Add Examples?
Keep command section or move to Options?
See ref: exp init improvements #3071 (review)
Update DVC core with changes to exp (init) help output decided here?
See ref: exp init improvements #3071 (review) below.
Extract details to new Exps guide? Review this possible structure -- mentioned in guide: Experiment Management (phase 2) #2911

content/docs/command-reference/exp/init.md

shcheklein · 2021-12-08T03:33:17Z

content/docs/command-reference/exp/init.md

+└── src/
+```
+
+> Note that `params.yaml` is the only required file (see `dvc params`).


side note ... for the sake of my education: why is it required?

To parse the parameters and put them into dvc.yaml.

Yeah but even if you don't have params an empty file is still required.

Related to iterative/dvc#6446 (comment)

It's also related to iterative/dvc#6605. The params section of dvc.yaml currently expects keys from a file (see below), not a filename itself, so all the parameters need to be defined before creating the stage. If we want to stop requiring this file at the time of dvc exp init, we need to change params to accept filenames.

stages: train: cmd: python train.py deps: - data - src params: - params.py: - batch_size - data_path - epochs - latent_dim - num_samples

What about generating a dvc.yaml without params if there's no parmas.yaml file? This is a core discussion though! Moved to iterative/dvc#6446 (comment)

Sorry, I misunderstood you, and now I think it's actually more of a docs issue 🤔 . It is possible to use dvc exp init without params. You can either use --explicit and not provide --params, or you can use dvc exp init -i and type n at the params prompt.

The note here is to indicate that if you do use either the default or some other params path, then that path must exist and be parseable to extract the parameters for the stage. So the current note probably needs to be reworded.

UPDATE: Looks like the discussion about empty params.yaml files is in a few core repo tickets now (e.g. iterative/dvc#7138) so I won't keep discussing here (still a bit confused though).

It is possible to use dvc exp init without params...
So the current note probably needs to be reworded.

Clarified in 2273431.

content/docs/command-reference/exp/init.md

jorgeorpinel · 2021-12-10T17:39:13Z

content/docs/command-reference/exp/init.md

 ### The `command` argument

-The `command` argument is optional, if you are using `--interactive` mode. The


BTW a question I have is whether to keep this section or move command under Options like we did for targets in https://dvc.org/doc/command-reference/repro#options. I personally like the section more but I remember we discussed it (cc @shcheklein) this and using Options was picked, so for consistency I'd move this under Options as well.

From #3015 (review)

per #3071 (review)

per #3071 (comment)

per #3071 (review)

mention --explicit avoids a params.yaml file too.

dberenbaum

Looks good! A couple small notes to help clarify for users what the expectations are for the different paths.

content/docs/command-reference/exp/init.md

dberenbaum · 2021-12-15T16:54:25Z

content/docs/command-reference/exp/init.md

@@ -19,43 +18,57 @@ usage: dvc exp init [-h] [-q | -v] [--run] [--interactive] [-f]

 ## Description


Somewhere in the description, it might be useful to explain that dvc exp init by default expects that input data, parameters, and source code paths exist before running an experiment, and that the command is expected to generate models, metrics, and plots.

That's more or a problem for exp run though, @dberenbaum (already linked form the --run option). But should we state that exp run (and repro for that matter) expect that the stage definition and code are good? Hopefully its evident.

That's more or a problem for exp run though

Well that's the point of exp init, right? Better to have users understand what's needed up front than to have them run exp init only to fail on exp run.

Doesn't need to be part of this PR. It could also be handled by some of the suggested changes to the core command rather than the docs.

Better to have users understand what's needed up front than to have them run exp init only to fail on exp run.

I'm not sure. If users need to read that on the cmd ref., is exp init serving as a self-explanatory command? I think those notes should be in the command's output fist (and added to this doc as a result of that product update).

shcheklein · 2021-12-16T02:57:51Z

looks good to me (I didn't check the details though, since Dave've done this already)

Do we want to add an example? (may be even animation?)

per #3071 (comment)

jorgeorpinel · 2021-12-21T10:56:17Z

Do we want to add an example?

Yep @shcheklein it's a check box I didn't get to yet ⌛ (could be a separate PR)

may be even animation?

Good idea in general to consider gifs but in this case I think a static console sample with the entire -i output would be better so you can read them calmly. Animations would be good for samples that focus on progress, mutation, etc. Let's keep it in mind.

rel. #3071 (comment)

jorgeorpinel · 2021-12-21T11:35:24Z

UPDATE: I added a simple example just to give a sense of the interactive-mode UI. Not sure which features we want to highlight with examples otherwise.

dberenbaum

Looks good. Adding an example is great, but why not focus on a model training example?

dberenbaum · 2021-12-22T16:48:59Z

content/docs/command-reference/exp/init.md

@@ -19,43 +18,57 @@ usage: dvc exp init [-h] [-q | -v] [--run] [--interactive] [-f]

 ## Description


That's more or a problem for exp run though

Well that's the point of exp init, right? Better to have users understand what's needed up front than to have them run exp init only to fail on exp run.

Doesn't need to be part of this PR. It could also be handled by some of the suggested changes to the core command rather than the docs.

dberenbaum · 2021-12-22T16:55:17Z

content/docs/command-reference/exp/init.md

+
+Let's prepare a raw data ingestion script to start running experiments on it.
+
+```dvc


Why not a model training example? Data ingestion is a nice application to show how this can be useful even for non-training stages, but model training is the primary use case. It's unlikely for someone to run experiments on a data ingestion script.

The data ingestion example produced a very small dvc.yaml file and then I had that note about using stage add to produce the same. Idk, I thought it was interesting. Anyway, I agree the training scenario is more applicable so I replaced all that now.

dberenbaum · 2021-12-22T16:56:56Z

content/docs/command-reference/exp/init.md

+by our code in question, so the default is used. Models, metrics, and plots are
+omitted as there are none for an ingestion process.
+
+The resulting `dvc.yaml` file is simply:


Might be worth mentioning pipelines briefly to explain what's in dvc.yaml.

I'm actively avoiding mixing in that concept here but it is mentioned strategically once in the Description. I did elaborate on this sentence a bit though.

per #3071 (review)

jorgeorpinel · 2021-12-22T21:32:34Z

why not focus on a model training example?

OK @dberenbaum . Done in 56f4a46.

dberenbaum

Nice!

jorgeorpinel · 2021-12-23T02:26:12Z

Thanks!

jorgeorpinel · 2021-12-23T02:27:00Z

Looks like the failing link check is a false positive. I'll see if this is already reported to the link checker repo. Merging 🔀

content/docs/command-reference/exp/index.md

jorgeorpinel · 2022-01-12T23:13:23Z

content/docs/command-reference/exp/init.md

 # exp init

-Codify project using [DVC metafiles](/doc/user-guide/project-structure) to run
-[experiments](/doc/user-guide/experiment-management).
+Quickly setup any project to use [DVC Experiments].


I meant this (per last check box in this PR desc.) @dberenbaum @skshetry . Currently dvc exp init -h prints Initialize DVC in the current directory. Expects directory to be a Git repository... Do we want to update that text?

I think that's the dvc init -h text 😄 . dvc exp init -h says Initialize experiments. However, I think we agreed this new language is better, so I'm happy to put in a PR to update in the core repo.

I think that's the dvc init -h text 😄

yep 🤦 sry. OK, thanks for the PR

* ref: first copy edits to exp init * ref: clarify exp init explanations * ref: clarify `exp init` option descriptions * ref: re-describe `exp init` + reorder in nav and `exp` help per #3071 (review) * ref: clarify params.yaml is needed only with defaults in params.yaml per #3071 (comment) * ref: clarify what --interactive prompts user for per #3071 (review) * ref: link from exp init to config section and mention --explicit avoids a params.yaml file too. * ref: simplify exp init --explicit explanation * ref: explain why params.yaml are required (by default) in exp init per #3071 (comment) * ref: copy edits to exp init * ref: add simple example to exp init rel. #3071 (comment) * ref: use model training example in exp init per #3071 (review) * ref: shorten sample block

ref: first copy edits to exp init

153d526

jorgeorpinel self-assigned this Dec 7, 2021

jorgeorpinel mentioned this pull request Dec 8, 2021

guide: DVC Experiments Overview #2909

Merged

5 tasks

This comment has been minimized.

Sign in to view

shcheklein reviewed Dec 8, 2021

View reviewed changes

content/docs/command-reference/exp/init.md Show resolved Hide resolved

shcheklein reviewed Dec 8, 2021

View reviewed changes

content/docs/command-reference/exp/init.md Outdated Show resolved Hide resolved

shcheklein reviewed Dec 8, 2021

View reviewed changes

content/docs/command-reference/exp/init.md Outdated Show resolved Hide resolved

jorgeorpinel commented Dec 10, 2021

View reviewed changes

jorgeorpinel mentioned this pull request Dec 10, 2021

[WIP] cmd-ref: document exp init #3015

Merged

Merge branch 'master' into ref/exp/init +

38099d3

This was referenced Dec 10, 2021

Epic: exp init iterative/dvc#6446

Closed

exp init: handling non-existent params file iterative/dvc#7140

Closed

exp init: handling of nonexistent files iterative/dvc#7138

Closed

jorgeorpinel added 2 commits December 13, 2021 18:39

Merge branch 'master' into ref/exp/init

0509a51

ref: clarify exp init explanations

97592c7

jorgeorpinel mentioned this pull request Dec 14, 2021

exp init: unclear need for --explicit flag iterative/dvc#7143

Closed

ref: clarify exp init option descriptions

e70ea6d

jorgeorpinel marked this pull request as ready for review December 14, 2021 02:43

jorgeorpinel added 5 commits December 14, 2021 14:31

Merge branch 'master' into ref/exp/init

a97e611

ref: re-describe exp init + reorder in nav and exp help

8b89b8f

per #3071 (review)

ref: clarify params.yaml is needed only with defaults in params.yaml

2273431

per #3071 (comment)

ref: clarify what --interactive prompts user for

3d91656

per #3071 (review)

ref: link from exp init to config section and

69c8082

mention --explicit avoids a params.yaml file too.

jorgeorpinel requested review from shcheklein and dberenbaum December 15, 2021 00:32

dberenbaum suggested changes Dec 15, 2021

View reviewed changes

shcheklein temporarily deployed to dvc-org-ref-exp-init-nfacz5bbh December 16, 2021 02:43 Inactive

Merge branch 'master' into ref/exp/init

8f21601

jorgeorpinel temporarily deployed to dvc-org-ref-exp-init-nfacz5bbh December 21, 2021 10:08 Inactive

jorgeorpinel added 2 commits December 21, 2021 03:42

ref: simplify exp init --explicit explanation

be7a5af

ref: explain why params.yaml are required (by default) in exp init

600e83c

per #3071 (comment)

jorgeorpinel temporarily deployed to dvc-org-ref-exp-init-nfacz5bbh December 21, 2021 10:46 Inactive

jorgeorpinel requested a review from dberenbaum December 21, 2021 10:52

jorgeorpinel added 2 commits December 21, 2021 04:33

ref: copy edits to exp init

6e3d2fa

ref: add simple example to exp init

42aa486

rel. #3071 (comment)

jorgeorpinel temporarily deployed to dvc-org-ref-exp-init-nfacz5bbh December 21, 2021 11:34 Inactive

dberenbaum reviewed Dec 22, 2021

View reviewed changes

jorgeorpinel added 2 commits December 22, 2021 14:09

Merge branch 'master' into ref/exp/init

5e09813

ref: use model training example in exp init

56f4a46

per #3071 (review)

jorgeorpinel temporarily deployed to dvc-org-ref-exp-init-nfacz5bbh December 22, 2021 21:32 Inactive

jorgeorpinel requested a review from dberenbaum December 22, 2021 21:40

dberenbaum approved these changes Dec 22, 2021

View reviewed changes

ref: shorten sample block

fac16bd

jorgeorpinel temporarily deployed to dvc-org-ref-exp-init-nfacz5bbh December 23, 2021 02:22 Inactive

jorgeorpinel merged commit 33a303c into master Dec 23, 2021

jorgeorpinel deleted the ref/exp/init branch December 23, 2021 02:27

jorgeorpinel mentioned this pull request Dec 23, 2021

(Some) plain links in MD should be checked iterative/link-check#7

Open

jorgeorpinel mentioned this pull request Jan 6, 2022

guide: Experiment Management (phase 2) #2911

Closed

16 tasks

jorgeorpinel commented Jan 12, 2022

View reviewed changes

content/docs/command-reference/exp/index.md Show resolved Hide resolved

jorgeorpinel commented Jan 12, 2022

View reviewed changes

dberenbaum mentioned this pull request Jan 14, 2022

update exp init help text iterative/dvc#7271

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ref: `exp init` improvements #3071

ref: `exp init` improvements #3071

jorgeorpinel commented Dec 7, 2021 •

edited

Loading

This comment has been minimized.

shcheklein Dec 8, 2021

dberenbaum Dec 8, 2021

jorgeorpinel Dec 8, 2021 •

edited

Loading

dberenbaum Dec 10, 2021

jorgeorpinel Dec 11, 2021 •

edited

Loading

dberenbaum Dec 13, 2021

jorgeorpinel Dec 14, 2021 •

edited

Loading

jorgeorpinel Dec 10, 2021 •

edited

Loading

dberenbaum left a comment

dberenbaum Dec 15, 2021

jorgeorpinel Dec 21, 2021 •

edited

Loading

dberenbaum Dec 22, 2021

jorgeorpinel Dec 22, 2021 •

edited

Loading

shcheklein commented Dec 16, 2021

jorgeorpinel commented Dec 21, 2021

jorgeorpinel commented Dec 21, 2021

dberenbaum left a comment

dberenbaum Dec 22, 2021

dberenbaum Dec 22, 2021

jorgeorpinel Dec 22, 2021 •

edited

Loading

dberenbaum Dec 22, 2021

jorgeorpinel Dec 22, 2021

jorgeorpinel commented Dec 22, 2021

dberenbaum left a comment

jorgeorpinel commented Dec 23, 2021

jorgeorpinel commented Dec 23, 2021

jorgeorpinel Jan 12, 2022

dberenbaum Jan 14, 2022 •

edited

Loading

jorgeorpinel Jan 20, 2022

		### The `command` argument

		The `command` argument is optional, if you are using `--interactive` mode. The

		@@ -19,43 +18,57 @@ usage: dvc exp init [-h] [-q \| -v] [--run] [--interactive] [-f]

		## Description


		Let's prepare a raw data ingestion script to start running experiments on it.

		```dvc

ref: exp init improvements #3071

ref: exp init improvements #3071

Conversation

jorgeorpinel commented Dec 7, 2021 • edited Loading

This comment has been minimized.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorgeorpinel Dec 8, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorgeorpinel Dec 11, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorgeorpinel Dec 14, 2021 • edited Loading

Choose a reason for hiding this comment

jorgeorpinel Dec 10, 2021 • edited Loading

Choose a reason for hiding this comment

dberenbaum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorgeorpinel Dec 21, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorgeorpinel Dec 22, 2021 • edited Loading

Choose a reason for hiding this comment

shcheklein commented Dec 16, 2021

jorgeorpinel commented Dec 21, 2021

jorgeorpinel commented Dec 21, 2021

dberenbaum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorgeorpinel Dec 22, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorgeorpinel commented Dec 22, 2021

dberenbaum left a comment

Choose a reason for hiding this comment

jorgeorpinel commented Dec 23, 2021

jorgeorpinel commented Dec 23, 2021

Choose a reason for hiding this comment

dberenbaum Jan 14, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ref: `exp init` improvements #3071

ref: `exp init` improvements #3071

jorgeorpinel commented Dec 7, 2021 •

edited

Loading

jorgeorpinel Dec 8, 2021 •

edited

Loading

jorgeorpinel Dec 11, 2021 •

edited

Loading

jorgeorpinel Dec 14, 2021 •

edited

Loading

jorgeorpinel Dec 10, 2021 •

edited

Loading

jorgeorpinel Dec 21, 2021 •

edited

Loading

jorgeorpinel Dec 22, 2021 •

edited

Loading

jorgeorpinel Dec 22, 2021 •

edited

Loading

dberenbaum Jan 14, 2022 •

edited

Loading