exp init experience

[dmpetrov]

$ dvc -V
2.9.2

$ git init
$ dvc init
$ dvc exp init python train.py
ERROR: unexpected error - [Errno 2] No such file or directory: 'params.yaml'

• Why is it unexpected error? It should be like “ERROR: hyperparameters file ‘params.yaml’ is missing”. 
🤔 Why dvc exp init does not create it if it is required?
• Do we really need [Errno 2]? In what cases the code helps users? This makes the message longer and breaks human readability.

$ dvc exp run
ERROR: failed to reproduce 'dvc.yaml': [Errno 2] No such file or directory: '/Users/dmitry/src/test/myproj/data'

• full path. It is supposed to be relative. Why: 1) to make it short - the full path provides zero value in most of use cases; 2) it makes it hard to submit reports (some users are not comfortable to share their directories struture - See '/Users/dmitry/src/test/ - bc it might contain confidential info like user and project names.

• why dvc exp init checked params.yaml but didn’t check (or create) data/ & src/?

$ mkdir data
$ dvc exp run
ERROR: failed to reproduce 'dvc.yaml': [Errno 2] No such file or directory: '/Users/dmitry/src/test/myproj/src'

• the same questions - [Errno 2], Full path.

• why it was not checked in dvc init? Or was not created?

$ mkdir src
$ dvc exp run
…
python: can't open file 'train.py': [Errno 2] No such file or directory
ERROR: failed to reproduce 'dvc.yaml': failed to run: python train.py, exited with 2

• “exited with 2” Is it useful information? Where the exit code 2 belongs to? It might confuse users. Error code can be in debug errors not here.

• Another issue - ideally, we should have a reference to stage (a user might have multiple stages with the same or similar commands). The message might look like:
 ERROR: failed to execute stage ‘train’ from ‘dvc.yaml’

From a clean git repo:

$ dvc init
$ dvc exp init -i
…
Path to a parameters file [params.yaml, n to omit]:
'params.yaml' does not exist. Please retry with an existing parameters file.

• If I do not have params.yaml I stuck here. Why don’t we offer to create an empty params file?

• A kind of conclusion: dvc exp init is not fully supported without interactive mode.
  It looks like today the only supported flow is — interactive. The default dvc init has multiple assumptions about the project structure and it is not clear how a user should know about the assumptions. Why dvc init won’t create all the directories & files? This was a part of the initial idea - to run a single cmd for a given train.py in a live coding section while dvc init wraps it to an experiment.

This is how it can look like:

$ dvc init python train.py
Creating experiment project structure:
├── data/
├── metrics.json
├── models/
├── params.yaml
├── plots/
└── src/
Created `dvc.yaml` file with ‘train’ stage. To run, use "dvc exp run".

[dberenbaum]

@dmpetrov I broke this up into 3 related issues mentioned above to make it more manageable to take action on these. Please add anything missed from those, and we can continue discussion there. 🙏

[dberenbaum]

After speaking with @dmpetrov, one of the high-level issues is that the defaults for dvc exp init are asking the user for too much, too soon, and expecting/assuming too much from the user. Some ideas on how we can fix it:

1. Make dvc exp init -i the default.
2. Make dvc exp init --explicit the default.
3. Make dvc exp init -i the default if no command argument is provided, otherwise make dvc exp init --explicit the default.

@skshetry Let me know if you have thoughts.

In the interactive mode, this might also relate to earlier discussions about whether a blank entry should be skipped or use the default path.

[dberenbaum]

Other options:

4. Create another type like simple or blank that creates a stage with only a command (no deps or outputs). This might also allow us to get rid of --explicit.
5. Allow for --data, --params, etc. to be passed without arguments, in which case they will be added to the stage with default values. This would be useful if --explicit is the default or if there is blank stage type. If we do this, we probably need short flags for these to enable something like dvc exp init -dp.

IMO making --explicit the default might make the command too close to dvc stage add, but the other ideas could all work.

[dmpetrov]

IMO making --explicit the default might make the command too close to dvc stage add, but the other ideas could all work.

Agree. There are some good options but this one seems like the best one.

[dberenbaum]

Agree. There are some good options but this one seems like the best one.

@dmpetrov We might be saying slightly different things 😄. However, if you want to start from this simple default, I think the best way to achieve this is option 4 above. We can have 3 template types: default (a stage with only a command), ml (the current default), and dl.

Why add another template type? One goal of dvc exp init was to introduce an easy way to setup typical project structures, such as deep learning with checkpoints and dvclive since it's confusing to configure manually. If we always start with a blank stage, there's no real difference between these template types (at least in non-interactive mode), and their purpose becomes confusing.

[dmpetrov]

@dberenbaum oh, right 🙂 --explicit is going to be the same as stage add. It is not a problem to be the same, the problem is it does not serve the purpose of exp init - being a shortcut/cookiecutter for experiments.

There are couple more questions:

• --type=dl has the same issue as the default one - dvc exp init --type dl is not functional until you create all the data/src/metrics files and dirs.

• Why there are no plots in --type=dl (we have plots in the regular one)?

• --interactive also does not create a proper structure for the default options. It is very easy to check the file and dirs for existence in interactive mode and ask if they'd like to create the files. That's what the user will see as the result:

$ dvc exp run
WARNING: Your workspace contains staged Git changes which will be unstaged before running this experiment.
ERROR: failed to reproduce 'dvc.yaml': [Errno 2] No such file or directory: '/Users/dmitry/src/test/exp_init_i/data'

My opinion (similar to the Conclusion in the initial issue description) - dvc exp init (including --interactive) cannot successfully finish if the specified (or default) files and directory structure is not created. Otherwise, we just moving exp init issues to the laster dvc exp run stage where the issues cannot be solved automatically. So, we are basically moving the issues to the user's shoulders.

[dberenbaum]

@dmpetrov It seems there are multiple use cases:

1. Taking a code script without dependencies or outputs and running it as an experiment. The blank template/--explicit default can solve it, or we can make some alias for dvc stage add to do this.
2. Adapting an existing project with parameters, metrics, data, etc. into a DVC experiment. Could you take a look at exp init: handling of nonexistent files #7138 and provide your feedback? Creating empty files may still move the issues to the user's shoulders. Better messaging can give hints to users on what to do.

Another possible use case:
3. Adding dependencies and outputs gradually.

One possible way to do this is to allow dvc exp init to append new information to an existing stage. For example:

$ dvc exp init python train.py
Created train stage in dvc.yaml.
$ cat dvc.yaml
stages:
  train:
    cmd: python train.py
$ dvc exp init --data --models model.h5
Modified train stage in dvc.yaml.
$ cat dvc.yaml
stages:
  train:
    cmd: python train.py
    deps:
    - data
    outs:
    - model.h5
$ dvc exp init --data raw
ERROR: The specification conflicts with stage 'train' in 'dvc.yaml'. Use '--force' to overwrite.
$ dvc exp init -f --data raw
Modified train stage in dvc.yaml.
$ cat dvc.yaml
stages:
  train:
    cmd: python train.py
    deps:
    - raw
    outs:
    - model.h5 

Note that this isn't possible in stage add since every dep/out type may be passed multiple times in a stage, making it impossible to know when to append vs. overwrite.

[dmpetrov]

I was expecting to have a cookie-cutter based on a user's script. We need to generate a template with a predefined structure that users can use to adopt the project step by step. The predefined structure also serves an education goal - user knows possibilities and how to adopt these.

I think we have to have a proper cookie-cutter as one of the options of dvc exp init. It would be great to have this as the default one.

--explicit as default might be an easy starting point but we are losing the idea of cookie-cutter and educational part of it. As a result, we push people to read all the exp docs.

Adding dependencies and outputs gradually is a good idea but looks like overkill 🙂 As we discussed before, we should promote the idea of modifying dvc.yaml manually, not by commands.

In any of these scenarios, I don't think we should keep a user with not working dvc exp run (as much as we can). Missing files/dir should lead to command failure (or error messages with a repeat in --interactive mode).

[dberenbaum]

In any of these scenarios, I don't think we should keep a user with not working dvc exp run (as much as we can). Missing files/dir should lead to command failure (or error messages with a repeat in --interactive mode).

Right, I think everyone can agree with this. We can prioritize that for now while we continue to think through other ways to improve the experience.

[skshetry]

We should still try to guide users on exp run, as these may still happen if dvc.yaml is modified by users. That does not mean that we cannot improve exp init message.

Another problem that I see these days is that exp init has a concept of data/models/source, etc. which we don't have in dvc.yaml (which is translated into dependencies/outputs instead) which makes the successive exp run confusing.

[skshetry]

If we make --explicit the default, we'll force users to iterate their stage through dvc.yaml later, I am not sure if this a good or a bad thing though, but they have different concepts that don't translate well (as mentioned above).

[dmpetrov]

@skshetry good point about --explicit. A whole set of defaults might lead to a better default experience.

[skshetry]

@dberenbaum and I discussed creating workspace structure on defaults, and showing workspace tree on non-interactive mode as we do for interactive modes. We also need to work on #4112 to support tracking everything inside of params file.

[skshetry]

@dmpetrov mentioned in our discussion that plots is an advanced feature, so we may want to make it optional here in the exp init.

[dberenbaum]

Should we convert this to a discussion after #7331 rather than close it completely? The immediate action points have been addressed, but I don't think everything has been resolved.

---

Looking at #7331, It might be helpful to separate inputs and outputs in the UI to clarify which will be created. No action needed now, but to me it seems confusing why some paths are created and others aren't.

@tapadipti shared her experience with dvc exp init and mentioned that it's confusing that for data, you have to do dvc add, but not for models.

[skshetry]

Sure, let's turn it into discussion after #7331, I'll remove the closing marker.

@tapadipti shared her experience with dvc exp init and mentioned that it's confusing that for data, you have to do dvc add, but not for models.

Since we create data directory now (after #7331), it makes sense to also track them by defaults. It's a bit unclear if we should always do this or only when we are creating those directories ourselves during exp init.

[skshetry]

I just remembered that I have an open PR for tracking dependencies in #6914, but haven't moved forward with it because it requires us to basically re-implement simpler dvc add. Tracking dependencies but without ensuring any graph correctness is going to be simple (duplicated outputs check, overlapped output, etc), but otherwise we'll end up re-implementing dvc add.

We cannot reuse dvc add as it's more complex, has its own messy output and progress bar, and in dvc exp init, we need to check correctness together with the pipeline stage.

[dberenbaum]

(duplicated outputs check, overlapped output, etc)

Sorry, can you explain? I didn't get your point.

[skshetry]

@dberenbaum, we need to check if the data output is in conflict with other existing .dvc files or stages and with the pipeline stage that we are going to generate. The checks that we do are:

1. That the graph is acyclic (is a DAG),
2. That the output is not duplicated,
3. That the output does not overlap with an already existing output, and
4. That the .dvc file to be generated is not inside the output of a different stage, and
5. The .dvc file does not exist already.

This is a bit complicated and involves collecting all dvc.yaml and .dvc files. #6914 does already handle all of that, but it does so by duplicating what dvc add does already. I'm just looking for solving this in a better way, or waiting to come up with a better solution.

We could however simplify the implementation by choosing not to verify, as it is also done later on every operation.

[dberenbaum]

Let's leave it for now since I believe some of the planned data management changes could impact this, and it might make more sense to revisit later.