stage add: fails with dirs that will be created later by the cmd(s)

[jorgeorpinel]

Bug Report

Description

If the stage being defined creates a dir and files in this dir are stage outputs, stage add will fail (because it can't write the corresponding .gitignore file).

Reproduce

$ dvc stage add -n hidir -o dir/hi 'mkdir dir && echo hi > dir/hi'
ERROR: unexpected error - [Errno 2] No such file or directory: '/.../dir/.gitignore'

Expected

I guess the full path to the .gitignore in question should be created if needed. Or maybe create it at repro? Not sure how, but I would expect the stage definition not to fail for this.

Environment information

Output of dvc doctor:

$ dvc doctor
DVC version: 2.0.17 (pip)
---------------------------------
Platform: Python 3.6.9 on Linux-5.4.72-microsoft-standard-WSL2-x86_64-with-Ubuntu-18.04-bionic
Supports: gdrive, hdfs, http, https, s3, ssh, oss
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: None
Workspace directory: ext4 on /dev/sdb
Repo: dvc, git

[dberenbaum]

@skshetry This also can cause dvc exp init to fail:

$ dvc exp init --model models/model.h5 python src/train.py
ERROR: unexpected error - [Errno 2] No such file or directory: '/Users/dave/repo/models/.gitignore'

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

[dberenbaum]

@skshetry I don't want to slow down other work for this, but this seems like a fairly serious bug for exp init. It actually breaks the example in https://dvc.org/doc/command-reference/exp/init#example-interactive-mode. What do you think about priority and solution?

[skshetry]

@dberenbaum, for exp init, we can consider creating the parent directory of those outputs, where the .gitignore file will be added.

However, we can also choose not to generate .gitignore entries on exp init/stage add in these edge cases, as successive dvc repro and dvc exp run will try to generate them as well.

If we do want to always generate .gitignore, we need to fix how we create .gitignore files. We need to change the gitignore-generator to backtrack up to the dvc's root directory, likewise on dvc remove.

[dberenbaum]

successive dvc repro and dvc exp run will try to generate them as well.

In that case, do you know why we try to create .gitignore entries on stage add?

[dberenbaum]

Marking this as p1 until at least the dvc exp init scenario is resolved (see iterative/dvc.org#3430 (comment)).

[daavoo]

In that case, do you know why we try to create .gitignore entries on stage add?

Any answer to this?

[dberenbaum]

Bumping down to p2 following #7752

[dberenbaum]

In that case, do you know why we try to create .gitignore entries on stage add?

Any answer to this?

See #7740 (comment)