exp: include all expected files

[dberenbaum]

There are several issues where files/changes from the workspace end up not being reflected in the experiment. This is a meta-ticket to track those issues:

Tasks

Preview Give feedback

1. exp run: deleted git-tracked files are restored after exp run #6297
   A: experiments bug research +3
   
2. exp run: include new staged files #8998
   A: experiments +1
   
3. exp save: Handle staged files like exp run #9058
   A: experiments p1-important +2
   
4. exp run/save: track top-level params/metrics/plots #8924
   A: experiments p1-important +2
   
5. exp save: Include untracked dvc.lock #9238
   A: experiments p1-important +2
   
6. Track output folder when _inside_dvc_exp. dvclive#510
   A: dvc p1-important +2
   
7. exp run: local config settings are not propagated into temp workspace #7254
   A: experiments bug p1-important +3
   
8. dvc exp run --queue: deleting file 'params.yaml' for each command call #7842
   A: experiments bug p1-important +3
   
9. exp run: copy (certain) git-ignored files to tmp folder on --run-all #5800
   A: experiments feature request p3-nice-to-have +3
   

Edit: reordered to reflect priority

[dberenbaum]

Related: #9165 (comment)

[daavoo]

Related: #9165 (comment)

Added #9238 to the list

[daavoo]

Added iterative/dvclive#510 to the list (kind of extracted from #9165 )

[daavoo]

@dberenbaum I think we are in good shape for getting this done in this sprint, except for #5800 .
I have tried to go through the discussion and its still unclear to me what is the request 😅 Do you recall anything in particular?

[dberenbaum]

Makes sense, thanks @daavoo! Now we are tracking everything that I can think of to reasonably track by default. If there are other files people expect to be included, we could:

1. Add an option to specify files to copy over.
2. Copy everything over from the workspace.

I'm torn on whether it's worth prioritizing anything else, but what do you think about including something like 1? Do you think it's easy to add?

[daavoo]

what do you think about including something like 1?

Sounds good to give the option although it feels like if you have to know about that option + provide the right values you might as well do git add, right?

We could anyhow do something like the exp save --include-untracked

Do you think it's easy to add?

I think so, yes. 1 day.

[dberenbaum]

Sounds good to give the option although it feels like if you have to know about that option + provide the right values you might as well do git add, right?

We could anyhow do something like the exp save --include-untracked

Yeah, I'm not sure that exp save --include-untracked has an obvious use case. The use cases I have in mind are:

1. Credentials or other sensitive info that shouldn't be tracked by Git (like the OP in term: replace 'DVC-file' and 'stage file' with correct terms (#3960) #5280)
2. Large files that you don't want to track (for example, it's static or easy to regenerate; see https://github.com/iterative/cse/issues/99)

So I think it needs to be more like --copy-untracked where the files get copied to the tmp dir but remain untracked.

[daavoo]

So I think it needs to be more like --copy-untracked where the files get copied to the tmp dir but remain untracked.

So, is it about the cases where --tmp / --queue fail because the experiment relies on something that is:

• Not git tracked
• Not defined as an absolute path

Correct?

---

For that, I am not sure about estimation without a clear scope.

[dberenbaum]

So, is it about the cases where --tmp / --queue fail because the experiment relies on something that is:

* Not git tracked

* Not defined as an absolute path

Correct?

Yes

For that, I am not sure about estimation without a clear scope.

I think you just defined the scope very well 😄. Do you still have questions, or you just would need to look closer to give a decent estimation?

[daavoo]

I think you just defined the scope very well 😄. Do you still have questions, or you just would need to look closer to give a decent estimation?

Forgot to estimate. 1 day was right 😅
I left some questions on the P.R. #9302