How to sort out extension ehowing huge amount of "pending changes"

[dibus2]

When starting the VS code extension it shows 110K Pending Files.
Application is extremely laggy to the point where it makes using the EC2 machine impossible.
The DVC section in the GIT tab shows nothing (as I suspect there are too many files).
The way the repo is setup is as such:

• project_1 / pipeline_1 / dvc.yaml
• project_1 / pipeline_2/ dvc.yaml
• project_1 / pipeline_3/ dvc.yaml
• ...
• project_1 / pipeline_13/ dvc.yaml
• project_2/ ...

I also tried to specialized the dvc extension to a single pipeline but it doens't seem to be helping. This is what I have in my settings.json
"dvc.focusedProjects": [
"project_1/pipeline_1"]
Note that pipeline_1 folder contains the dvc.yaml file.

Any help on sorting this out is appreciated.

Thanks.

[mattseddon]

Have you tried running dvc pull before opening the extension? Is there a folder that should be gitignored that isn't?

[dibus2]

Hi Matt,

thanks for looking at this. I had not done a pull just before starting it to be fair so I did, got the number down to 108K. My git status shows nothing so this is all the status added by DVC.

You are bringing a good point though, that some of these pipilines are maintained only by some teams and ideally other teams don't have to pull the data from all other pipelines if they don't need it.

Cheers,

[dibus2]

Here is the breakdown of the git DVC section actually. It wasn't starting earlier but this time it managed to start.

I honestly don't understand why there are all these files in the Uncommitted section. When I look at these files they are actually files for which there is a .dvc file in the repo but that I have not pulled because they are not part of any pipelines and are as such not needed in the current commit of the repo.

[dibus2]

Also, shouldn't these all be ignored since I specified a specific folder through dvc.focusedProjects that does not include any of these .dvc file in it?

[mattseddon]

I would guess that there are duplicates between uncommitted and not in cache as there is an unknown uncommitted status that things fall into when they are not in the cache (I forget the history on this), you can search the DVC issues if you want to know more.

For focused projects, these relate to - "A subset of paths to the workspace's available DVC projects. Using this option will override project auto-discovery." I am guessing that you've set this to a pipeline instead of a project, so it failed validation and bypassed the option. Try using the "Select Project(s) to Focus (set dvc.focusedProjects)" quick pick to select a project to focus. This should give a list of valid options.

[dmpetrov]

@dibus2 thank you for reporting this!

[shcheklein]

@dibus2 thanks for creating the issue! a few questions:

• where do you have .dvc directories for those projects? (just to better understand how things are isolated from each other)
• 50K+ files - is it a single directory? what is the part in the tree to it? are those parquet files? is it input for the pipeline?

[dibus2]

@shcheklein

• so each one of the pipeline_x folders contain a .dvc.yaml file.
• the files are spread across different projects/ in which we have a /data folder in each one of them.

For instance in project_1/data/

• data_set_1.dvc
• data_set_2.dvc
• data_set_3.dvc
  ...
• data_set_10.dvc

Now most of these are usually not being pulled because they are not used anymore or not yet and maybe that's a problem to keep the .dvc files around?
From a content perspective, these are mostly JSON files actually, there are few parquet but they are mostly json files.

[dibus2]

I would guess that there are duplicates between uncommitted and not in cache as there is an unknown uncommitted status that things fall into when they are not in the cache (I forget the history on this), you can search the DVC issues if you want to know more.

For focused projects, these relate to - "A subset of paths to the workspace's available DVC projects. Using this option will override project auto-discovery." I am guessing that you've set this to a pipeline instead of a project, so it failed validation and bypassed the option. Try using the "Select Project(s) to Focus (set dvc.focusedProjects)" quick pick to select a project to focus. This should give a list of valid options.

I actually do not find the quick pick to select the project option @mattseddon

[shcheklein]

@dibus2 sorry, just to clarify. I specifically mean .dvc directories (not files). Do you have a single one in the root or multiple (I'm referring to the --subdir option).

Now most of these are usually not being pulled because they are not used anymore or not yet and maybe that's a problem to keep the .dvc files around?

yes, if you don't need them - better to drop them I guess (you can always recover them from the Git history - that's the beauty of DVC).

[dibus2]

ah sorry, yes I have only one in the top folder @shcheklein

[shcheklein]

so, if the projects are independent - can we consider making them subprojects? that might help I think DVC and extension and Studio a lot. (each DVC command won't be analyzing all the existing pipelines, it can "focus" only on a single one at a time).

[dibus2]

actually let me explore that option. How do I do that?

[shcheklein]

I think you can just try to do dvc init --subdir in one of the subdirectories. Then it might be needed to migrate some .dvc/config options the the /.dvc/config. That should be enough to start I think. And let's see if we hit some issues.

[dibus2]

Hi @shcheklein,
I looked into that, I can't really do this and the reason is because we are actually sharing common utils files and datasets across these different projects.

However, I did cleanup the repo removing all the .dvc files that were not needed in the current commit and I got it down to only 14 pending piles which I think can be ignored. However, I still can't do anything with the extension.

It's getting stuck into I'm not sure None of the commands are responding.

I did notice that the data dvc status takes about 40 seconds and it seems to run this quite a bit and I wonder if it's not just spinning its wheels on this?

[shcheklein]

I looked into that, I can't really do this and the reason is because we are actually sharing common utils files and datasets across these different projects.

I guess util files are fine (Python files?)

Datasets - it depends a bit. It's not usually a problem in DVC to duplicate it (if cache is shared and you use symlinks / reflinks / hardlinks - there will be no impact on space or anything).

I see that plots diff also takes quite a long time.

I suspect that even collecting a full dag is probably an expensive operation - could you run dvc status in CLI? how much time does it take?

I see also that even git log takes 2 seconds to run - weird. Is it the same in CLI?

[dibus2]

so @shcheklein regarding time to run dvc status see screenshot

Regarding the git log I m not sure where you see it in the log it doesn't take any time in the cli.

Regarding the plots/ I noticed that we are actually tracking output folders in a lot of the pipelines / and that lead to not being able to show them through the show plots in the extension (at least I assume because it says there is no plots to show)

Even though we have 100s of plots being tracked here is one example

[mattseddon]

I suspect that processing the data for these plots is blocking the extension host thread. We could add something like a focus pipeline option but that would be at least a couple of days of work.

[shcheklein]

@mattseddon do we run plots diff before we open the plots view?

Since data status takes 55s to run even in CLI, not sure that the plots is the (only) issue.

It seems it just takes time to get all the DVC files together.

How many dvc.yamls do we have? How many .dvc files? How many files overall in the tree?