pull: Failed to pull data from the cloud. <filename> is git-ignored.

[benjamintanweihao]

Bug Report

Description

I tested this between DVC versions 1.11 and 2.0. My setup is the following:

I have a folder called env/resources where I store Tensorflow binaries which are tracked by DVC. env/resources is also git-ignored.

When I do dvc pull in earlier versions of DVC (i.e 1.x) everything works as expected. However, when I do the same in version 2, I get pull: Failed to pull data from the cloud. <filename> is git-ignored.

Reproduce

Example:

1. dvc init
2. Copy dataset.zip to the directory
3. dvc add dataset.zip
4. dvc push
5. Add the directory to .gitignore.
6. git clone the repository somwhere else.
7. dvc pull

Expected

dvc should have let me pull in the file even though the directory is git-ignored.

Environment information

Output of dvc version:

DVC version: 2.0.0a0+bb4604 (snap)
---------------------------------
Platform: Python 3.6.9 on Linux-5.4.0-62-generic-x86_64-with-Ubuntu-18.04-bionic
Supports: All remotes
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/nvme0n1p2
Caches: local
Remotes: s3
Workspace directory: ext4 on /dev/nvme0n1p2
Repo: dvc, git

[skshetry]

@benjamintanweihao, could you please share the reason why you add the directory to .gitignore?

DVC already adds the tracked directory to the .gitignores.

[benjamintanweihao]

That was done before we used dvc. Even if dvc were to add an entry to the gitignore file, it should have the same effect.

…

On Wed, Feb 3, 2021 at 5:02 PM Saugat Pachhai ***@***.***> wrote: @benjamintanweihao <https://github.com/benjamintanweihao>, could you please share the reason why you add the directory to .gitignore? DVC already adds the tracked directory to the .gitignores. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5396 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGSINH5P3NY3NN7ERQ6QD3S5EGLRANCNFSM4XAMMJOQ> .

[skshetry]

Hi @benjamintanweihao, is it possible to exclude those .gitignores?

[macio232]

Up

[skshetry]

@macio232, could you please provide more details? How is it affecting you? And, is that possible to unignore them? Thanks. 🙂

[macio232]

Actually, all the details are in the first post :) Files/directories added to .gitignore (for whatever reason) and tracked by DVC can not be pulled, which makes not much sense for me and is different from the previous behavior.

I was testing on the pre-release version mentioned in the first post here, and I don't know if the bug (or feature) is present in the latest version (which is 2.0.5 while typing). I can not verify this because I no longer have access to the project.

[efiop]

@macio232 That behaviour was introduced in 2.0 intentionally. You need to exclude corresponding *.dvc (or dvc.yaml) files from .gitignore, otherwise dvc won't be able to discover them.

[efiop]

Just to clarify: we've started ignoring such dvcfiles in 2.0 to improve the performance #5265 @macio232 Would it be possible to unignore those dvcfiles in your scenario?

[benjamintanweihao]

We have the following scenario: We have a bunch of DVC-tracked files in a folder called "resources", where we want to make sure that developers do not accidentally git commit and push. Therefore, the "resources" folder is git-ignored. Of course, this means that when we commit the *.dvc files we would have to do a git add -f but this is a small price to pay instead of having to do a git-filter-branch when someone accidentally commits a 100MB file into the master branch.

…

On Sat, Mar 13, 2021 at 8:43 AM Ruslan Kuprieiev ***@***.***> wrote: Just to clarify: we've started ignoring such dvcfiles in 2.0 to improve the performance #5265 <#5265> @macio232 <https://github.com/macio232> Would it be possible to unignore those dvcfiles in your scenario? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5396 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGSING7C2X657WMN3QI77LTDKYLXANCNFSM4XAMMJOQ> .

[macio232]

@efiop exactly as @benjamintanweihao wrote - the usual strategy is to put the whole resources/data directory to .gitignore because of data confidentiality reasons. Sometimes you also have local data resources which you do not want to track either by git and DVC.

Moreover, I have the data directory added to .gitignore in all of my projects. I want to upgrade to 2.0, but this requires taking care of each individual file that should be gitignored.

[efiop]

@macio232 @benjamintanweihao Would it be possible to add an exclusion rule only for *.dvc and dvc.yaml?

[macio232]

@macio232 @benjamintanweihao Would it be possible to add an exclusion rule only for *.dvc and dvc.yaml?

Yes! I think this is a solution, and I feel stupid I didn't think of it myself :) Thank you @efiop !

[OSobky]

@macio232 @benjamintanweihao Would it be possible to add an exclusion rule only for *.dvc and dvc.yaml?

Hello @efiop, what was the exclusion rule? I am not sure why did it disappear.