dvc add produces /tmp in .dvc/.gitignore

[RomanVeretenov]

dvc version 0.91.1
installed via pip on ubuntu 18.04

Each dvc add produces extra line '/tmp' in .dvc/.gitignore.

 % git diff .dvc/.gitignore

diff --git a/.dvc/.gitignore b/.dvc/.gitignore
index 9bd2370..cfd5c3c 100644
--- a/.dvc/.gitignore
+++ b/.dvc/.gitignore
@@ -10,3 +10,6 @@
 /tmp

 /tmp
+/tmp
+/tmp
+/tmp

[efiop]

@RomanVeretenov Does it happen after specific dvc command?

[efiop]

@RomanVeretenov Could you show us git check-ignore $(pwd)/.dvc/tmp output, please? Is there anything special about your repo location?

[shcheklein]

@efiop it happens on older repos. Try to clone our get started and run a few commands (pull, checkout, etc).

[RomanVeretenov]

@RomanVeretenov Could you show us git check-ignore $(pwd)/.dvc/tmp output, please? Is there anything special about your repo location?

output is empty

[efiop]

@shcheklein It does, but it only adds the missing /tmp(as it is supposed to be), not adding it multiple times as in this issue. I am not able to reproduce the issue.

@RomanVeretenov Ok, that is bad, it means that git is not seeing that /tmp in .gitignore or not understanding it right. Is there anything special about the location that your repo is located in?

Could you show:

$ python -c 'import os; print(os.getcwd())'
$ python -c 'import os; print(os.path.realpath(os.getcwd()))'

please?

So far this looks like there is something wrong with your environment or git specifically.

[RomanVeretenov]

$ python -c 'import os; print(os.getcwd())'
/home/ds
$ python -c 'import os; print(os.path.realpath(os.getcwd()))'
/home/ds

yep, the repo root is '/home/ds' =)

[efiop]

@RomanVeretenov Using /home/ds is questionable, but nothing functionally wrong about it 🙂 Looks alright. Maybe you have .dvc/ gitignored somewhere or something?

Need to check your .gitignores in /home and /home/ds to see if they are sane(maybe they have something like !!.dvc/tmp, !!tmp or something that forces git to not ignore .dvc/tmp. Would also check global gitignore rules, if you have those.

[RomanVeretenov]

There is no /home/.gitignore
Also I have no global gitignore
Also .git/info/exclude is empty

/home/ds % cat .gitignore

*.jpg
*.JPG
*.png
*.PNG
extracted.csv

also I have recursively listed all gitignore's
None of them but .dvc/.gitignore conatins tmp

find . -name '.gitignore' -exec echo {} \; -exec grep tmp {} \;
... here goes gitignores from all nested folders
./.dvc/.gitignore
/tmp
/tmp
./.gitignore
...

Each dvc status call adds a /tmp to the end of .dvc/.gitignore

[efiop]

@RomanVeretenov Could you try to reproduce it with a newly created dvc project? E.g.

mkdir myrepo
cd myrepo
git init
dvc init
git add .
git commit -m "init"
dvc status
git status

I'm still not able to reproduce your issue, most likely there is something off with your environment, can't put my finger on anything yet.

[RomanVeretenov]

@RomanVeretenov Could you try to reproduce it with a newly created dvc project? E.g.

mkdir myrepo
cd myrepo
git init
dvc init
git add .
git commit -m "init"
dvc status
git status

I'm still not able to reproduce your issue, most likely there is something off with your environment, can't put my finger on anything yet.

It works ok on a clean repo

 ~/code/myrepo/.dvc
 % cat .gitignore
/config.local
/updater
/lock
/updater.lock
/tmp
/state-journal
/state-wal
/state
/cache

after executing all given commands

[efiop]

It works ok on a clean repo

@RomanVeretenov So there is something with your .gitignores in the rest of the project. You'll have to take a look and see what is different between it and a clean one. It might be .gitignores somewhere, might be the fact that you are in /home/ds (esp if you are working as ds user), might be that you don't have permissions to access some files (again, if your git project is not owned by you and you don't have shared mode enabled in git config). But it is clear that this is not dvc-issue, it is your git repo and environment that are not quite right. Sorry, but you are pretty much on your own right now, as I can't put my finger on anything specific 🙁

[efiop]

Please let us know how it goes. I'll close this issue for now.

[shcheklein]

@RomanVeretenov

When /tmp is already part of the .dvc/.gitignore:

what does git check-ignore ".dvc/tmp" return?

also, what will the script like this return in your case:

from dvc.scm.git import Git
git = Git()
git._ignored(".dvc/tmp")

I wonder if it's some yet-another-gitpython's-bug.

[efiop]

@shcheklein Already checked: #3561 (comment) , it is not gitpython.

[shcheklein]

@RomanVeretenov @efiop

@shcheklein Already checked: #3561 (comment) , it is not gitpython.

sorry, missed that ... this is really weird!

@RomanVeretenov can you just do it all manually. Save the old .dvc/.gitignore somewhere, create a new one with a single entry /tmp and and run from the project's root git check-ignore .dvc/tmp?

Do you use submodules or any other advanced Git's stuff?

[RomanVeretenov]

Will re-check everything next week

[ammarasmro]

I was having this issue and discussed it with Ivan on Discord. Here is part of our conversation.

For some reason on my repo, running
echo "test 1" >> models/README.md
dvc add models
echo "test 2" >> models/README.md
dvc add models

Is appending models to .gitignore twice. I tested it on a new repo and you're right it doesn't append it more than once. I'll try to reset dvc and see if that works

I tried DVC destroy and started again. Didn't help. The only thing that worked is that I started with a clean models folder, added to dvc/git/.gitignore, and then added the data to it.

[efiop]

From https://discordapp.com/channels/485586884165107732/563406153334128681/703247236368302091

git init
mkdir models
echo "test 1" >> models/README.md
git add models/README.md
git commit -m "Add requirements file"
git checkout -b b1
dvc init
dvc add models
git add models.dvc .gitignore
cat .gitignore
echo "test 2" >> models/README.md
dvc add models
cat .gitignore

So looks like we should handle it better on DVC side. I suppose @RomanVeretenov had something similar. Reopening

[efiop]

Ok, looks like we broke is_tracked during one of the rounds of optimizing dvc for a use case with many thousands of dvc-files.

Python 3.7.0 (default, Dec 26 2018, 22:48:20)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information
>>> from dvc.scm.git import Git
>>> g = Git(".")
>>> g.is_tracked("tests/__init__.py")
True
>>> g.is_tracked("tests")
False
>>> g.is_tracked("tests/")
False
>>> g.is_tracked("tests/func")
False
>>> def func(path):
...     return bool(g.repo.git.ls_files(path))
...
>>> func("tests")
True
>>> func("tests/")
True
>>> func("tests/unit")
True

func represents an old implementation and works like charm. Need to look into it. And, more importantly, add better tests for it.

EDIT: it was an older bug after all.

[efiop]

Big thanks to @ammarasmro for investigating! 🙏

[ammarasmro]

Glad I could help :)

[efiop]

@RomanVeretenov @ammarasmro Guys, could you please try installing dvc from master via pip install git+https://github.com/iterative/dvc and see if it fixes the issue for you? From now on, dvc will properly detect dirs that are tracked by git already and should just error-out.

[ammarasmro]

I got this error

ERROR: output 'models' is already tracked by SCM (e.g. Git)

It might help to add a suggestion on how to proceed. Like do they just need to delete the files? Add it to .gitignore manually? or do people need a fancy tool like BFG

[efiop]

@ammarasmro Great point! Created #3678 for that.

[nickdelgrosso]

I don't think this issue has been resolved. I noticed that this issue was happening even for a fresh dvc init repo with the most recent verision (version details below), but it turns out it's because I committed a file (a credentials file) to the .dvc/tmp directory. Once I moved it somewhere else, the issue stopped occuring. So my guess is that the .gitignore-filling step is just checking if the /tmp directory is mentioned at all that is being tracked, even if it's only a single exceptional file, and addin the directory to .gitignore if it sees tracking happening there.

DVC version: 1.2.0
Python version: 3.7.7
Platform: Windows-10-10.0.18362-SP0
Binary: False
Package: pip

[efiop]

Hi @nickdelgrosso ! Sorry for the delay. So dvc actually uses git check-ignore,

dvc/dvc/scm/git.py

Line 189 in 815444f

self.repo.git.check_ignore(path)

. Maybe you've force-commited a file in tmp? 🤔

[nickdelgrosso]

Yep, that's exactly what I did. My mistake--just wanted to share in case someone else did the same thing.

…

On Thu, Jul 30, 2020, 5:42 AM Ruslan Kuprieiev ***@***.***> wrote: Hi @nickdelgrosso <https://github.com/nickdelgrosso> ! Sorry for the delay. So dvc actually uses git check-ignore, https://github.com/iterative/dvc/blob/815444ffdc5f00e0ae031c3d72dd683a0f698cc8/dvc/scm/git.py#L189 . Maybe you've force-commited a file in tmp? 🤔 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3561 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA5AO57FI5B6PQIQQEPFLJDR6DT3XANCNFSM4LXO6HUA> .

[efiop]

There's not much we can do in that case, unfortunately :(

[nickdelgrosso]

Is there any way to detect that it's been happening and make sure that the .dvc .gitignore file doesn't contain duplicate entries? At least it could correct for the duplicate /tmp entry writing behavior from dvc.

…

On Thu, Jul 30, 2020 at 9:17 PM Ruslan Kuprieiev ***@***.***> wrote: There's not much we can do in that case, unfortunately :( — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3561 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA5AO52I5BNP7CLRX6KKNN3R6HBK5ANCNFSM4LXO6HUA> .

[efiop]

@nickdelgrosso Sure, we could read the whole gitignore and check for duplicates in

dvc/dvc/scm/git.py

Line 212 in ed1c3a7

with open(gitignore, "a+", encoding="utf-8") as fobj:

, though it might affect the perfomance a little bit (we do those gitignore checks on every dvc run), but that's likely negligible. It would probably be best to raise an error when we detect the duplicates, just to prevent weird bugs, but there might potentially be scenarios where the user really-really wants to git add some gitignored file. We could start by raising an error though, seems like a good approach. Or do you still need that file in tmp?

[nickdelgrosso]

All good points. Yes, raising an error would probably be best. I definitely didn't need that file in tmp, I just didn't realize that its presence was a problem.

…

On Fri, Jul 31, 2020 at 8:52 PM Ruslan Kuprieiev ***@***.***> wrote: @nickdelgrosso <https://github.com/nickdelgrosso> Sure, we could read the whole gitignore and check for duplicates in https://github.com/iterative/dvc/blob/ed1c3a7a15d6101622a6d4ac708d9c548d6e1fd5/dvc/scm/git.py#L212 , though it might affect the perfomance a little bit (we do those gitignore checks on every dvc run), but that's likely negligible. It would probably be best to raise an error when we detect the duplicates, just to prevent weird bugs, but there might potentially be scenarios where the user really-really wants to git add some gitignored file. We could start by raising an error though, seems like a good approach. Or do you still need that file in tmp? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3561 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA5AO555UVO2WUXGXMS4KO3R6MHHZANCNFSM4LXO6HUA> .

[efiop]

@nickdelgrosso Created #4327 . Thanks!