Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

git: failed to check if path is gitignored #6152

Closed
gbmarc1 opened this issue Jun 10, 2021 · 12 comments
Closed

git: failed to check if path is gitignored #6152

gbmarc1 opened this issue Jun 10, 2021 · 12 comments
Labels
awaiting response we are waiting for your reply, please respond! :) bug Did we break something?

Comments

@gbmarc1
Copy link
Contributor

gbmarc1 commented Jun 10, 2021

Bug Report

Description

Repro fails at then end when calculating a hash for an output for both dvc repro and dvc exp. I am not even sure the dvc.lock is updated appropriately.

Reproduce

dvc repro train
stages:
...
train:
    always_changed: true
    cmd: export PYTHONPATH=$(pwd)/src:\${PYTHONPATH} && ${systemExecutable} -m ${projectName}.cli.main train-config
    deps: [data/processed/dataset.pkl] 
    metrics:
       - metrics/eval_raw_summary.json: {cache: false} 
       - metrics/eval_category_summary.json: {cache: false}
    outs: ['src/${projectName}/artifacts']
params: [params.aggr, params.alpha, params.batch_size, params.beta, params.cuda, params.dropout, params.epochs, params.eta, params.gamma, params.language_model, params.learning_rate, params.log_level, params.lr_decay, params.min_freq, params.min_freq_type, params.model, params.num_heads, params.pw_scalar, params.weight_decay]

Error stack Trace:

Traceback (most recent call last):
File "venv/lib/python3.8/site-packages/dvc/repo/reproduce.py", line 196, in _reproduce_stages
    ret = _reproduce_stage(stage, **kwargs)
File "venv/lib/python3.8/site-packages/dvc/repo/reproduce.py", line 39, in _reproduce_stage
    stage = stage.reproduce(**kwargs)
File "venv/lib/python3.8/site-packages/funcy/decorators.py", line 39, in wrapper
    return deco(call, *dargs, **dkwargs)
File "venv/lib/python3.8/site-packages/dvc/stage/decorators.py", line 36, in rwlocked
    return call()
File "venv/lib/python3.8/site-packages/funcy/decorators.py", line 60, in __call__
    return self._func(*self._args, **self._kwargs)
File "venv/lib/python3.8/site-packages/dvc/stage/__init__.py", line 410, in reproduce
    self.run(**kwargs)
File "venv/lib/python3.8/site-packages/funcy/decorators.py", line 39, in wrapper
    return deco(call, *dargs, **dkwargs)
File "venv/lib/python3.8/site-packages/dvc/stage/decorators.py", line 36, in rwlocked
    return call()
File "venv/lib/python3.8/site-packages/funcy/decorators.py", line 60, in __call__
    return self._func(*self._args, **self._kwargs)
File "venv/lib/python3.8/site-packages/dvc/stage/__init__.py", line 529, in run
    self.save(allow_missing=allow_missing)
File "venv/lib/python3.8/site-packages/dvc/stage/__init__.py", line 440, in save
    self.save_outs(allow_missing=allow_missing)
File "venv/lib/python3.8/site-packages/dvc/stage/__init__.py", line 460, in save_outs
    out.save()
File "venv/lib/python3.8/site-packages/dvc/output.py", line 520, in save
    self.ignore()
File "venv/lib/python3.8/site-packages/dvc/output.py", line 497, in ignore
    if self.repo.scm.is_tracked(self.fspath):
File "venv/lib/python3.8/site-packages/dvc/scm/git/__init__.py", line 343, in _backend_func
    return func(*args, **kwargs)
File "venv/lib/python3.8/site-packages/dvc/scm/git/backend/dulwich.py", line 229, in is_tracked
    for path in self.repo.open_index():
File "venv/lib/python3.8/site-packages/dulwich/repo.py", line 1248, in open_index
    return Index(self.index_path())
File "venv/lib/python3.8/site-packages/dulwich/index.py", line 319, in __init__
    self.read()
File "venv/lib/python3.8/site-packages/dulwich/index.py", line 344, in read
    for name, entry in read_index(f):
File "venv/lib/python3.8/site-packages/dulwich/index.py", line 241, in read_index
    yield read_cache_entry(f, version)
File "venv/lib/python3.8/site-packages/dulwich/index.py", line 167, in read_cache_entry
    ) = struct.unpack(">LLLLLL20sH", f.read(20 + 4 * 6 + 2))
    struct.error: unpack requires a buffer of 46 bytes
The above exception was the direct cause of the following exception:

ls -al src/${projectName}/artifacts :

drwxrwxr-x  2 user user      4096 Jun 10 12:59 .
drwxrwxr-x 10 user user      4096 Jun 10 11:23 ..
-rw-rw-r--  1 user user       400 Jun 10 12:59 cli.pkl
-rw-rw-r--  1 user user   1648945 Jun 10 12:59 kwargs.pkl
-rw-rw-r--  1 user user    204906 Jun 10 12:59 labels.pkl
-rw-rw-r--  1 user user       399 Jun 10 12:59 mask_field.pkl
-rw-rw-r--  1 user user   1568971 Jun 10 12:59 model_info.json
-rw-rw-r--  1 user user 292264571 Jun 10 12:57 model.pkl
-rw-rw-r--  1 user user     24458 Jun 10 12:59 perf_category.pkl
-rw-rw-r--  1 user user    257734 Jun 10 12:59 perf_raw.pkl
-rw-rw-r--  1 user user       388 Jun 10 12:59 target_field.pkl
-rw-rw-r--  1 user user    591247 Jun 10 12:59 text_field.pkl
-rw-rw-r--  1 user user       382 Jun 10 12:59 weight_field.pkl

cat dvc.lock:

schmea: 2.0

stages:
...
  train:
    cmd: export PYTHONPATH=$(pwd)/src:${PYTHONPATH} && /home/marc/workspace/elc/venv/bin/python3 -m elc.cli.main train-config
    deps:
    - path: data/processed/dataset.pkl
      md5: a4e9ce2716a8aa3fe3c360c4c5409d4e
      size: 9654023523
    params:
      params.yaml:
        params.aggr: mean
        params.alpha: false
        params.batch_size: 32
        params.beta: 0.5
        params.cuda: '0'
        params.dropout: 0.29446913344386494
        params.epochs: 50
        params.eta: 0.0
        params.gamma: 1.9999999999999996
        params.language_model: distilbert-base-cased
        params.learning_rate: 9.999999999999999e-06
        params.log_level: info
        params.lr_decay: 0.9999999999999994
        params.min_freq: 50
        params.min_freq_type: target-path
        params.model: reference-single-task-binary
        params.num_heads: 2
        params.pw_scalar: -1.0
        params.weight_decay: 8.807265731136809e-07
    outs:
    - path: metrics/eval_category_summary.json
      md5: c0bbf36f55aa6351c49d4c881e95c87b
      size: 95
    - path: metrics/eval_raw_summary.json
      md5: 484166304b6df6124b98b1f71c35e0ad
      size: 120
    - path: src/elc/artifacts
      md5: 30357b1fa01d666b493c0769df243eab.dir
      size: 296562379
      nfiles: 11

Expected

DVC should not fail and update the dvc.lock file.

Environment information

Output of dvc doctor:

$ dvc doctor
DVC version: 2.3.0 (pip)                                                                                                                                                   ---------------------------------
Platform: Python 3.8.5 on Linux-5.4.0-74-generic-x86_64-with-glibc2.10
Supports: azure, http, https
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/mapper/isw_eeibgicei_Volume1
Caches: local
Remotes: azure
Workspace directory: ext4 on /dev/mapper/isw_eeibgicei_Volume1
Repo: dvc, git

Additional Information (if any):

@efiop
Copy link
Contributor

efiop commented Jun 10, 2021

@gbmarc1 Hi. Could you show full dvc doctor output, please? Looks like part of it didn't get pasted

@efiop efiop added the awaiting response we are waiting for your reply, please respond! :) label Jun 10, 2021
@gbmarc1
Copy link
Contributor Author

gbmarc1 commented Jun 10, 2021

@gbmarc1 Hi. Could you show full dvc doctor output, please? Looks like part of it didn't get pasted

Its already there. Last code block

@efiop
Copy link
Contributor

efiop commented Jun 10, 2021

@gbmarc1 It is not full, the version is missing. E.g.

$ dvc doctor                                                          
DVC version: 2.3.0+91ab72                                             
---------------------------------                                     
Platform: Python 3.8.3 on Linux-5.8.0-55-generic-x86_64-with-glibc2.29
Supports: azure, gdrive, gs, hdfs, webhdfs, http, https, s3, ssh, oss 
Cache types: hardlink, symlink                                        
Cache directory: ext4 on /dev/nvme0n1p2                               
Caches: local                                                         
Remotes: https, gdrive, https                                         
Workspace directory: ext4 on /dev/nvme0n1p2                           
Repo: dvc, git                                                        

@gbmarc1
Copy link
Contributor Author

gbmarc1 commented Jun 10, 2021

@gbmarc1 It is not full, the version is missing. E.g.

$ dvc doctor                                                          
DVC version: 2.3.0+91ab72                                             
---------------------------------                                     
Platform: Python 3.8.3 on Linux-5.8.0-55-generic-x86_64-with-glibc2.29
Supports: azure, gdrive, gs, hdfs, webhdfs, http, https, s3, ssh, oss 
Cache types: hardlink, symlink                                        
Cache directory: ext4 on /dev/nvme0n1p2                               
Caches: local                                                         
Remotes: https, gdrive, https                                         
Workspace directory: ext4 on /dev/nvme0n1p2                           
Repo: dvc, git                                                        

I modified the code block

@efiop
Copy link
Contributor

efiop commented Jun 10, 2021

@gbmarc1 Thanks! Could you also show pip freeze | grep dulwich, please?

To clarify, the error happens when we try to check if a certain path is gitignored already or not. Not related to big fiels or hashes, really.

@efiop efiop changed the title Hash fails for large files git: failed to check if path is gitignored Jun 10, 2021
@efiop efiop added the bug Did we break something? label Jun 10, 2021
@gbmarc1
Copy link
Contributor Author

gbmarc1 commented Jun 11, 2021

dulwich==0.20.23
Note that I have other projects that use the same underlying mechanics and everything is fine. The only difference with this project is that the out binaries are bigger. Looks like some limit has been trespassed and it fails. I also have reinstalled the environement multiple times and it yields to the same result everytime

@pmrowla
Copy link
Contributor

pmrowla commented Jun 11, 2021

@gbmarc1 are you using git worktrees or sparse checkouts? (or any other less commonly used/advanced git feature)

(the error is in dulwich when it tries to load the git index, this usually means it's an unsupported git index version)

@gbmarc1
Copy link
Contributor Author

gbmarc1 commented Jun 11, 2021

🤦‍♂️I simply add a faulty index
rm -f .git/index did the trick
Sorry and thanks for the help!

@gbmarc1 gbmarc1 closed this as completed Jun 11, 2021
@efiop
Copy link
Contributor

efiop commented Jun 11, 2021

@gbmarc1 Were you able to confirm with other tools that it was faulty? Maybe git cli was failing too?

@agrizzli
Copy link

I have a very similar issue. It happens as follow up issue after the problem described in #7186 . Index somehow gets broken. And it is unclear why, because a very similar workflow was working fine in the same settings. git cli then fails in return git status, but works on some other commands like git branch. E.g. git status results in:

fatal: unknown index entry format 0x31310000

Although rm -rf .git/index + git reset do the trick, the same error happens again when re-running the experiment. In my case, however -- similarly to #7168 --, dvc repro does not produce the initial error, but dvc exp run does.

2022-08-12 19:20:38,184 ERROR: unexpected error - unpack requires a buffer of 46 bytes                                                                                                                     
------------------------------------------------------------                                                                                                                                               
Traceback (most recent call last):                                                                                                                                                                         
  File "/home/arthurmueller/sparta-analysis/venv/lib/python3.8/site-packages/dvc/cli/__init__.py", line 185, in main                                                                                       
    ret = cmd.do_run()                                                                                                                                                                                     
  File "/home/arthurmueller/sparta-analysis/venv/lib/python3.8/site-packages/dvc/cli/command.py", line 22, in do_run                                                                                       
    return self.run()                                                                                                                                                                                      
  File "/home/arthurmueller/sparta-analysis/venv/lib/python3.8/site-packages/dvc/commands/experiments/run.py", line 32, in run                                                                             
    results = self.repo.experiments.run(                                                                                                                                                                   
  File "/home/arthurmueller/sparta-analysis/venv/lib/python3.8/site-packages/dvc/repo/experiments/__init__.py", line 519, in run                                                                           
    return run(self.repo, *args, **kwargs)                                                                                                                                                                 
  File "/home/arthurmueller/sparta-analysis/venv/lib/python3.8/site-packages/dvc/repo/__init__.py", line 48, in wrapper                                                                                    
    return f(repo, *args, **kwargs)                                                                                                                                                                        
  File "/home/arthurmueller/sparta-analysis/venv/lib/python3.8/site-packages/dvc/repo/experiments/run.py", line 49, in run                                                                                 
    return repo.experiments.reproduce_one(                                                                                                                                                                 
  File "/home/arthurmueller/sparta-analysis/venv/lib/python3.8/site-packages/dvc/repo/experiments/__init__.py", line 108, in reproduce_one                                                                 
    staged, _, _ = self.scm.status(untracked_files="no")                                                                                                                                                   
  File "/home/arthurmueller/sparta-analysis/venv/lib/python3.8/site-packages/scmrepo/git/__init__.py", line 289, in _backend_func                                                                          
    result = func(*args, **kwargs)                                                                                                                                                                         
  File "/home/arthurmueller/sparta-analysis/venv/lib/python3.8/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 769, in status                                                                 
    staged, unstaged, untracked = git_status(                                                                                                                                                              
  File "/home/arthurmueller/sparta-analysis/venv/lib/python3.8/site-packages/dulwich/porcelain.py", line 1226, in status                                                                                   
    tracked_changes = get_tree_changes(r)                                                                                                                                                                  
  File "/home/arthurmueller/sparta-analysis/venv/lib/python3.8/site-packages/dulwich/porcelain.py", line 1344, in get_tree_changes                                                                         
    index = r.open_index()                                                                                                                                                                                 
  File "/home/arthurmueller/sparta-analysis/venv/lib/python3.8/site-packages/dulwich/repo.py", line 1280, in open_index                                                                                    
    return Index(self.index_path())                                                                                                                                                                        
  File "/home/arthurmueller/sparta-analysis/venv/lib/python3.8/site-packages/dulwich/index.py", line 319, in __init__                                                                                      
    self.read()                                                                                                                                                                                            
  File "/home/arthurmueller/sparta-analysis/venv/lib/python3.8/site-packages/dulwich/index.py", line 344, in read                                                                                          
    for name, entry in read_index(f):                                                                                                                                                                      
  File "/home/arthurmueller/sparta-analysis/venv/lib/python3.8/site-packages/dulwich/index.py", line 241, in read_index                                                                                    
    yield read_cache_entry(f, version)                                                                                                                                                                     
  File "/home/arthurmueller/sparta-analysis/venv/lib/python3.8/site-packages/dulwich/index.py", line 167, in read_cache_entry                                                                              
    ) = struct.unpack(">LLLLLL20sH", f.read(20 + 4 * 6 + 2))                                                                                                                                               
struct.error: unpack requires a buffer of 46 bytes 
------------------------------------------------------------                                                                                                                                               
2022-08-12 19:20:38,470 DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out)                                                                                          
2022-08-12 19:20:38,470 DEBUG: Removing '/home/arthurmueller/.nVw4GMyorsgKmZKoTjpunD.tmp'                                                                                                                  
2022-08-12 19:20:38,470 DEBUG: Removing '/home/arthurmueller/.nVw4GMyorsgKmZKoTjpunD.tmp'                                                                                                                  
2022-08-12 19:20:38,470 DEBUG: Removing '/home/arthurmueller/.nVw4GMyorsgKmZKoTjpunD.tmp'                                                                                                                  
2022-08-12 19:20:38,470 DEBUG: Removing '/home/arthurmueller/sparta-analysis/.dvc/cache/.W8kkPjHjerHDbw8Q9ov8Lj.tmp'                                                                                       
2022-08-12 19:20:38,490 DEBUG: Version info for developers:
DVC version: 2.17.0 (pip)
---------------------------------
Platform: Python 3.8.10 on Linux-5.4.0-121-generic-x86_64-with-glibc2.29
Supports:
        hdfs (fsspec = 2022.7.1, pyarrow = 9.0.0), 
        webhdfs (fsspec = 2022.7.1),
        http (aiohttp = 3.8.1, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.1, aiohttp-retry = 2.8.3),
        ssh (sshfs = 2022.6.0)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/mapper/ubuntu--vg-ubuntu--lv
Caches: local
Remotes: local
Workspace directory: ext4 on /dev/mapper/ubuntu--vg-ubuntu--lv
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2022-08-12 19:20:38,491 DEBUG: Analytics is enabled.
2022-08-12 19:20:38,533 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmp1lapuh2f']'
2022-08-12 19:20:38,534 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmp1lapuh2f']'

DVC doctor

DVC version: 2.17.0 (pip)
---------------------------------
Platform: Python 3.8.10 on Linux-5.4.0-121-generic-x86_64-with-glibc2.29
Supports:
        hdfs (fsspec = 2022.7.1, pyarrow = 9.0.0),
        webhdfs (fsspec = 2022.7.1),
        http (aiohttp = 3.8.1, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.1, aiohttp-retry = 2.8.3),
        ssh (sshfs = 2022.6.0)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/mapper/ubuntu--vg-ubuntu--lv
Caches: local
Remotes: local
Workspace directory: ext4 on /dev/mapper/ubuntu--vg-ubuntu--lv
Repo: dvc, git

pip freeze | grep dulwich --> dulwich==0.20.45

@pmrowla
Copy link
Contributor

pmrowla commented Aug 15, 2022

@agrizzli are you using git worktrees or sparse checkout? Could you please run git --version and post the output

@agrizzli
Copy link

agrizzli commented Aug 15, 2022

git --version is git version 2.25.1

I am not using neither git worktrees not sparse checkout.

However I figured out the true reason behind my error, which seems to be different to #7186. I was using some old dependency that has not received new outputs anymore.

First, I was using for some repro runs:

stage1:
    ...
    outs:
     - path/to/store/big/model/
         cache: false

stage2:
    ...
    deps:
    - path/to/store/big/model/

And later I removed the mentioning of the first outs, but forgot to remove the corresponding deps part. Thus, DVC was probably trying to get a necessary dependency, but getting only some old reference. After removing the part in deps this error disappeared.

The link to gbmarc1 is that I also used big binaries, but is was not the true reason, I guess. However, corresponding file is neither listed in any .gitignore file.

The traceback, which is only similar to #7186 in the final error message is the following: traceback.txt. The failure happens after the stage named pipelines/absa_single/dvc.yaml:eval_on_datasets, which means stage2 in the above snippet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
awaiting response we are waiting for your reply, please respond! :) bug Did we break something?
Projects
None yet
Development

No branches or pull requests

4 participants