Permission denied with windows share accessed from a linux dvc client

[gillhofer]

Bug Report

Please provide information about your setup

Remote is set to a network drive on a windows share. Reading/Writing to this share is possible.

Output of dvc version:

$ dvc version

DVC version: 1.9.1 (pip)
---------------------------------
Platform: Python 3.6.8 on Linux-4.18.0-147.3.1.el8_1.x86_64-x86_64-with-centos-8.1.1911-Core
Supports: http, https
Cache types: reflink, hardlink, symlink
Repo: dvc, git

Additional Information (if any):

If applicable, please also provide a --verbose output of the command, eg: dvc add --verbose.

[user@localhost <directory>] $ dvc -v push
ERROR: failed to upload '.dvc/cache/93/ce4560ea6eeb07f6c5e5b45542512a' to '../../../../../mnt/networkdrives/<XXX>/DVC/dvc-remote/93/ce4560ea6eeb07f6c5e5b45542512a' - [Errno 1] Operation not permitted: '/mnt/networkdrives/<XXX>/DVC/dvc-remote/93/ce4560ea6eeb07f6c5e5b45542512a.eePk9P9PgCP958UeNydbLp.tmp'
ERROR: failed to upload '.dvc/cache/96/0c155eac0009d3b6dbc019a84f1b6c.dir' to '../../../../../mnt/networkdrives/<XXX>/DVC/dvc-remote/96/0c155eac0009d3b6dbc019a84f1b6c.dir'
ERROR: failed to upload '.dvc/cache/58/870323aea899a9a42397776edbced7' to '../../../../../mnt/networkdrives/<XXX>/DVC/dvc-remote/58/870323aea899a9a42397776edbced7' - [Errno 1] Operation not permitted: '/mnt/networkdrives/<XXX>/DVC/dvc-remote/58/870323aea899a9a42397776edbced7.mA8BsnQLEBGokvS3k7WuqD.tmp'
ERROR: failed to upload '.dvc/cache/ba/9a3ef332715fc48c8928ff317a835b' to '../../../../../mnt/networkdrives/<XXX>/DVC/dvc-remote/ba/9a3ef332715fc48c8928ff317a835b' - [Errno 1] Operation not permitted: '/mnt/networkdrives/<XXX>/DVC/dvc-remote/ba/9a3ef332715fc48c8928ff317a835b.DWsNop3is3D7M2Dnwzd34j.tmp'
ERROR: failed to push data to the cloud - 4 files failed to upload`

The files do appear in the remote directory

If the cache gets set onto the same windows fileserver the error is more verbose.

$ dvc add -v unittests/integration_test/testdata/
2020-10-29 07:44:56,040 DEBUG: Check for update is enabled.
2020-10-29 07:44:56,043 DEBUG: fetched: [(3,)]
2020-10-29 07:44:57,640 DEBUG: Adding 'unittests/integration_test/testdata' to 'unittests/integration_test/.gitignore'.
2020-10-29 07:44:57,641 DEBUG: Path '<XY>/unittests/integration_test/testdata' inode '403027213'
2020-10-29 07:44:57,642 DEBUG: fetched: []
2020-10-29 07:44:57,644 DEBUG: Path '<XY>/unittests/integration_test/testdata/testfile.slow' inode '403027257'
2020-10-29 07:44:57,644 DEBUG: fetched: []
2020-10-29 07:44:57,644 DEBUG: Path '<XY>/unittests/integration_test/testdata/testfile.fast' inode '403027259'
2020-10-29 07:44:57,645 DEBUG: fetched: []
2020-10-29 07:44:57,645 DEBUG: Path '<XY>/unittests/integration_test/testdata/.gitignore' inode '403027228'
2020-10-29 07:44:57,645 DEBUG: fetched: []
2020-10-29 07:44:59,373 DEBUG: Uploading '../../../../../tmp/tmp7txde11j' to '../../../../../mnt/networkdrives/<XXX>/DVC/dvc-cache/.59MZDNHC55i69CxyW2HX7b.tmp'
Adding...
2020-10-29 07:44:59,387 DEBUG: fetched: [(0,)]
2020-10-29 07:44:59,389 ERROR: unexpected error - [Errno 1] Operation not permitted: '/mnt/networkdrives/<XXX>/DVC/dvc-cache/.59MZDNHC55i69CxyW2HX7b.tmp.3QNBXQxKiMMXQ9Puv3DkWy.tmp'
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/dvc/main.py", line 76, in main
    ret = cmd.run()
  File "/usr/local/lib/python3.6/site-packages/dvc/command/add.py", line 22, in run
    external=self.args.external,
  File "/usr/local/lib/python3.6/site-packages/dvc/repo/__init__.py", line 54, in wrapper
    return f(repo, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/dvc/repo/scm_context.py", line 4, in run
    result = method(repo, *args, **kw)
  File "/usr/local/lib/python3.6/site-packages/dvc/repo/add.py", line 90, in add
    stage.save()
  File "/usr/local/lib/python3.6/site-packages/dvc/stage/__init__.py", line 386, in save
    self.save_outs(allow_missing=allow_missing)
  File "/usr/local/lib/python3.6/site-packages/dvc/stage/__init__.py", line 398, in save_outs
    out.save()
  File "/usr/local/lib/python3.6/site-packages/dvc/output/base.py", line 278, in save
    if not self.changed():
  File "/usr/local/lib/python3.6/site-packages/dvc/output/base.py", line 220, in changed
    status = self.status()
  File "/usr/local/lib/python3.6/site-packages/dvc/output/base.py", line 217, in status
    return self.workspace_status()
  File "/usr/local/lib/python3.6/site-packages/dvc/output/base.py", line 205, in workspace_status
    if self.changed_checksum():
  File "/usr/local/lib/python3.6/site-packages/dvc/output/base.py", line 191, in changed_checksum
    return self.hash_info != self.get_hash()
  File "/usr/local/lib/python3.6/site-packages/dvc/output/base.py", line 180, in get_hash
    return self.cache.get_hash(self.tree, self.path_info)
  File "/usr/local/lib/python3.6/site-packages/funcy/decorators.py", line 39, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/usr/local/lib/python3.6/site-packages/dvc/cache/base.py", line 50, in use_state
    return call()
  File "/usr/local/lib/python3.6/site-packages/funcy/decorators.py", line 60, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.6/site-packages/dvc/cache/base.py", line 709, in get_hash
    hash_info = tree.get_hash(path_info)
  File "/usr/local/lib/python3.6/site-packages/funcy/decorators.py", line 39, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/usr/local/lib/python3.6/site-packages/dvc/tree/base.py", line 45, in use_state
    return call()
  File "/usr/local/lib/python3.6/site-packages/funcy/decorators.py", line 60, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.6/site-packages/dvc/tree/base.py", line 271, in get_hash
    hash_info = self.get_dir_hash(path_info, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/funcy/decorators.py", line 39, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/usr/local/lib/python3.6/site-packages/dvc/tree/base.py", line 45, in use_state
    return call()
  File "/usr/local/lib/python3.6/site-packages/funcy/decorators.py", line 60, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.6/site-packages/dvc/tree/base.py", line 340, in get_dir_hash
    return self.repo.cache.local.save_dir_info(dir_info)
  File "/usr/local/lib/python3.6/site-packages/funcy/decorators.py", line 39, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/usr/local/lib/python3.6/site-packages/dvc/cache/base.py", line 50, in use_state
    return call()
  File "/usr/local/lib/python3.6/site-packages/funcy/decorators.py", line 60, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.6/site-packages/dvc/cache/base.py", line 301, in save_dir_info
    hi, tmp_info = self._get_dir_info_hash(dir_info)
  File "/usr/local/lib/python3.6/site-packages/dvc/cache/base.py", line 284, in _get_dir_info_hash
    self.tree.upload(from_info, to_info, no_progress_bar=True)
  File "/usr/local/lib/python3.6/site-packages/dvc/tree/base.py", line 360, in upload
    no_progress_bar=no_progress_bar,
  File "/usr/local/lib/python3.6/site-packages/dvc/tree/local.py", line 331, in _upload
    self.protect(tmp_file)
  File "/usr/local/lib/python3.6/site-packages/dvc/tree/local.py", line 291, in protect
    os.chmod(path, mode)
PermissionError: [Errno 1] Operation not permitted: '/mnt/networkdrives/<XXX>/DVC/dvc-cache/.59MZDNHC55i69CxyW2HX7b.tmp.3QNBXQxKiMMXQ9Puv3DkWy.tmp'
------------------------------------------------------------
2020-10-29 07:44:59,416 DEBUG: Version info for developers:
DVC version: 1.9.1 (pip)
---------------------------------
Platform: Python 3.6.8 on Linux-4.18.0-147.3.1.el8_1.x86_64-x86_64-with-centos-8.1.1911-Core
Supports: http, https
Cache types: symlink
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2020-10-29 07:44:59,418 DEBUG: Analytics is enabled.
2020-10-29 07:44:59,727 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpoqoq9g77']'
2020-10-29 07:44:59,728 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpoqoq9g77']'

Also in this case the files do appear on the remote filesystem.

[pmrowla]

I'm assuming this network share is mounted via SMB/CIFS?

You most likely need to adjust your fstab mount settings in order for chmod to work correctly. IIRC it will likely require either mounting the share with noperm or with uid and gid set to the user and group ID values for the user running DVC.

[gillhofer]

Thank you for your reply.

According to /etc/fstab it is a NFS mount.

source target nfs auto,nofail,noatime,nolock,intr,tcp,actimeo=1800 0 0

importantly, thes user running dvc is able to create files as well as chmod them.

sudo -H -u pipeline-user bash -c "touch test"
sudo -H -u pipeline-user bash -c "chmod u-x test"
-rw-rwxrwx 1       1024 users         0 <Date> test

[pared]

@gillhofer could you run
sudo -H -u pipeline-user bash -c "touch file && python -c \"import os;os.chmod('file',0o444)\"" on mounted NFS?

[gillhofer]

Yes this command works fine.

However, I noticed that that this issue is due to cached files beeing created by a different user (we use a shared cache).

I guess this is a NFS configuration issue now. Do you have hints on the necessary permissions?

[pared]

@gillhofer would it be possible to add both users to shared "cache-users" group?
DVC does support shared group access between users.
Here you can find more context about this use case.

[gillhofer]

Thank you, we could solve it by just using a CIFS mount! 👍

However, for some reason a dvc pull requires a md5 calculation of all the checked out files, despite the shared cache. Is there a way to prevent this?

[pared]

@gillhofer
I have not too much experience with CIFS, so I might be asking stupid question, but are the files in remote read only? Recalculation of md5's usually means that files in remote are not read only.

[gillhofer]

@pared They are not read only. They are RW. Is the cache supposed to be read-only? I might have missed that in your documentation.

Is DVC checking whether the file could be modified to prevent cache-corruption? How would it detect that a file was modified with another user which has RW access?

[pared]

@gillhofer,

Is the cache supposed to be read-only

it does not have to be, but we have a mechanism in place that makes simple verification if your remote has not been meddled with (in earlier versions we used to set hardlink cache by default resulting in users editing their caches/remotes). When putting data into remote, we chmod it to be read only. Later on, when we pull the data, we assume that it has not been changed if its still read only. If it is not, we need to recalculate the checksum to make sure the file has same md5 as it is in dvc.lock file. And that's what I suppose might be causing your recalculations.

I think that CIFS might not handle part of code responsible for this "protection".

Could you try adding some piece of data with -v option? Or try to run python -c "import os;os.chmod('{data}',0o444)" on some file on this share?

EDIT:

Is DVC checking whether the file could be modified to prevent cache-corruption?

Yes

How would it detect that a file was modified with another user which has RW access?

It cannot without recalculation.

[gillhofer]

@pared
Ok, that explains the behaviour.

However, with this setup there seems to be a general issue when CIFS mounts are used as cache, as CIFS does not deal at all with file permissions. From the CIFS Documentation.

The core CIFS protocol does not provide unix ownership information or mode for files and directories. Because of this, files and directories will generally appear to be owned by whatever values the uid= or gid= options are set, and will have permissions set to the default file_mode and dir_mode for the mount. Attempting to change these values via chmod/chown will return success but have no effect.

This explains why your code runs fine, but has no effect on the file permissions.

If I understand this right, than either some config options is necessary to prevent md5 calculation on CIFS mounts or using something other than CIFS for mounting.

If this is right, than CIFS is not really practical in our case for having access to a shared cache, which brings us back to the start.

[pared]

@gillhofer
How about getting back to NFS and trying to set up group?

[shcheklein]

Recalculation of md5's usually means that files in remote are not read only.

hmm ... we should probably update docs to mention that we don't support CIFS for DVC for now? I wonder how common/popular CIFS is.

[gillhofer]

@pared

@gillhofer
How about getting back to NFS and trying to set up group?

We will have to do that eventually. But I guess this will be a more prolonged endeavour as we need to change parts of our grown infrastructure. In the meantime we need to wait out md5 calculations and understand that not writing to remote and cache is crucial.

[pared]

@shcheklein I would not say that we don't support CIFS. Only thing that is not working is a optimization we implemented in #3472, but it does not prevent DVC from working properly. It just makes it work slower.

@gillhofer That raises new question: whether we should have some kind of mechanism allowing user to say "I know what I am doing, trust the remote/cache even if its unprotected". It sounds like it could be useful, in some scenarios, especially CI, where you not necessarily edit anything, but rather check out and run.

[gillhofer]

@pared

would it be possible to add both users to shared "cache-users" group?
DVC does support shared group access between users.

Maybe I did not understand everything involved completely, so please bare with me.

Is there a way around a shared "cache-users" group? We have the problem that the cache is written to by LDAP users, and the pipeline-user is a non-LDAP user.
This either requires to

• make the pipeline-user a LDAP user
• modify the LDAP database to fake the group membership of the pipeline-user
• something different

[pared]

@gillhofer
I am not an expert on LDAP, and don't know how your set up looks like. I would need some more information on that to be able to help. Are LDAP users treated differently on windows share than other users?

[pared]

Closing as stale, @gillhofer feel free to reopen if the problem still occurs.