Permission denied with windows share accessed from a linux dvc client

[gillhofer]

Bug Report

Please provide information about your setup

Remote is set to a network drive on a windows share. Reading/Writing to this share is possible.

Output of dvc version:

$ dvc version

DVC version: 1.9.1 (pip)
---------------------------------
Platform: Python 3.6.8 on Linux-4.18.0-147.3.1.el8_1.x86_64-x86_64-with-centos-8.1.1911-Core
Supports: http, https
Cache types: reflink, hardlink, symlink
Repo: dvc, git

Additional Information (if any):

If applicable, please also provide a --verbose output of the command, eg: dvc add --verbose.

[user@localhost <directory>] $ dvc -v push
ERROR: failed to upload '.dvc/cache/93/ce4560ea6eeb07f6c5e5b45542512a' to '../../../../../mnt/networkdrives/<XXX>/DVC/dvc-remote/93/ce4560ea6eeb07f6c5e5b45542512a' - [Errno 1] Operation not permitted: '/mnt/networkdrives/<XXX>/DVC/dvc-remote/93/ce4560ea6eeb07f6c5e5b45542512a.eePk9P9PgCP958UeNydbLp.tmp'
ERROR: failed to upload '.dvc/cache/96/0c155eac0009d3b6dbc019a84f1b6c.dir' to '../../../../../mnt/networkdrives/<XXX>/DVC/dvc-remote/96/0c155eac0009d3b6dbc019a84f1b6c.dir'
ERROR: failed to upload '.dvc/cache/58/870323aea899a9a42397776edbced7' to '../../../../../mnt/networkdrives/<XXX>/DVC/dvc-remote/58/870323aea899a9a42397776edbced7' - [Errno 1] Operation not permitted: '/mnt/networkdrives/<XXX>/DVC/dvc-remote/58/870323aea899a9a42397776edbced7.mA8BsnQLEBGokvS3k7WuqD.tmp'
ERROR: failed to upload '.dvc/cache/ba/9a3ef332715fc48c8928ff317a835b' to '../../../../../mnt/networkdrives/<XXX>/DVC/dvc-remote/ba/9a3ef332715fc48c8928ff317a835b' - [Errno 1] Operation not permitted: '/mnt/networkdrives/<XXX>/DVC/dvc-remote/ba/9a3ef332715fc48c8928ff317a835b.DWsNop3is3D7M2Dnwzd34j.tmp'
ERROR: failed to push data to the cloud - 4 files failed to upload`

The files do appear in the remote directory

If the cache gets set onto the same windows fileserver the error is more verbose.

$ dvc add -v unittests/integration_test/testdata/
2020-10-29 07:44:56,040 DEBUG: Check for update is enabled.
2020-10-29 07:44:56,043 DEBUG: fetched: [(3,)]
2020-10-29 07:44:57,640 DEBUG: Adding 'unittests/integration_test/testdata' to 'unittests/integration_test/.gitignore'.
2020-10-29 07:44:57,641 DEBUG: Path '<XY>/unittests/integration_test/testdata' inode '403027213'
2020-10-29 07:44:57,642 DEBUG: fetched: []
2020-10-29 07:44:57,644 DEBUG: Path '<XY>/unittests/integration_test/testdata/testfile.slow' inode '403027257'
2020-10-29 07:44:57,644 DEBUG: fetched: []
2020-10-29 07:44:57,644 DEBUG: Path '<XY>/unittests/integration_test/testdata/testfile.fast' inode '403027259'
2020-10-29 07:44:57,645 DEBUG: fetched: []
2020-10-29 07:44:57,645 DEBUG: Path '<XY>/unittests/integration_test/testdata/.gitignore' inode '403027228'
2020-10-29 07:44:57,645 DEBUG: fetched: []
2020-10-29 07:44:59,373 DEBUG: Uploading '../../../../../tmp/tmp7txde11j' to '../../../../../mnt/networkdrives/<XXX>/DVC/dvc-cache/.59MZDNHC55i69CxyW2HX7b.tmp'
Adding...
2020-10-29 07:44:59,387 DEBUG: fetched: [(0,)]
2020-10-29 07:44:59,389 ERROR: unexpected error - [Errno 1] Operation not permitted: '/mnt/networkdrives/<XXX>/DVC/dvc-cache/.59MZDNHC55i69CxyW2HX7b.tmp.3QNBXQxKiMMXQ9Puv3DkWy.tmp'
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/dvc/main.py", line 76, in main
    ret = cmd.run()
  File "/usr/local/lib/python3.6/site-packages/dvc/command/add.py", line 22, in run
    external=self.args.external,
  File "/usr/local/lib/python3.6/site-packages/dvc/repo/__init__.py", line 54, in wrapper
    return f(repo, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/dvc/repo/scm_context.py", line 4, in run
    result = method(repo, *args, **kw)
  File "/usr/local/lib/python3.6/site-packages/dvc/repo/add.py", line 90, in add
    stage.save()
  File "/usr/local/lib/python3.6/site-packages/dvc/stage/__init__.py", line 386, in save
    self.save_outs(allow_missing=allow_missing)
  File "/usr/local/lib/python3.6/site-packages/dvc/stage/__init__.py", line 398, in save_outs
    out.save()
  File "/usr/local/lib/python3.6/site-packages/dvc/output/base.py", line 278, in save
    if not self.changed():
  File "/usr/local/lib/python3.6/site-packages/dvc/output/base.py", line 220, in changed
    status = self.status()
  File "/usr/local/lib/python3.6/site-packages/dvc/output/base.py", line 217, in status
    return self.workspace_status()
  File "/usr/local/lib/python3.6/site-packages/dvc/output/base.py", line 205, in workspace_status
    if self.changed_checksum():
  File "/usr/local/lib/python3.6/site-packages/dvc/output/base.py", line 191, in changed_checksum
    return self.hash_info != self.get_hash()
  File "/usr/local/lib/python3.6/site-packages/dvc/output/base.py", line 180, in get_hash
    return self.cache.get_hash(self.tree, self.path_info)
  File "/usr/local/lib/python3.6/site-packages/funcy/decorators.py", line 39, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/usr/local/lib/python3.6/site-packages/dvc/cache/base.py", line 50, in use_state
    return call()
  File "/usr/local/lib/python3.6/site-packages/funcy/decorators.py", line 60, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.6/site-packages/dvc/cache/base.py", line 709, in get_hash
    hash_info = tree.get_hash(path_info)
  File "/usr/local/lib/python3.6/site-packages/funcy/decorators.py", line 39, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/usr/local/lib/python3.6/site-packages/dvc/tree/base.py", line 45, in use_state
    return call()
  File "/usr/local/lib/python3.6/site-packages/funcy/decorators.py", line 60, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.6/site-packages/dvc/tree/base.py", line 271, in get_hash
    hash_info = self.get_dir_hash(path_info, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/funcy/decorators.py", line 39, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/usr/local/lib/python3.6/site-packages/dvc/tree/base.py", line 45, in use_state
    return call()
  File "/usr/local/lib/python3.6/site-packages/funcy/decorators.py", line 60, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.6/site-packages/dvc/tree/base.py", line 340, in get_dir_hash
    return self.repo.cache.local.save_dir_info(dir_info)
  File "/usr/local/lib/python3.6/site-packages/funcy/decorators.py", line 39, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/usr/local/lib/python3.6/site-packages/dvc/cache/base.py", line 50, in use_state
    return call()
  File "/usr/local/lib/python3.6/site-packages/funcy/decorators.py", line 60, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.6/site-packages/dvc/cache/base.py", line 301, in save_dir_info
    hi, tmp_info = self._get_dir_info_hash(dir_info)
  File "/usr/local/lib/python3.6/site-packages/dvc/cache/base.py", line 284, in _get_dir_info_hash
    self.tree.upload(from_info, to_info, no_progress_bar=True)
  File "/usr/local/lib/python3.6/site-packages/dvc/tree/base.py", line 360, in upload
    no_progress_bar=no_progress_bar,
  File "/usr/local/lib/python3.6/site-packages/dvc/tree/local.py", line 331, in _upload
    self.protect(tmp_file)
  File "/usr/local/lib/python3.6/site-packages/dvc/tree/local.py", line 291, in protect
    os.chmod(path, mode)
PermissionError: [Errno 1] Operation not permitted: '/mnt/networkdrives/<XXX>/DVC/dvc-cache/.59MZDNHC55i69CxyW2HX7b.tmp.3QNBXQxKiMMXQ9Puv3DkWy.tmp'
------------------------------------------------------------
2020-10-29 07:44:59,416 DEBUG: Version info for developers:
DVC version: 1.9.1 (pip)
---------------------------------
Platform: Python 3.6.8 on Linux-4.18.0-147.3.1.el8_1.x86_64-x86_64-with-centos-8.1.1911-Core
Supports: http, https
Cache types: symlink
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2020-10-29 07:44:59,418 DEBUG: Analytics is enabled.
2020-10-29 07:44:59,727 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpoqoq9g77']'
2020-10-29 07:44:59,728 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpoqoq9g77']'

Also in this case the files do appear on the remote filesystem.

[pmrowla]

I'm assuming this network share is mounted via SMB/CIFS?

You most likely need to adjust your fstab mount settings in order for chmod to work correctly. IIRC it will likely require either mounting the share with noperm or with uid and gid set to the user and group ID values for the user running DVC.

[gillhofer]

Thank you for your reply.

According to /etc/fstab it is a NFS mount.

source target nfs auto,nofail,noatime,nolock,intr,tcp,actimeo=1800 0 0

importantly, thes user running dvc is able to create files as well as chmod them.

sudo -H -u pipeline-user bash -c "touch test"
sudo -H -u pipeline-user bash -c "chmod u-x test"
-rw-rwxrwx 1       1024 users         0 <Date> test

[pared]

@gillhofer could you run
sudo -H -u pipeline-user bash -c "touch file && python -c \"import os;os.chmod('file',0o444)\"" on mounted NFS?

[gillhofer]

Yes this command works fine.

However, I noticed that that this issue is due to cached files beeing created by a different user (we use a shared cache).

I guess this is a NFS configuration issue now. Do you have hints on the necessary permissions?

[pared]

@gillhofer would it be possible to add both users to shared "cache-users" group?
DVC does support shared group access between users.
Here you can find more context about this use case.

[gillhofer]

Thank you, we could solve it by just using a CIFS mount! 👍

However, for some reason a dvc pull requires a md5 calculation of all the checked out files, despite the shared cache. Is there a way to prevent this?

[pared]

@gillhofer
I have not too much experience with CIFS, so I might be asking stupid question, but are the files in remote read only? Recalculation of md5's usually means that files in remote are not read only.

[gillhofer]

@pared They are not read only. They are RW. Is the cache supposed to be read-only? I might have missed that in your documentation.

Is DVC checking whether the file could be modified to prevent cache-corruption? How would it detect that a file was modified with another user which has RW access?

[pared]

@gillhofer,

Is the cache supposed to be read-only

it does not have to be, but we have a mechanism in place that makes simple verification if your remote has not been meddled with (in earlier versions we used to set hardlink cache by default resulting in users editing their caches/remotes). When putting data into remote, we chmod it to be read only. Later on, when we pull the data, we assume that it has not been changed if its still read only. If it is not, we need to recalculate the checksum to make sure the file has same md5 as it is in dvc.lock file. And that's what I suppose might be causing your recalculations.

I think that CIFS might not handle part of code responsible for this "protection".

Could you try adding some piece of data with -v option? Or try to run python -c "import os;os.chmod('{data}',0o444)" on some file on this share?

EDIT:

Is DVC checking whether the file could be modified to prevent cache-corruption?

Yes

How would it detect that a file was modified with another user which has RW access?

It cannot without recalculation.