using ssh as a remote, unexpected error - Channel closed

[aabramovrepo]

Hi there,

I'm facing the following issue while using ssh as a remote for storing data on a disk station manager by Synology. Here is the error I'm getting after dvc push:

ERROR: unexpected error - Channel closed.

The same remote configuration works just fine when using a standard Desktop machine with Ubuntu 18 as a server. The DVC remote was created as follows:

dvc remote add remote-ssh ssh://<user name>@<ip address>:<path on the server>

dvc config core.remote remote-ssh

dvc remote modify remote-ssh ask_password true

Does anybody have an idea what needs to be changed in the ssh config?

Thanks in advance for any advice!

Best,
Alexey

[aabramovrepo]

Here is SFTP subsystem configuration from /etc/ssh/sshd_config:

Subsystem sftp internal-sftp -f DAEMON -u 000

[efiop]

Hi @aabramovrepo !

Could you show full log for dvc push -v, please? Also, are you able to ssh into that machine yourself?

[efiop]

@aabramovrepo Also, are you able to sftp into that machine from CLI?

[aabramovrepo]

Hi, thanks for the quick response!

Could you show full log for dvc push -v, please? Also, are you able to ssh into that machine yourself?

DEBUG: Establishing ssh connection with '<ip address>' through port '22' as user '<user name>'              
DEBUG: Establishing ssh connection with '<ip address>' through port '22' as user '<user name>'              
DEBUG: Establishing ssh connection with '<ip address>' through port '22' as user '<user name>'              
DEBUG: Establishing ssh connection with '<ip address>' through port '22' as user <user name>'              
DEBUG: SELECT count from state_info WHERE rowid=?                                                       
DEBUG: fetched: [(15275,)]
DEBUG: UPDATE state_info SET count = ? WHERE rowid = ?
ERROR: unexpected error - Channel closed.
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/dvc/main.py", line 50, in main
    ret = cmd.run()
  File "/usr/local/lib/python3.6/dist-packages/dvc/command/data_sync.py", line 51, in run
    recursive=self.args.recursive,
  File "/usr/local/lib/python3.6/dist-packages/dvc/repo/__init__.py", line 35, in wrapper
    ret = f(repo, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/dvc/repo/push.py", line 27, in push
    return self.cloud.push(used, jobs, remote=remote)
  File "/usr/local/lib/python3.6/dist-packages/dvc/data_cloud.py", line 63, in push
    show_checksums=show_checksums,
  File "/usr/local/lib/python3.6/dist-packages/dvc/remote/local.py", line 378, in push
    download=False,
  File "/usr/local/lib/python3.6/dist-packages/dvc/remote/local.py", line 351, in _process
    download=download,
  File "/usr/local/lib/python3.6/dist-packages/dvc/remote/local.py", line 272, in status
    md5s, jobs=jobs, name=str(remote.path_info)
  File "/usr/local/lib/python3.6/dist-packages/dvc/remote/ssh/__init__.py", line 333, in cache_exists
    ret = list(itertools.compress(checksums, in_remote))
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 586, in result_iterator
    yield fs.pop().result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.6/dist-packages/dvc/remote/ssh/__init__.py", line 326, in exists_with_progress
    return self.batch_exists(chunks, callback=pbar.update_desc)
  File "/usr/local/lib/python3.6/dist-packages/dvc/remote/ssh/__init__.py", line 294, in batch_exists
    channels = ssh.open_max_sftp_channels()
  File "/usr/local/lib/python3.6/dist-packages/dvc/remote/ssh/connection.py", line 306, in open_max_sftp_channels
    self._sftp_channels.append(self._ssh.open_sftp())
  File "/usr/local/lib/python3.6/dist-packages/paramiko/client.py", line 556, in open_sftp
    return self._transport.open_sftp_client()
  File "/usr/local/lib/python3.6/dist-packages/paramiko/transport.py", line 1097, in open_sftp_client
    return SFTPClient.from_transport(self)
  File "/usr/local/lib/python3.6/dist-packages/paramiko/sftp_client.py", line 169, in from_transport
    chan.invoke_subsystem("sftp")
  File "/usr/local/lib/python3.6/dist-packages/paramiko/channel.py", line 72, in _check
    return func(self, *args, **kwds)
  File "/usr/local/lib/python3.6/dist-packages/paramiko/channel.py", line 283, in invoke_subsystem
    self._wait_for_event()
  File "/usr/local/lib/python3.6/dist-packages/paramiko/channel.py", line 1226, in _wait_for_event
    raise e
paramiko.ssh_exception.SSHException: Channel closed.
------------------------------------------------------------

yes, I can ssh into the server machine by myself.

[aabramovrepo]

@aabramovrepo Also, are you able to sftp into that machine from CLI?

Nope! I'm getting the following error message:

subsystem request failed on channel 0
Connection closed

[efiop]

@aabramovrepo Oh, if you are not able to sftp into that machine without dvc, then I would start with sorting out that issue first. I'm not that familiar with sftp configuration and synology so I don't really know if that is a known issue or not. Googling around for synology subsystem request failed on channel 0 I'm finding plenty of links like https://community.synology.com/enu/forum/17/post/115230 discussing seemingly the same issue. Please check out those links, see if anything works for you. I'll close this issue for now, as it is a synology issue. Please ping us and reopen this issue if sftp from cli starts working, but dvc still doesn't work. Thanks! 🙂

[shcheklein]

@efiop should we update our docs (and website?) to mention that SFTP support is required?

[efiop]

@shcheklein It seems pretty obvious, but sure treeverse/dvc.org#781

[shcheklein]

@efiop may be I don't quite understand this (like is SFTP the only way to deal with SSH?), could you elaborate a little bit why is it obvious? May be we don't need that ticket indeed.

[efiop]

@shcheklein Not aware of any other way. But we don't mention SFTP in our docs at all, so I guess it is worth documenting.

[aabramovrepo]

@efiop I have resolved the problem with sftp, now I can ssh and sftp into the disk station machine, furthermore, I can copy files and directories there via scp. However, dcp push gives me the following error message:

ERROR: failed to upload '.dvc/cache/94/d190064798aeae79ea2a1eafd0042e' to 'ssh://gituser@<ip address>/volume1/DVC/94/d190064798aeae79ea2a1eafd0042e' - unable to create remote directory '/volume1': [Errno 13] Permission denied

Do you know what might be the reason for that? I'm a bit surprised seeing no colon between ip address and directory on the server. dvc remote list prints the following:

remote-ssh-nas ssh://gituser@<ip address>:/volume1/DVC

for copying files via scp the following command is used (with colon):

scp -r .dvc/cache/ gituser@<ip address>:/volume1/DVC

and it works just fine.

Thanks in advance for your help!

Cheers,
Alexey