SSHException: Channel closed

[gurunathparasaram]

• I had an error saying "paramiko.ssh_exception.SSHException: Channel closed" while using dvc pull or push to an ssh remote.
• I am able to ssh into the machine and also sftp into that machine via sftp user_name@machine_ip.
• Don't know if the following couple of points are relevant:
  • I tried increasing the no. of open files via ulimit from 1024 to 30000...but issue didn't get resolved.
  • Also, a couple of weeks ago, I was getting this "Secssh" warning but the data was pushed.

System details
DVC version: 1.0.0a0
Python version: 3.6.9
Platform: Linux-5.0.0-38-generic-x86_64-with-Ubuntu-19.04-disco
Binary: False
Package: snap
Supported remotes: azure, gdrive, gs, hdfs, http, https, s3, ssh, oss
Cache: reflink - not supported, hardlink - supported, symlink - supported
Repo: dvc, git

Error log:

2020-05-07 19:51:15,283 DEBUG: PRAGMA user_version;                                                               
2020-05-07 19:51:15,283 DEBUG: fetched: [(3,)]                           
2020-05-07 19:51:15,283 DEBUG: CREATE TABLE IF NOT EXISTS state (inode INTEGER PRIMARY KEY, mtime TEXT NOT NULL, si
ze TEXT NOT NULL, md5 TEXT NOT NULL, timestamp TEXT NOT NULL)                                                     
2020-05-07 19:51:15,283 DEBUG: CREATE TABLE IF NOT EXISTS state_info (count INTEGER)                              
2020-05-07 19:51:15,284 DEBUG: CREATE TABLE IF NOT EXISTS link_state (path TEXT PRIMARY KEY, inode INTEGER NOT NULL
, mtime TEXT NOT NULL)                                                                                            
2020-05-07 19:51:15,284 DEBUG: INSERT OR IGNORE INTO state_info (count) SELECT 0 WHERE NOT EXISTS (SELECT * FROM st
ate_info)                                                                                                         
2020-05-07 19:51:15,284 DEBUG: PRAGMA user_version = 3;                  
2020-05-07 19:51:15,288 DEBUG: cache '/home/some_user/.dvc/cache/79/b577b671931842826cc9703b209461.dir' expecte
d '79b577b671931842826cc9703b209461.dir' actual 'None'                                                            
2020-05-07 19:51:15,365 DEBUG: Preparing to download data from 'ssh://some_user@machine/path/to//dvc_remo
te'                                                                                                               
2020-05-07 19:51:15,365 DEBUG: Preparing to collect status from ssh://some_user@machine/path/to//dvc_remo
te                                                                                                                
2020-05-07 19:51:15,365 DEBUG: Collecting information from local cache...                                         
2020-05-07 19:51:15,366 DEBUG: cache '/home/some_user/.dvc/cache/79/b577b671931842826cc9703b209461.dir' expecte
d '79b577b671931842826cc9703b209461.dir' actual 'None'                                                            
2020-05-07 19:51:15,366 DEBUG: Collecting information from remote cache...                  
Enter a private key passphrase or a password for host 'machine' port '22' user 'some_user':                  
                                                                                     
2020-05-07 19:51:18,265 DEBUG: Establishing ssh connection with 'machine' through port '22' as user 'some_user
'                                                                                                                 
2020-05-07 19:51:20,298 DEBUG: SELECT count from state_info WHERE rowid=?                                         
2020-05-07 19:51:20,298 DEBUG: fetched: [(0,)]                                                                    
2020-05-07 19:51:20,298 DEBUG: UPDATE state_info SET count = ? WHERE rowid = ?                                     
2020-05-07 19:51:20,310 ERROR: unexpected error - Channel closed.                

Traceback (most recent call last):                                                                                
  File "/snap/dvc/379/lib/python3.6/site-packages/dvc/main.py", line 49, in main                                   
    ret = cmd.run()                                                                                               
  File "/snap/dvc/379/lib/python3.6/site-packages/dvc/command/data_sync.py", line 32, in run
    recursive=self.args.recursive,                                                                                 
  File "/snap/dvc/379/lib/python3.6/site-packages/dvc/repo/__init__.py", line 30, in wrapper                      
    ret = f(repo, *args, **kwargs)                                                                                
  File "/snap/dvc/379/lib/python3.6/site-packages/dvc/repo/pull.py", line 30, in pull                              
    recursive=recursive,                                                                                          
  File "/snap/dvc/379/lib/python3.6/site-packages/dvc/repo/fetch.py", line 45, in _fetch                           
    recursive=recursive,                                                                                          
  File "/snap/dvc/379/lib/python3.6/site-packages/dvc/repo/__init__.py", line 291, in used_cache
    filter_info=filter_info,                                                                                       
  File "/snap/dvc/379/lib/python3.6/site-packages/dvc/stage/__init__.py", line 791, in get_used_cache             
    cache.update(out.get_used_cache(*args, **kwargs))                                                              
  File "/snap/dvc/379/lib/python3.6/site-packages/dvc/output/base.py", line 454, in get_used_cache                
    self.checksum, self._collect_used_dir_cache(**kwargs),                                                         
  File "/snap/dvc/379/lib/python3.6/site-packages/dvc/output/base.py", line 374, in _collect_used_dir_cache       
    show_checksums=False,                                                                                         
  File "/snap/dvc/379/lib/python3.6/site-packages/dvc/data_cloud.py", line 82, in pull                             
    cache, jobs=jobs, remote=remote, show_checksums=show_checksums                                                
  File "/snap/dvc/379/lib/python3.6/site-packages/dvc/remote/base.py", line 80, in wrapper  
    return f(remote_obj, *args, **kwargs)                                                                         
  File "/snap/dvc/379/lib/python3.6/site-packages/dvc/remote/local.py", line 577, in pull
    download=True,                                                                                                 
  File "/snap/dvc/379/lib/python3.6/site-packages/dvc/remote/local.py", line 475, in _process                     
    download=download,                                                                                            
  File "/snap/dvc/379/lib/python3.6/site-packages/dvc/remote/local.py", line 334, in _status                      
    md5s, jobs=jobs, name=str(remote.path_info)                                                                    
  File "/snap/dvc/379/lib/python3.6/site-packages/dvc/remote/ssh/__init__.py", line 340, in cache_exists
    ret = list(itertools.compress(checksums, in_remote))                     
    
   ret = list(itertools.compress(checksums, in_remote))                                                    [14/77]
  File "/snap/dvc/379/usr/lib/python3.6/concurrent/futures/_base.py", line 586, in result_iterator
    yield fs.pop().result()
  File "/snap/dvc/379/usr/lib/python3.6/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/snap/dvc/379/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/snap/dvc/379/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/snap/dvc/379/lib/python3.6/site-packages/dvc/remote/ssh/__init__.py", line 333, in exists_with_progress
    return self.batch_exists(chunks, callback=pbar.update_msg)
  File "/snap/dvc/379/lib/python3.6/site-packages/dvc/remote/ssh/__init__.py", line 301, in batch_exists
    channels = ssh.open_max_sftp_channels()
  File "/snap/dvc/379/lib/python3.6/site-packages/dvc/remote/ssh/connection.py", line 309, in open_max_sftp_channel
s
    self._sftp_channels.append(self._ssh.open_sftp())
  File "/snap/dvc/379/lib/python3.6/site-packages/paramiko/client.py", line 556, in open_sftp
    return self._transport.open_sftp_client()
  File "/snap/dvc/379/lib/python3.6/site-packages/paramiko/transport.py", line 1097, in open_sftp_client
    return SFTPClient.from_transport(self)
  File "/snap/dvc/379/lib/python3.6/site-packages/paramiko/sftp_client.py", line 169, in from_transport
    chan.invoke_subsystem("sftp")
  File "/snap/dvc/379/lib/python3.6/site-packages/paramiko/channel.py", line 72, in _check
    return func(self, *args, **kwds)
  File "/snap/dvc/379/lib/python3.6/site-packages/paramiko/channel.py", line 283, in invoke_subsystem
    self._wait_for_event()
  File "/snap/dvc/379/lib/python3.6/site-packages/paramiko/channel.py", line 1226, in _wait_for_event
    raise e
paramiko.ssh_exception.SSHException: Channel closed.
------------------------------------------------------------

Please let me know if you need more details. Any pointers on debugging my settings(if they may be wrong) would be helpful. Thanks.

[efiop]

@gurunath-p Did anything change in your setup or on the server? Maybe you are using separate credentials in your dvc config that got expired? Would definitely double check your config and creds. Can't put my finger or anything specific 🙁

[gurunathparasaram]

@efiop spent some hours checking this...seems like the connection cannot be established, checked some issues on paramiko similar to this:

• Used the same credentials as stored in dvc to ssh and sftp which work on the bash CLI.
• Also, read in the "dvc remote" docs page that for sftp, paths relative to the Chroot for the sftp connection are to be mentioned in the .dvc config, when the root used for SFTP connection may be different than the actual physical root.
• Here is the extract from my /etc/ssh/sshd_config file:

Match User dvc_user
ForceCommand internal-sftp
PasswordAuthentication yes
ChrootDirectory /mnt/dvc_remote
PermitTunnel no
AllowAgentForwarding no
AllowTcpForwarding no
X11Forwarding no

And my .dvc/config file:

[core]
remote = origin
['remote "origin"']
url = ssh://dvc_user@remote_machine:/mnt/dvc_remote
ask_password = true

• I also tried setting the url in the above .dvc/config file to not explicitly point to the /mnt/dvc_remote directory where the dvc files are stored, as shown below, which didn't work:
  url = ssh://dvc_user@remote_machine:/

• Here I tested connected to the remote server using a sample script shown below via the paramiko SSHClient and I am able to connect to the remote machine:

import paramiko
s = paramiko.SSHClient()
s.set_missing_host_key_policy(paramiko.AutoAddPolicy())
conn = s.connect("123.456.78.9", username ='user', password='somerandomstring', port=22)

command = 'pwd'
(stdin, stdout, stderr) = s.exec_command(command)

print(stdout.read())
s.close()

• I tried fiddling around to set the Chroot directory of the sftp connection to the actual physical root and restarted the ssh service but even that didn't work.
• Also, tried creating a new user on the remote machine and tried similar things, but that too didn't work.
• Do you see anything wrong in the config I pasted above in the /etc/ssh/sshd_config or my dvc config file?
• Also, I am checking paramiko issues/channels to see if anything helps. Thanks.

[efiop]

Here I tested connected to the remote server using a sample script shown below via the paramiko SSHClient and I am able to connect to the remote machine:

Could you also try to open sftp connection through paramiko? If that doesn't work - something is up with paramiko.

Also, you've mentioned that it used to work before. Could you downgrade to dvc 0.93.0 and see if that makes it work?

[efiop]

Closing as stale.