ssh: speed up open()

[mathias3]

Bug Report

DVC api for python with ssh server has extremely low download capacity

        repo = "git@bitbucket.org:xxxxxxxxxxx-dvc-test.git"

        logger.info(f"Loading model with DVC, path={path}, repo={repo}")

        with dvc.api.open(path, repo, mode="rb") as f:

            logger.info(f"Opened file object: {f}")

            model = pickle.load(f)

            logger.info(f"Loaded pickled model: {model}")

this gives around 100kb/s download (tested in three idependent tests on diffrent machines}
dvc pull, dvc push at the same to the same repo and ssh server work with almost maximum network capacity ( around 2 MB/sec
)

2020-07-17 12:25:38,357 DEBUG: Popen(['git', 'version'], cwd=/home/xxxxxxxx/app, universal_newlines=False, shell=None, istream=None)                                      

2020-07-17 12:25:38,361 DEBUG: Popen(['git', 'version'], cwd=/home/xxxxxxxxx/app, universal_newlines=False, shell=None, istream=None)                                      

Cloning                                                         |0.00 [00:00,      ?obj/s]2020-07-17 12:25:38,367 DEBUG: Popen(['git', 'version'], cwd=/home/xxxxxxxxx/app,

 universal_newlines=False, shell=None, istream=None)                                                                                                                                                        

2020-07-17 12:25:38,371 DEBUG: Popen(['git', 'clone', '--no-single-branch', '--progress', '-v', 'git@bitbucket.org:xxxxxxxx-dvc-test.git', '/tmp/tmphut6vnmndvc-clone'], cwd=/home/xxxxxxxxx/app, universal_newlines=True, shell=None, istream=None)                                                                                                                    

2020-07-17 12:25:42,193 DEBUG: Popen(['git', 'cat-file', '--batch-check'], cwd=/tmp/tmphut6vnmndvc-clone, universal_newlines=False, shell=None, istream=<valid stream>)                                     

2020-07-17 12:25:42,197 DEBUG: Popen(['git', 'cat-file', '--batch'], cwd=/tmp/tmphut6vnmndvc-clone, universal_newlines=False, shell=None, istream=<valid stream>)                                           

2020-07-17 12:25:42,204 DEBUG: Popen(['git', 'check-ignore', '/tmp/tmphut6vnmndvc-clone/.dvc/config.local'], cwd=/tmp/tmphut6vnmndvc-clone, universal_newlines=False, shell=None, istream=None)             

2020-07-17 12:25:42,209 DEBUG: Popen(['git', 'check-ignore', '/tmp/tmphut6vnmndvc-clone/.dvc/tmp'], cwd=/tmp/tmphut6vnmndvc-clone, universal_newlines=False, shell=None, istream=None)                      

2020-07-17 12:25:42,213 DEBUG: Popen(['git', 'check-ignore', '/tmp/tmphut6vnmndvc-clone/.dvc/cache'], cwd=/tmp/tmphut6vnmndvc-clone, universal_newlines=False, shell=None, istream=None)                    

2020-07-17 12:25:43,175 INFO: Opened file object: <paramiko.sftp_file.SFTPFile object at 0x7f901eee83d0>                                                                                                    

2020-07-17 13:15:51,701 INFO: Loaded pickled model: xxxxxxx88

as you can see it takes 50 minutes to load 400mb model

Please provide information about your setup

Output of dvc version:

$ dvc version
DVC version: 1.1.7
Python version: 3.7.1
Platform: Linux-5.4.0-42-generic-x86_64-with-debian-buster-sid
Binary: True
Package: deb
Supported remotes: azure, gdrive, gs, hdfs, http, https, s3, ssh, oss
Filesystem type (workspace): ('ext4', '/dev/mapper/ubuntu--vg-root')

**Additional Information (if any):
**

If applicable, please also provide a --verbose output of the command, eg: dvc add --verbose.

[efiop]

Hi @mathias3 ! Did you dvc pull a particular model only? Or everything? How did you measure dvc pull performance?

[mathias3]

2020-07-30 13:38:15,972 DEBUG: Downloading 'ssh://root@xxxxxxxxxxxxdvc-storage/55/e1a87b995976b55ed068813f5a1bb8' to '.dvc/cache/55/e1a87b995976b55ed068813f5a1bb8'                                                                                                                                                               
2020-07-30 13:38:15,973 DEBUG: Downloading 'ssh://root@xxxxxxxxxxx-dvc-storage/91/963ffdc0dc94c1d2067eef4982b8ad' to '.dvc/cache/91/963ffdc0dc94c1d2067eef4982b8ad'                                                                                                                                                               
                                                                                                                                                                       
                                                                                                                                                                        
2020-07-30 13:38:15,976 DEBUG: Downloading 'ssh://root@xxxxxxxxxxdvc-storage/16/8b3fa34092f43451a9efa5488cf0d1' to '.dvc/cache/16/8b3fa34092f43451a9efa5488cf0d1'
2020-07-30 13:38:19,283 DEBUG: Downloading 'ssh://root@xxxxxxxxx-dvc-storage/9c/ded1117e8c8cdaeae96ed158c3a251' to '.dvc/cache/9c/ded1117e8c8cdaeae96ed158c3a251'                                  2020-07-30 13:38:20,025 DEBUG: Downloading 'ssh://root@xxxxxxxxxxx-dvc-storage/80/35597101578b1f27fbb9009413b3d3' to '.dvc/cache/80/35597101578b1f27fbb9009413b3d3'
2020-07-30 13:38:44,704 DEBUG: Downloading 'ssh://root@xxxxxxx-dvc-storage/c1/9edfc255478eb6dc44358f8e1a7673' to '.dvc/cache/c1/9edfc255478eb6dc44358f8e1a7673'
2020-07-30 13:38:46,954 DEBUG: Downloading 'ssh://root@xxxxxxxxx-dvc-storage/0f/72f75fec8840081f3ed8025b4781f6' to '.dvc/cache/0f/72f75fec8840081f3ed8025b4781f6'
2020-07-30 13:39:01,343 DEBUG: Downloading 'ssh://root@xxxxxxxxxxxdvc-storage/21/a846788a18df2281c968c1adaafb60' to '.dvc/cache/21/a846788a18df2281c968c1adaafb60'
2020-07-30 13:39:04,124 DEBUG: Downloading 'ssh://root@xxxxxxxxxxxx-dvc-storage/bd/412a9a843a8608e128ce6d57ed1cfa' to '.dvc/cache/bd/412a9a843a8608e128ce6d57ed1cfa'                                  
2020-07-30 13:58:15,388 DEBUG: Path '.dvc/cache/b6/42b7be7979bdde907254a4dde28a83' inode '11801939'                                                                                                                
2020-07-30 13:58:15,388 DEBUG: fetched: []                                                                                                                                                                         
2020-07-30 13:58:15,389 DEBUG: Path '.dvc/cache/91/963ffdc0dc94c1d2067eef4982b8ad' inode '11811706'                                                                                                                
2020-07-30 13:58:15,389 DEBUG: fetched: [('1595569421551134208', '67772', 'a0426566d1aa2236a528ddad2b043af1', '1595570369791958272')]                                                                              
2020-07-30 13:58:15,390 DEBUG: Path '.dvc/cache/16/8b3fa34092f43451a9efa5488cf0d1' inode '11811707'                                                                                                                
2020-07-30 13:58:15,391 DEBUG: fetched: [('1595569876209925888', '15251', '9cded1117e8c8cdaeae96ed158c3a251', '1595570369792394496')]
2020-07-30 13:58:15,391 DEBUG: Path '.dvc/cache/b0/cf6b7539d3f3979a96bab3c5912ead.dir' inode '11796872'
2020-07-30 13:58:15,392 DEBUG: fetched: [('1596109093659614208', '232', 'b0cf6b7539d3f3979a96bab3c5912ead.dir', '1596109093665938944')]
2020-07-30 13:58:15,392 DEBUG: Path '.dvc/cache/55/e1a87b995976b55ed068813f5a1bb8' inode '11811703'
2020-07-30 13:58:15,393 DEBUG: fetched: []
2020-07-30 13:58:15,394 DEBUG: Path '.dvc/cache/a0/426566d1aa2236a528ddad2b043af1' inode '11811709'
2020-07-30 13:58:15,394 DEBUG: fetched: [('1595570369586948608', '232', 'b0cf6b7539d3f3979a96bab3c5912ead.dir', '1595570369792903168')]
2020-07-30 13:58:15,395 DEBUG: Path '.dvc/cache/9c/ded1117e8c8cdaeae96ed158c3a251' inode '11811878'
2020-07-30 13:58:15,395 DEBUG: fetched: []
2020-07-30 13:58:15,396 DEBUG: Path '.dvc/cache/80/35597101578b1f27fbb9009413b3d3' inode '11812526'
2020-07-30 13:58:15,397 DEBUG: fetched: []
2020-07-30 13:58:15,398 DEBUG: Path '.dvc/cache/c1/9edfc255478eb6dc44358f8e1a7673' inode '11818998'
2020-07-30 13:58:15,398 DEBUG: fetched: []
2020-07-30 13:58:15,399 DEBUG: Path '.dvc/cache/0f/72f75fec8840081f3ed8025b4781f6' inode '11834640'
2020-07-30 13:58:15,400 DEBUG: fetched: []
2020-07-30 13:58:15,401 DEBUG: Path '.dvc/cache/21/a846788a18df2281c968c1adaafb60' inode '11834642'
2020-07-30 13:58:15,401 DEBUG: fetched: []
2020-07-30 13:58:15,402 DEBUG: Path '.dvc/cache/bd/412a9a843a8608e128ce6d57ed1cfa' inode '11834649'
2020-07-30 13:58:15,402 DEBUG: fetched: []

Here I pulled whole repo and there are 10 models of similar size- Today I have very weak connection but I puled all in 20 minutes

So to sum up - yesterday 400 mb dvc.open in 50 minutes. Yesterday whole repo (4GB) with dvc pull 2 minutes. Today on weak internet whole repo (4 gb) dvc pull in 20 minutes ( full internet connection capacity at the moment).

[efiop]

@mathias3 Got it. Makes sense. Currently, we are streaming directly from remote using stfp.open https://github.com/iterative/dvc/blob/99ffa7c6fd9dbb63c04e950e557ca0816529fe6b/dvc/tree/ssh/connection.py#L322 , which could be tweaked to make it faster (prefetch, increase buffer size etc). Another option is to add a flag that would tell dvc whether you want to pull the whole thing and then open local file file or stream it directly from remote (e.g. https://github.com/iterative/dvc/blob/99ffa7c6fd9dbb63c04e950e557ca0816529fe6b/dvc/repo/__init__.py#L587 , where we explicitly tell it to stream right now). Sounds like both options would work in your particular scenario since you are loading the whole thing anyway.

A workaround for now, might be to

from dvc.repo import Repo

Repo.get(repo, path, out="mymodel.pkl")

with open("mymodel.pkl") as fobj:
    model = pickle.load(fobj)

which should pull internally, giving you a better speed. Would that work you for now?

[efiop]

Closing as stale.

[mathias3]

This hack of yours indeed works, but will there be any leas troublesome and clear wav of solving it on your side. It looks and feels like dirty hack

[efiop]

@mathias3 Yes, the ssh issue is a well-known one, we'll get to optimizing it in the future. We are of course happy to accept a patch too. But still, if you need the whole file locally, dvc pull will likely be faster, epecially if you want multiple files.