Speed up dvc status for large projects

[Ykid]

Please provide information about your setup
DVC version(i.e. dvc --version), Platform and method of installation (pip, homebrew, pkg Mac, exe (Windows), DEB(Linux), RPM(Linux))

dvc version

DVC version: 0.82.8
Python version: 3.7.6
Platform: Linux-5.0.0-37-generic-x86_64-with-debian-buster-sid
Binary: False
Package: pip
Cache: reflink - not supported, hardlink - not supported, symlink - supported
Filesystem type (cache directory): ('ext4', '/dev/sda1')
Filesystem type (workspace): ('ext4', '/dev/nvme0n1p2')

expect dvc status can return in a shorter time, currently it takes 30s.

a few stats

du -shL my-project # 107G

dvc status -v

log.txt

[pared]

I think we should try to reproduce such big repo and see what takes so much time, I suspect db access.

[shcheklein]

@pared @Ykid could we run it with a profiler please:

python -m cProfile -o status.prof -m dvc status -v

[Suor]

Reading through discord dialog and looking at the log I don't think this is about big repo. This is probably about many imported files, which create many clones on dvc status.

[Ykid]

@Suor grep --include=\*.{dvc,} -rn my-data-dir -e ".git" | wc -l gives me 64. I think more or less like that. I have a dvc data registry and import some directories and files from there into the project. So does it sound better if I do dvc get followed by dvc add ? those imported files is not going to be updated very frequently after all.

num of files versioned by dvc is around 9k from find -L my-data-dir -type f ! -name "*.dvc" | wc -l. dvc pull shows 8.31k ( the number changes in each line of dvc pull though, but 8.31k is the most time consuming one )

may I know if there's any way to improve it ?

[Suor]

So does it sound better if I do dvc get followed by dvc add

This is obviously an issue on our end. I will think how this may be sped up.

[Suor]

@Ykid Can you say how many different sources do you use in those 64 import stages? Source is a pair of (url, rev) in deps.repo.

[Ykid]

@Suor There is one url as it is our data registry. I follow data registries to set it up. For the number of pairs, here it is:

• 60: (repo, revA)
• 4: (repo, revB)

[Suor]

@Ykid thanks.

[Suor]

A note from discord - git repo is big, both history and checked out things:

84K ./tools
1.8M  ./.dvc
988M  ./notebooks
20K ./my-project.egg-info
16K ./configs
870M  ./.git
16M ./my-project
92K ./dockerfiles
28K ./tests
6.3M  ./data
24K ./pipeline
1.9G  .

The last change only caches single instance of repo, not all of them, which prevents us from needing new git clone, but makes a copy of the whole repo sans dvc cache each time. The reason for that is we do git pull each time, i.e. modifying the directory, which means caching cannot be used for it. Things are furthermore complicated by summon publishing stuff, which modifies the directory returned by external_repo(), which also requires a separate copy.

I am trying to untangle it now.

[Ykid]

The last change only caches single instance of repo, not all of them

May I know what this means ?