Using dvc only for dataset management (e.g. no dvc run pipeline).

[JoeyCarson]

I am dealing with a large hierarchical data set. One where artifacts are pulled from various directories to generate contiguous data sets that are then fed to ML processes downstream. I don't want to use dvc to reproduce the pipeline, at least not yet. My needs are rather to be able to version the overall image dataset hierarchy, for the purpose of manual inspection of the whole hierarchy and moving images into groups or removing them altogether when necessary.

This enables folks with less ML expertise control the data set they want to build by grouping the content together that they want to pick up when generating the data set. The data set is not a list of images, rather it is a list of lower dimensional feature vectors extracted from those images.

I'm finding dvc taking a potentially unreasonable amount of time to just add and commit. Perhaps I don't understand what I'm doing or haven't set my expectations correctly.

I wanted to keep these operations small in order to ensure things were working well. I have done the following. I have approximately 300K in total in this set right now.

1. store 60K images on local file system, under the data/ directory.
2. dvc add data/
3. dvc push -r remote. I forgot to commit here since things took so long and I wanted to see if pushing worked.
4. store 120K additional images to another sub directory under the data/ directory.
5. dvc add data/ -> goes through all of the files in data/ regardless. I ran -v here and showed the previous files.
6. dvc push -r remote.
7. dvc commit. Here dvc is taking the greater amount of 99% of system memory (13 GB) and appears to be causing disk thrashing. It's been running nearly for a day so far.

I am just looking for some guidance in managing a dataset of this nature using dvc in a way that will not eat up so much time, disk, compute, etc. If I'm doing something suboptimal, then I want to shine some light on that.

[shcheklein]

Here dvc is taking the greater amount of 99% of system memory (13 GB) and appears to be causing disk thrashing. It's been running nearly for a day so far.

sounds like a bug to me.

you actually don't even need to run it if you didn't use dvc add --no-commit or something similar.

large hierarchical data set

so, why keep it under a single dvc add if you can use dvc add data/feature1, dvc add data/feature2, etc?

I'm finding dvc taking a potentially unreasonable amount of time to just add and commit.

commit - see above, we'll try to reproduce and see what's happening here
add - adding more files requires analyzing the directory again for now unfortunately, though we cache certain things. Let's see anyway what's happening here as well - could you please share dvc version first?

[JoeyCarson]

Thanks @shcheklein, I am thinking that adding each individual directory is better as well, though I wasn't sure if it would take a very long time to do that as well. I was also trying to keep the command to sync down the dataset simple, so that only the root directory would need to be synchronized.

My thinking here is that perhaps I didn't understand that simply running add more than once on a root directory containing large numbers of files could be causing dvc to run through all of the files each time.

For instance, if I add and commit each subdirectory, should each add only be concerned with that directory? If so, then yes I will do that, but I will need to terminate the existing commit command that has been running so long.

Also, how would this impact when a contributor needs to sync down the full dataset? Will they need to dvc checkout every single dvc file to get every single subdirectory local?

[JoeyCarson]

dvc --version shows 0.63.2

[shcheklein]

@JoeyCarson thanks! could you run specifically, dvc version please - it can show a bit more information. Also, could you please update to the latest version to avoid hitting some already solved issues.

[shcheklein]

For instance, if I add and commit each subdirectory, should each add only be concerned with that directory?

Yes. It will more granular and it'll be analyzing only one subdirectory. So, if the nature of your datasets is hierarchical by design it's better to manage them separately.

If so, then yes I will do that, but I will need to terminate the existing commit command that has been running so long.

trying to reproduce this ... though, like I mentioned you probably don't even need dvc commit.

to dvc checkout every single dvc file to get every single subdirectory local?

no, by default dvc checkout does checkout all the files. But you can specify it a target to do a granular checkout.

Also, in your workflow I would take a look at import/get - there were specifically made to manage feature and data registries. Let me know if you need more details here, we can discuss.

[JoeyCarson]

dvc version shows the following output
DVC Version: 0.63.2
Python version: 3.6.8
Platform: Windows-10-10.0.18362.-SP0.
Binary: False

I figured that using the individual directories would be faster and easier. I will go ahead and do the following, hopefully to not lose the existing cache.

0. rename the directory to use in the remote blob store, so that caches don't overwrite.
1. rename the .dvc directory to .dvc.bak.
2. dvc init
3. Add the smaller directories first. (But how do I handle artifacts that are in the parent directory of those smaller directories without causing it dvc to index those child directories too? I can add each of the individual files in that case I guess.)
4. Skip dvc commit, because it's just not necessary.
5. dvc push -r remoteName

Additionally I will have a look into the import workflow and come back with any questions I have. Thanks for the support!

[shcheklein]

@JoeyCarson sounds good! Please, also take a look here - https://dvc.org/doc/user-guide/running-dvc-on-windows - those are caveats that we found so far running DVC on Windows. Some of them can significantly improve performance.

[shcheklein]

@JoeyCarson also, how large is you data? if you don't edit manually specific files I would also consider turning on this optimization - https://dvc.org/doc/user-guide/large-dataset-optimization

otherwise DVC is copying files on Windows into local cache by default.

[shcheklein]

@JoeyCarson one more quick question. When you do dvc commit, does it print anything? Something like "Creating unpacked dir ..." ?

[JoeyCarson]

Yes! It does and shows that it only makes it to 77% of whatever this means. In fact during the last commit, it showed that twice, both times only getting to 77%.

[shcheklein]

@JoeyCarson thanks! looks like one of our optimization does not like Windows that much. We will test it here.

[JoeyCarson]

Happy to point that out. I am also finding that when running dvc add for one of those smaller subdirectories (only 500 files and one empty child directory inside), it is still hanging a while without doing anything. I also created a dvc repo (ran dvc init separately) and several minutes after I've added the subdirectory, I still see no .dvc/.cache directory. This is the first add, so I expect it to show up, though it doesn't.